From p.f.moore at gmail.com Thu Jun 1 03:44:45 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Jun 2017 08:44:45 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> Message-ID: On 1 June 2017 at 01:08, Donald Stufft wrote: > A sdist is a .tar.gz or a .zip file with a directory structure like (along > with whatever additional files the project needs in the sdist): [...] I'm confused. Isn't this basically what PEP 517 says already? You've added some details and clarification, but that could just as easily be done in a separate document/PEP. The details aren't needed for PEP 517 itself. In practical terms, tools that want to leave the API details to someone else can call out to pip, as I suggested. And I suspect many tools probably will, simply because it's easier than dealing with the two APIs directly. Paul From donald at stufft.io Thu Jun 1 06:34:37 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 06:34:37 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> Message-ID: <8EC01291-FC52-45F8-8D15-BD51924ABE84@stufft.io> > On Jun 1, 2017, at 3:44 AM, Paul Moore wrote: > > On 1 June 2017 at 01:08, Donald Stufft wrote: >> A sdist is a .tar.gz or a .zip file with a directory structure like (along >> with whatever additional files the project needs in the sdist): > [...] > > I'm confused. Isn't this basically what PEP 517 says already? You've > added some details and clarification, but that could just as easily be > done in a separate document/PEP. The details aren't needed for PEP 517 > itself. > Yes, it?s basically what PEP 517 says already just more specific and detailed. I don?t know what more people want from ?defining what an sdist is?, because that?s basically all an sdist is. I?ve always been of the opinion that PEP 517 is already defining (and then modifying) what an sdist is and I don?t know what more people would want. PEP 517 needs to do it because PEP 517 wants to change the definition of what a sdist is, and you can?t really change the definition without in fact defining the new thing. I mean we could make a new PEP that just defines sdist (minus the pyproject.toml part) then make PEP 517 extend that PEP and add the pyproject.toml? but that seems kind of silly to me? Splitting it out into it?s own PEP gains us nothing and to me, feels like extra process for process?s sake. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at gedmin.as Thu Jun 1 08:12:40 2017 From: marius at gedmin.as (Marius Gedminas) Date: Thu, 1 Jun 2017 15:12:40 +0300 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> Message-ID: <20170601121240.fpkpqrj2rbuznrqn@platonas> On Wed, May 31, 2017 at 08:08:51PM -0400, Donald Stufft wrote: > I think this should cover the case of actually making the project pip > installable (assuming of course the setup.py or build backend doesn?t do > something silly like always sys.exit(1) instead of produce the expected > outcome) My personal favorite was PyGame doing raw_input() from its setup.py. Marius Gedminas -- After having done some test using hi-tech istruments (moving my mouse during a kernel build) [...] -- Davide Libenzi on lkml -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: not available URL: From thomas at kluyver.me.uk Thu Jun 1 13:18:48 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 01 Jun 2017 18:18:48 +0100 Subject: [Distutils] Malicious packages on PyPI Message-ID: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> Are we aware of this? http://evilpackage.fatezero.org/ I recall there were a couple of these before which were taken down, but someone appears to have made a cookiecutter template so you can very easily claim names on PyPI, and anyone who installs that package will submit their information to that site. A couple that are up at the moment: https://pypi.python.org/pypi/requirements-txt/1.1.1 https://pypi.python.org/pypi/ztz/0.1.1 Do we delete them? Do we try to detect similar packages being uploaded and block them? I suspect it's a waste of time to try to prevent this in general, but maybe it's worth protecting likely names that people might 'pip install' by mistake, such as requirements-txt. Thomas From thomas at kluyver.me.uk Thu Jun 1 13:24:35 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 01 Jun 2017 18:24:35 +0100 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> Message-ID: <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> On closer examination, those packages do not actually appear to upload any information - they seem to be empty packages placed there to serve as a warning. It's not clear to me whether the data on the fatezero.org website is from other packages which really do upload data, or if it's fake. On Thu, Jun 1, 2017, at 06:18 PM, Thomas Kluyver wrote: > Are we aware of this? > http://evilpackage.fatezero.org/ > > I recall there were a couple of these before which were taken down, but > someone appears to have made a cookiecutter template so you can very > easily claim names on PyPI, and anyone who installs that package will > submit their information to that site. A couple that are up at the > moment: > > https://pypi.python.org/pypi/requirements-txt/1.1.1 > https://pypi.python.org/pypi/ztz/0.1.1 > > Do we delete them? Do we try to detect similar packages being uploaded > and block them? I suspect it's a waste of time to try to prevent this in > general, but maybe it's worth protecting likely names that people might > 'pip install' by mistake, such as requirements-txt. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From matt at nycresistor.com Thu Jun 1 13:32:19 2017 From: matt at nycresistor.com (Matt Joyce) Date: Thu, 1 Jun 2017 13:32:19 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> Message-ID: https://github.com/fate0/cookiecutter-evilpy-package/tree/master/%7B%7Bcookiecutter.package_name%7D%7D that's the package repo on github. It's basically a test dummy package that reports users who have ran that package template. the site referenced lists the package name that the user ran to get posted to the site. there appear to be many packages in pypi that are built off this fatezero template. it is non destructive... as a test payload. but the method used is obviously highly successful as an attack vector. there may be more nefarious packages already in pypi. pypi is not a very good package management solution. most folks I advise to build from pypi in CI/CD but push to production via a real package management solution such as apt or yum. always double check sources coming from the internet. -Matt On Thu, Jun 1, 2017 at 1:24 PM, Thomas Kluyver wrote: > On closer examination, those packages do not actually appear to upload > any information - they seem to be empty packages placed there to serve > as a warning. > > It's not clear to me whether the data on the fatezero.org website is > from other packages which really do upload data, or if it's fake. > > On Thu, Jun 1, 2017, at 06:18 PM, Thomas Kluyver wrote: > > Are we aware of this? > > http://evilpackage.fatezero.org/ > > > > I recall there were a couple of these before which were taken down, but > > someone appears to have made a cookiecutter template so you can very > > easily claim names on PyPI, and anyone who installs that package will > > submit their information to that site. A couple that are up at the > > moment: > > > > https://pypi.python.org/pypi/requirements-txt/1.1.1 > > https://pypi.python.org/pypi/ztz/0.1.1 > > > > Do we delete them? Do we try to detect similar packages being uploaded > > and block them? I suspect it's a waste of time to try to prevent this in > > general, but maybe it's worth protecting likely names that people might > > 'pip install' by mistake, such as requirements-txt. > > > > Thomas > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Thu Jun 1 13:40:50 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 01 Jun 2017 18:40:50 +0100 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> Message-ID: <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> On Thu, Jun 1, 2017, at 06:32 PM, Matt Joyce wrote: > It's basically a test dummy package that reports users who have ran > that package template. That's what I thought, but all the code to do the upload seems to have been removed before s/he built those packages. Now it's just a harmless warning, unless I'm missing something. https://github.com/fate0/cookiecutter-evilpy-package/commit/a3ed1e1e060748b0444158ea3bc569dfbf57645e > the site referenced lists the package name that the user ran to get > posted to the site. there appear to be many packages in pypi that > are built off this fatezero template. There *appear* to be, but I checked several of the names listed there, and they're not on PyPI: https://pypi.python.org/pypi/tkinter https://pypi.python.org/pypi/memcached https://pypi.python.org/pypi/vtk https://pypi.python.org/pypi/python-dev https://pypi.python.org/pypi/opencv So I wonder if the data is fake. Or maybe they were already taken down? Or the installations are real, but not using those names. > pypi is not a very good package management solution. most folks I > advise to build from pypi in CI/CD but push to production via a real > package management solution such as apt or yum. always double check > sources coming from the internet. It's an open repository that anyone can upload to. That has its drawbacks and its advantages. -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Thu Jun 1 14:12:12 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Thu, 1 Jun 2017 13:12:12 -0500 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <8EC01291-FC52-45F8-8D15-BD51924ABE84@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> <8EC01291-FC52-45F8-8D15-BD51924ABE84@stufft.io> Message-ID: On Thu, Jun 1, 2017 at 5:34 AM, Donald Stufft wrote: > > On Jun 1, 2017, at 3:44 AM, Paul Moore wrote: > > On 1 June 2017 at 01:08, Donald Stufft wrote: > > A sdist is a .tar.gz or a .zip file with a directory structure like (along > with whatever additional files the project needs in the sdist): > > [...] > > I'm confused. Isn't this basically what PEP 517 says already? You've > added some details and clarification, but that could just as easily be > done in a separate document/PEP. The details aren't needed for PEP 517 > itself. > > > Yes, it?s basically what PEP 517 says already just more specific and > detailed. I don?t know what more people want from ?defining what an sdist > is?, because that?s basically all an sdist is. I?ve always been of the > opinion that PEP 517 is already defining (and then modifying) what an sdist > is and I don?t know what more people would want. > > PEP 517 needs to do it because PEP 517 wants to change the definition of > what a sdist is, and you can?t really change the definition without in fact > defining the new thing. I mean we could make a new PEP that just defines > sdist (minus the pyproject.toml part) then make PEP 517 extend that PEP and > add the pyproject.toml? but that seems kind of silly to me? Splitting it > out into it?s own PEP gains us nothing and to me, feels like extra process > for process?s sake. > PEP 518's pyproject.toml only specifies a single table, `build-system`, that matters. Can we just add a blurb to PEP 517 that says something to the effect of "If the following sub table exists, its location key can be used to pre-populate the metadata_directory of `get_wheel_metadata` automatically": [build-system.metadata] directory = some_dist_info_directory/ (pulled from the spec in 517 about what get_wheel_metadata is supposed to do) Then we could default that directory to something obvious, like the aforementioned ./DIST-INFO or ./.dist-info, or whatever, because isn't such a directory expected to contain enough information to create a wheel anyway? Like {package-name and {version} via METADATA? And typically included in sdists already? If it has a SOURCE-RECORD file [new], then pip and friends can use that t o know what files are needed for the build, and can use pyproject.toml (if it exists) for creating and/or updating it for later sdist generation. In the simple case, every normal file in a wheel is also in an sdist, verbatim, with no additional artifacts of any kind (pure python) and only additional metadata. The build doesn't care if things like LICENCE are in the tree. If there is no static SOURCE-RECORD, pip and friends fallback to a wholesale copy operation of the input source. The build backend's `get_wheel_metadata` (if defined) can update or backfill missing information within the METADATA file, and create the WHEEL file (or save that for `build_wheel`), if it finds the `metadata.directory` seeded from the static location referenced in pyproject.tom l is incomplete. In the end, the build frontend logic would look something like: (also seems like `get_wheel_metadata` should maybe return the final .dist-info directory it decided on, or just settle on DIST-INFO and enough of this name-version.dist-info nonsense already... should possibly be a required build api function with the understanding `build_wheel` might update it) * Is build-system.metadata.directory defined? YES: copy to {metadata_directory}/DIST-INFO NO: mkdir {metadata_directory}/DIST-INFO * Does {metadata_directory}/DIST-INFO/SOURCE-RECORD exist? YES: use that to isolate/prune/copy source tree for initial build, if desired, and also confirm hashes, if any NO: do nothing (we have something that might look like an sdist, but possibly incomplete [eg. still no METADATA]) * Is build-backend.MODULE.get_build_requires defined? YES: make sure those things exist then NO: do nothing * Is build-backend.MODULE.get_wheel_metadata defined? YES: call it like PEP 517 says, DIST-INFO is ready for updating NO: do nothing (we have something that might look like an sdist, but possibly incomplete [eg. still no METADATA]) * Is build-backend.MODULE.build_wheel defined? YES: call it like PEP 517 says, replace RECORD with the final record from build? NO: do nothing * Is {metadata_directory}/DIST-INFO/* valid and the resultant whl as well? YES: YAY! \o/ NO: BLOW UUUUUP * Does {metadata_directory}/DIST-INFO/SOURCE-RECORD exist [must reference pyproject.toml! too]? YES: use that to prune files when creating a proper sdist AFTER the build NO: sdist is original source tree + {metadata_directory}/DIST-INFO - RECORD(?) (we have enough information to produce an complete sdist that could be used to generate a valid wheel again) Because the build itself can output additional source files, that may be desirable to include in an sdist later, I honestly don't think you can pass through a "proper" sdist before a wheel. I think you can 99% of the time do that, but some builds using Cython and friends could actually have a custom initial build that generates standard .h/.c/.py, and even outputs an alternative p yproject.toml that *no longer needs* a custom build backend. Or just straight deletes it from SOURCE-RECORD once the custom build is done, because some artifacts are enough to rebuild a wheel next time. It seems to me the only possibly correct order is: 1. VCS checkout 2. partial sdist, but still likely an sdist, no promises! 3. wheel 4. proper sdist from generated SOURCE-RECORD, or updated static SOURCE-RECORD, or just original source tree + DIST-INFO I don't see a way to get a 100% valid sdist without first building the project and effectively asking the build backend (via its SOURCE-RECORD, if any) "Well golly, you did a build! What, *from both the source tree and build artifacts*, is important for wrapping up into a redistributable?" Maybe I'm overlooking something yuge (I've tried to follow this discussion, and have sort of checked out of python lately, but I'm fairly well-versed in packing lore and code), but in general I think we really are making sdists way way... way scarier than need be. They're pretty much whatever the build tells you is important for redistribution, at the end, with as much static meta data as possible, to the point of possibly obviating their need for pyproject.toml in the first place... maybe this aspect is what is hanging everyone up? A redistibutable source does not need to be as flexible as the original VCS input. An sdist is pinned to a specific version of a project, whereas VCS represents all possible versions (albeit only one is checkout out), and sdists *are not* wheels! The same expectations need not apply. Two sdists of the same version might not be identical; one might request the custom build backed via pyproject.toml, and the other might have already done some of the steps for whatever reason. Authors must decide which is more appropriate for sharing. This ended up longer than I meant, but hopefully it's not all noise. Thanks, -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Jun 1 13:37:34 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 1 Jun 2017 17:37:34 +0000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> Message-ID: <20170601173733.GK12827@yuggoth.org> On 2017-05-31 20:08:51 -0400 (-0400), Donald Stufft wrote: [...] > Both {name} and {version} MUST have any - characters escaped to a > _ to match the escaping done by Wheel. Thus a sdist for a project > named foo-bar with version 1.0-2 which is using a .tar.gz > container for the sdist would produce a file named > foo_bar-1.0_2.tar.gz. [...] While I agree with the reasoning, if this is going to end up enforced by common tooling in the near future a warning period would be appreciated as it implies a fairly significant behavior change which will require quite a lot of adjustments to bespoke automation (at least for some I maintain, and I'm pretty sure there are plenty more out there too). -- Jeremy Stanley From donald at stufft.io Thu Jun 1 14:22:19 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 14:22:19 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> <8EC01291-FC52-45F8-8D15-BD51924ABE84@stufft.io> Message-ID: > On Jun 1, 2017, at 2:12 PM, C Anthony Risinger wrote: > > Because the build itself can output additional source files, that may be desirable to include in an sdist later, I honestly don't think you can pass through a "proper" sdist before a wheel. I think you can 99% of the time do that, but some builds using Cython and friends could actually have a custom initial build that generates standard .h/.c/.py, and even outputs an alternative p > yproject.toml that *no longer needs* a custom build backend. Or just straight deletes it from SOURCE-RECORD once the custom build is done, because some artifacts are enough to rebuild a wheel next time. It seems to me the only possibly correct order is: > > 1. VCS checkout > 2. partial sdist, but still likely an sdist, no promises! > 3. wheel > 4. proper sdist from generated SOURCE-RECORD, or updated static SOURCE-RECORD, or just original source tree + DIST-INFO > > I don't see a way to get a 100% valid sdist without first building the project and effectively asking the build backend (via its SOURCE-RECORD, if any) "Well golly, you did a build! What, *from both the source tree and build artifacts*, is important for wrapping up into a redistributable?" > > Maybe I'm overlooking something yuge (I've tried to follow this discussion, and have sort of checked out of python lately, but I'm fairly well-versed in packing lore and code), but in general I think we really are making sdists way way... way scarier than need be. They're pretty much whatever the build tells you is important for redistribution, at the end, with as much static meta > data as possible, to the point of possibly obviating their need for pyproject.toml in the first place... maybe this aspect is what is hanging everyone up? A redistibutable source does not need to be as flexible as the original VCS input. An sdist is pinned to a specific version of a project, whereas VCS represents all possible versions (albeit only one is checkout out), and sdists > *are not* wheels! The same expectations need not apply. Two sdists of the same version might not be identical; one might request the custom build backed via pyproject.toml, and the other might have already done some of the steps for whatever reason. Authors must decide which is more appropriate for sharing. Do any projects build a copy of the library and use that for influencing what gets copied into the sdist today? As far as I am aware there is not any that do that. I think the standard thing to do in Cython is to produce the .c files as part of the sdist process, which is perfectly fine to do. With the newer PEPs it doesn?t _need_ to do that, since you can depend on Cython in your build steps and just ship the pyx files (although you?re still free to compute the .c files AOT and include them in the sdist). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Thu Jun 1 14:59:49 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Thu, 1 Jun 2017 13:59:49 -0500 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <0F77923A-B251-4A44-BC72-D717E0AD6CBC@stufft.io> <8EC01291-FC52-45F8-8D15-BD51924ABE84@stufft.io> Message-ID: On Thu, Jun 1, 2017 at 1:22 PM, Donald Stufft wrote: > > On Jun 1, 2017, at 2:12 PM, C Anthony Risinger > wrote: > > Because the build itself can output additional source files, that may be > desirable to include in an sdist later, I honestly don't think you can pass > through a "proper" sdist before a wheel. I think you can 99% of the time do > that, but some builds using Cython and friends could actually have a custom > initial build that generates standard .h/.c/.py, and even outputs an > alternative p > yproject.toml that *no longer needs* a custom build backend. Or just > straight deletes it from SOURCE-RECORD once the custom build is done, > because some artifacts are enough to rebuild a wheel next time. It seems to > me the only possibly correct order is: > > 1. VCS checkout > 2. partial sdist, but still likely an sdist, no promises! > 3. wheel > 4. proper sdist from generated SOURCE-RECORD, or updated static > SOURCE-RECORD, or just original source tree + DIST-INFO > > I don't see a way to get a 100% valid sdist without first building the > project and effectively asking the build backend (via its SOURCE-RECORD, if > any) "Well golly, you did a build! What, *from both the source tree and > build artifacts*, is important for wrapping up into a redistributable?" > > Maybe I'm overlooking something yuge (I've tried to follow this > discussion, and have sort of checked out of python lately, but I'm fairly > well-versed in packing lore and code), but in general I think we really are > making sdists way way... way scarier than need be. They're pretty much > whatever the build tells you is important for redistribution, at the end, > with as much static meta > data as possible, to the point of possibly obviating their need for > pyproject.toml in the first place... maybe this aspect is what is hanging > everyone up? A redistibutable source does not need to be as flexible as the > original VCS input. An sdist is pinned to a specific version of a project, > whereas VCS represents all possible versions (albeit only one is checkout > out), and sdists > *are not* wheels! The same expectations need not apply. Two sdists of the > same version might not be identical; one might request the custom build > backed via pyproject.toml, and the other might have already done some of > the steps for whatever reason. Authors must decide which is more > appropriate for sharing. > > > > Do any projects build a copy of the library and use that for influencing > what gets copied into the sdist today? As far as I am aware there is not > any that do that. I think the standard thing to do in Cython is to produce > the .c files as part of the sdist process, which is perfectly fine to do. > With the newer PEPs it doesn?t _need_ to do that, since you can depend on > Cython in your build steps and just ship the pyx files (although you?re > still free to compute the .c files AOT and include them in the sdist). > I admit I don't know of any either. Nor did I know the standard expectation in Cython was to generate things in the sdist phase. I have not personally used it for a project and was only using it as an example of a process that produces potentially redistributable artifacts. What you are saying though I think only emphasizes previous comments about needing to pin down, once and for all, what it means to "be an sdist" (which I agree with), right now, and decide who is responsible for constructing that. We don't need to go on a rampage pronouncing new formats or files requirements like LICENCE... just state the status-quo and accept it. Maybe we can start with simply defining an sdist as "some tree with a DIST-INFO". I'll avoid the package-name.dist-info for now and the problems therein, unless there is a simple consensus there. >From that, there seems to only be a small gap in the build api hooks, and a missing "[sdist-system]" phase (maybe that doesn't sound as nice as build-system) that I believe would be a small PEP or additional to 517. In all honesty, I think probably *both* the sdist-system and the build-system need to be consulted to fully populate DIST-INFO... this can be the `build_sdist` hook in both. In all honestly, as you have clarified, Cython is *not* a build-system! It's an sdist-system. The C compiler is the expected build-system. For many build-systems, `build_sdist` might even be a noop, but it might still want the opportunity to make adjustments before starting. It seems reasonable to me that both systems simply begin from a static DIST-INFO, if any, then work together to populate and update it. Something like setuptools_scm, as a sdist-system, might only generate a barebones METADATA with name and version info, then the selected build-system comes in and fills the rest. Or something like the Cython sdist-system might generate more source files and not even touch DIST-INFO, then the selected build-system comes in and fills the rest. Neither have anything to do with "the build". Something like Cython is effectively doing a partial build before producing a redistributable source tree, and if we skip that step and go straight to a build via the build-system, then the only real option for sdists at that point is to ask the same backend, post-build, for the important redistibutable parts, which may or may not reflect a stable/reproducible input to the same system. If I select "Cython" as the build-system, but I need to use a different compiler for currently-unsupported-platform-X, I'm going to have a bad day. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From xav.fernandez at gmail.com Thu Jun 1 15:31:59 2017 From: xav.fernandez at gmail.com (Xavier Fernandez) Date: Thu, 1 Jun 2017 21:31:59 +0200 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: This makes me remember https://hackernoon.com/building-a-botnet-on-pypi-be1ad280b8d6 on a related note. On Thu, Jun 1, 2017 at 7:40 PM, Thomas Kluyver wrote: > On Thu, Jun 1, 2017, at 06:32 PM, Matt Joyce wrote: > > It's basically a test dummy package that reports users who have ran that > package template. > > > That's what I thought, but all the code to do the upload seems to have > been removed before s/he built those packages. Now it's just a harmless > warning, unless I'm missing something. > > https://github.com/fate0/cookiecutter-evilpy-package/commit/ > a3ed1e1e060748b0444158ea3bc569dfbf57645e > > the site referenced lists the package name that the user ran to get posted > to the site. there appear to be many packages in pypi that are built off > this fatezero template. > > > There *appear* to be, but I checked several of the names listed there, and > they're not on PyPI: > > https://pypi.python.org/pypi/tkinter > https://pypi.python.org/pypi/memcached > https://pypi.python.org/pypi/vtk > https://pypi.python.org/pypi/python-dev > https://pypi.python.org/pypi/opencv > > So I wonder if the data is fake. Or maybe they were already taken down? Or > the installations are real, but not using those names. > > pypi is not a very good package management solution. most folks I advise > to build from pypi in CI/CD but push to production via a real package > management solution such as apt or yum. always double check sources coming > from the internet. > > > It's an open repository that anyone can upload to. That has its drawbacks > and its advantages. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at nycresistor.com Thu Jun 1 15:59:54 2017 From: matt at nycresistor.com (Matt Joyce) Date: Thu, 1 Jun 2017 15:59:54 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: I mean the easy attack vector is find a package where the package name does not match the import namespace. If the import namespace has no corresponding package in pypi... register it. Anyone who blind tries to grab a dependency will grab your module instead of the one they want. Horrible to do. But that's the attack vector. On Thu, Jun 1, 2017 at 3:31 PM, Xavier Fernandez wrote: > This makes me remember https://hackernoon.com/building-a-botnet-on-pypi- > be1ad280b8d6 on a related note. > > On Thu, Jun 1, 2017 at 7:40 PM, Thomas Kluyver > wrote: > >> On Thu, Jun 1, 2017, at 06:32 PM, Matt Joyce wrote: >> >> It's basically a test dummy package that reports users who have ran that >> package template. >> >> >> That's what I thought, but all the code to do the upload seems to have >> been removed before s/he built those packages. Now it's just a harmless >> warning, unless I'm missing something. >> >> https://github.com/fate0/cookiecutter-evilpy-package/commit/ >> a3ed1e1e060748b0444158ea3bc569dfbf57645e >> >> the site referenced lists the package name that the user ran to get >> posted to the site. there appear to be many packages in pypi that are >> built off this fatezero template. >> >> >> There *appear* to be, but I checked several of the names listed there, >> and they're not on PyPI: >> >> https://pypi.python.org/pypi/tkinter >> https://pypi.python.org/pypi/memcached >> https://pypi.python.org/pypi/vtk >> https://pypi.python.org/pypi/python-dev >> https://pypi.python.org/pypi/opencv >> >> So I wonder if the data is fake. Or maybe they were already taken down? >> Or the installations are real, but not using those names. >> >> pypi is not a very good package management solution. most folks I advise >> to build from pypi in CI/CD but push to production via a real package >> management solution such as apt or yum. always double check sources coming >> from the internet. >> >> >> It's an open repository that anyone can upload to. That has its drawbacks >> and its advantages. >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Jun 1 16:45:53 2017 From: brett at python.org (Brett Cannon) Date: Thu, 01 Jun 2017 20:45:53 +0000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: On Wed, 31 May 2017 at 14:14 Thomas Kluyver wrote: > On Wed, May 31, 2017, at 09:16 PM, Donald Stufft wrote: > > How you build the release-quality sdist isn?t really of concern of PEP 517 > any more than building a release quality wheel is, it?s up to the build > tool to implement that as it makes sense for them. > > > But if we have a hook for building something called an sdist, we need to > define what an sdist is. The definition you're referring to is a functional > definition sufficient for a tool that only wants to install it, but it > doesn't cover what an sdist means or how it fits into workflows. > > I could see this as an argument that the PEP should have *both* a > build_sdist and a prepare_build_files hook, if you don?t think that the > build_sdist hook is suitable on it?s own. > > > I would prefer that to the the current status of one hook that tries to > cover both use cases. > > I'd still rather specify prepare_build_files in this PEP, and leave the > sdist hook to a later PEP. I don't see a great need for a standard > interface to building an sdist: the developer doing that by calling their > build tool directly seems adequate for the majority of use cases. > So it sounds like the list_build_files() part of the API is still useful for isolated builds versus in-place builds, correct? As for Thomas' idea that "calling a build tool directly seems adequate" is somewhat interesting and not something I thought about. Let's look at Donald's list of current ways to get from something to installation (which I know we want to scale back): 1) VCS Checkout -> Installed 2) VCS Checkout -> Sdist -> Installed 3) VCS Checkout -> Wheel -> Installed 4) VCS Checkout -> Sdist -> Wheel -> Installed 5) VCS Checkout -> Editable Install 6) VCS Checkout -> Sdist -> Editable Install OK, so what Thomas is suggesting is this isn't all necessarily directly under pip's purview. So if you take that list and break out those steps into "back-end responsibility" and "front-end responsibility" then you end up with: Back-end (e.g. flit): 1) VCS Checkout -> wheel (driven by pip) 2) VCS Checkout -> sdist 2) sdist -> wheel (driven by pip) Front-end (e.g. pip): 1) wheel -> installed You can then generate the non-editable install steps above by combining these roles. And I think the point Thomas is trying to make is if you look at back-end#2 you can simply leave pip out of it in a post-PEP 517 world if you view an sdist as a trimmed down VCS checkout that pip just needs to know how to unpack. But thinking from a more fundamental perspective, why does pip even need to be involved with any part of PEP 517? If pip is meant to be viewed as a package manager then isn't its key focus on fetching and installing appropriate things? Well, it's because pip may have to build a wheel from a post-517 sdist that it downloaded from PyPI when a project doesn't provide e.g. a Windows wheel. That's why pip needs to be PEP 517-aware at all. Otherwise pip could just work in a wheel-only world and not even care about building wheels to begin with. But where does building sdists come into play? You need to be able to make them, and you need to be able to get them up to PyPI. But beyond pip needing to know how to unpack an sdist to make the wheel it wants to work with, pip doesn't actually *need* to produce an sdist. But I think *twine* is the tool that needs a way to specify how to produce an sdist. If we want to view twine as the tool to upload artifacts to PyPI then we need twine to know how to produce sdists and wheels in a PEP 517 world, not pip. Maybe I'm missing something fundamental here about pip and all of this is wrong, but from where I'm sitting it seems the key thing pip needs from PEP 517 is how to go from a bunch of source to a wheel so that it can install that wheel. Now pip needs to know how to *consume* an sdist that it gets from PyPI that has a pyproject.toml, but it technically doesn't need to know how to *produce* one if in the end it really just wants a wheel. Twine, OTOH, does need a way to produce an sdist as well as a wheel so it can upload those to PyPI. And so I think in a very wordy way, I just said we need to stop saying "pip needs a standardized way to produce an sdist" and instead start saying "twine needs a way to produce an sdist". And that leads to the question about whether PEP 517 must cover sdist *production* for twine because we want to have that solved before we have the *consumption* side in pip in place. Or put another way, are we okay with pip consuming things that twine simply can't make from a pyproject.toml file (yet)? A "yes" means PEP 517 shouldn't be held up, while a "no" means we need a solution for twine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jun 1 17:49:46 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Jun 2017 22:49:46 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: On 1 June 2017 at 21:45, Brett Cannon wrote: > And so I think in a very wordy way, I just said we need to stop saying "pip > needs a standardized way to produce an sdist" and instead start saying > "twine needs a way to produce an sdist". And that leads to the question > about whether PEP 517 must cover sdist production for twine because we want > to have that solved before we have the consumption side in pip in place. Or > put another way, are we okay with pip consuming things that twine simply > can't make from a pyproject.toml file (yet)? A "yes" means PEP 517 shouldn't > be held up, while a "no" means we need a solution for twine. The question is more, are we okay with pip consuming things (pyproject.toml source trees) for which we don't define any means of uploading to a package index (which is basically what a sdist is - a way of publishing sources on PyPI). My answer to that is "no". pip doesn't need to be able to provide a user interface that builds a sdist. It *could* provide one, as an enhancement, but it's not *necessary*. Whether the canonical "build a sdist" command should be pip or (say) twine is a side issue here (although the ability to *have* a canonical command, and not rely on each backend having its own incompatible way, *is* important IMO - but that's not the point here). However, there are a number of places where pip indirectly needs the ability to build a sdist (or do something closely equivalent). pip needs a way to deal with "pip install foo". There are 2 scenarios here: 1. The index (PyPI) contains an appropriate wheel. Then pip just installs it (using the wheel library). PEP 517 isn't involved, no build, easy. 2. There is no appropriate wheel. There are 2 subcases here: 2a. There is no sdist. We're stuck. See below. 2b. There is a sdist (whatever that means). As you say, pip needs to be able to consume that sdist. PEP 517 covers this as it stands. pip also needs a way to deal with "pip install . In this case, pip (under its current model) copies that directory to a working area. In that area, it runs the build command to create a wheel, and proceeds from there. In principle, there's little change in a PEP 517 world. But again, see below. There are other things here, but they are largely marginal, or similar to the above cases (the big remaining case is editable installs, but that's a whole other question). Notes: - The case 2a in the "pip install foo" example is of concern. At the moment, building sdists is trivial (python setup.py sdist) and there's effectively no barrier to publishing sdists. In a PEP 517 world, backends may or may not provide a "build a sdist" command (flit doesn't, and for a long time I believe didn't propose to do so - partially because what constituted a sdist at the time was inappropriate for them, but they were happy to be wheel-only). That means that users of that backend are basically unable to upload source, and the likelihood of scenario 2a goes *way* up. Making sure that PEP 517 mandates that backends at least *allow* tools like twine to build a sdist from a source tree significantly alleviates this risk. - Copying a local directory to a temporary area is demonstrably a serious performance issue for pip. We have a number of bugs raised over this. The problem is that we have to blindly copy *everything*, including arbitrarily large amounts of irrelevant data. We wanted to switch to "build a sdist from the local directory, proceed from the "we have a sdist" case. But some uses of setup.py break with that approach. One of the hopes for PEP 517 is that we avoid that problem, because we make it the backend's problem (by asking the backend to tell us how to know what to copy, in one form or another). If there's no support in the PEP for this, we end up having to accept that going forward we'll *still* have this problem (unless we add an extra mandatory backend hook in a new PEP, but backends can still claim support for only PEP 517 and not the new PEP). This is admittedly an enhancement to pip, not essential functionality, but the point of this whole process is to enable the packaging ecosystem (including pip!) to move forward, rather than being paralyzed by the constraints of the old setup.py system. So leaving the same constraints in the new system isn't helpful. - The local directory -> sdist -> wheel -> install model has its issues, but it's a much cleaner model for pip (if we can get to it without breaking use cases) - for all the same reasons that switching from direct installs to installs via wheels was a huge win for us. To implement that model internally, pip would need a means of building a sdist. I hope this helps. If nothing else, it makes your comments look less wordy by comparison :-) Paul From thomas at kluyver.me.uk Thu Jun 1 18:14:01 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 01 Jun 2017 23:14:01 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: <1496355241.1540571.995954864.5BB3C5ED@webmail.messagingengine.com> On Thu, Jun 1, 2017, at 10:49 PM, Paul Moore wrote: > pip also needs a way to deal with "pip install . In > this case, pip (under its current model) copies that directory to a > working area. In that area, it runs the build command to create a > wheel, and proceeds from there. In principle, there's little change in > a PEP 517 world. But again, see below. I still question whether the copying step is necessary for the frontend. Pip does it for setup.py builds (AIUI) because they might modify or create files in the working directory, and it wants to keep the source directory clean of that. Flit can create a wheel without modifying/creating any files in the working directory. So, should PEP 517 specify that the backend building a wheel has to do it without modifying/creating any files in the working directory? If the backend can't be sure it will do that, it should copy whatever it needs to a temporary directory and build from there. Tools falling back to setup.py would copy as part of the fallback build step. This seems to me neater than insisting that the backend copy all its files even if there's no need. As I mentioned on the PR, though, I don't feel strongly about this issue. It's simple enough to copy all the necessary files to another directory if that's what the build API requires. > In a PEP 517 world, > backends may or may not provide a "build a sdist" command (flit > doesn't, and for a long time I believe didn't propose to do so - Flit does as of a few days ago. But it's intended for developers releasing a package, and so it relies on the code being in a VCS repository and the appropriate VCS CLI being available. I don't want this to be required when pip builds a package from source to install. After quite a lot of discussion, I concluded that I want downloading and unpacking an sdist to get me something very close to a fresh VCS checkout of the corresponding release tag. Historically, we've often put the results of things like code-generation into sdists, but I think that was primarily because there was no good way to publish built distributions, and so we hacked sdists into serving a bdist-like function. Now that wheels are widely supported, I'm inclined to discourage doing things like that with sdists. Thomas From richard at python.org Thu Jun 1 18:25:17 2017 From: richard at python.org (Richard Jones) Date: Fri, 2 Jun 2017 08:25:17 +1000 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 03:40, Thomas Kluyver wrote: > On Thu, Jun 1, 2017, at 06:32 PM, Matt Joyce wrote: > There *appear* to be, but I checked several of the names listed there, and > they're not on PyPI: > > https://pypi.python.org/pypi/tkinter > https://pypi.python.org/pypi/memcached > https://pypi.python.org/pypi/vtk > https://pypi.python.org/pypi/python-dev > https://pypi.python.org/pypi/opencv > > So I wonder if the data is fake. Or maybe they were already taken down? Or > the installations are real, but not using those names. > Yes, we had the author take them down, please see https://github.com/pypa/pypi-legacy/issues/644 Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jun 1 18:28:09 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Jun 2017 23:28:09 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496355241.1540571.995954864.5BB3C5ED@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496355241.1540571.995954864.5BB3C5ED@webmail.messagingengine.com> Message-ID: On 1 June 2017 at 23:14, Thomas Kluyver wrote: > On Thu, Jun 1, 2017, at 10:49 PM, Paul Moore wrote: >> pip also needs a way to deal with "pip install . In >> this case, pip (under its current model) copies that directory to a >> working area. In that area, it runs the build command to create a >> wheel, and proceeds from there. In principle, there's little change in >> a PEP 517 world. But again, see below. > > I still question whether the copying step is necessary for the frontend. > Pip does it for setup.py builds (AIUI) because they might modify or > create files in the working directory, and it wants to keep the source > directory clean of that. Flit can create a wheel without > modifying/creating any files in the working directory. That's a very fair comment, and I honestly don't know how critical the copy step is - in the sense that I know we do it to prevent certain classes of issue, but I don't know what they are, or how serious they are. Perhaps Donald does? It's certainly true that setup.py based builds are particularly unpleasant for the obvious "running arbitrary code" reasons. But I'm not sure how happy I am simply saying "backends must ..." what? How would we word this precisely? It's not just about keeping the sources clean, it's also about not being affected by unexpected files in the source directory. Consider that a build using a compiler will have object files somewhere. Should a backend use existing object files in preference to sources? What about a backend based on a tool designed to do precisely that, like waf or make? What if the files came from a build with different compiler flags? Sure, it's user error or a backend bug, but it'll be reported to pip as "I tried to install foo and my program failed when I imported it". We get that sort of bug report routinely (users reporting bugs in build scripts as pip problems) and we'll never have a technical solution to all the ways they can occur, but preventative code like copying the build files to a clean location can minimise them. (As I say, I'm speculating about whether that's actually why we build in a temp location, but it's certainly the sort of thinking that goes into our design). Paul From prometheus235 at gmail.com Thu Jun 1 19:00:47 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Thu, 1 Jun 2017 18:00:47 -0500 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: This issue was also brought up in January at https://github.com/pypa/pypi-legacy/issues/585 then just as after the initial "typosquatting PyPI" report (June 2016) it's met with resounding silence. Attacking the messenger doesn't seem like a winning move from a security standpoint. Can we come up with a plan to address the underlying issue and protect users? Nick On Thu, Jun 1, 2017 at 5:25 PM, Richard Jones wrote: > On 2 June 2017 at 03:40, Thomas Kluyver wrote: > >> On Thu, Jun 1, 2017, at 06:32 PM, Matt Joyce wrote: >> There *appear* to be, but I checked several of the names listed there, >> and they're not on PyPI: >> >> https://pypi.python.org/pypi/tkinter >> https://pypi.python.org/pypi/memcached >> https://pypi.python.org/pypi/vtk >> https://pypi.python.org/pypi/python-dev >> https://pypi.python.org/pypi/opencv >> >> So I wonder if the data is fake. Or maybe they were already taken down? >> Or the installations are real, but not using those names. >> > > Yes, we had the author take them down, please see > https://github.com/pypa/pypi-legacy/issues/644 > > > Richard > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Thu Jun 1 19:11:42 2017 From: richard at python.org (Richard Jones) Date: Fri, 2 Jun 2017 09:11:42 +1000 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 09:00, Nick Timkovich wrote: > This issue was also brought up in January at https://github.com/pypa/pypi- > legacy/issues/585 then just as after the initial "typosquatting PyPI" > report (June 2016) it's met with resounding silence. Attacking the > messenger doesn't seem like a winning move from a security standpoint. > > Can we come up with a plan to address the underlying issue and protect > users? > We haven't yet, but I'm not holding that as proof that we couldn't. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From ja.geb at me.com Thu Jun 1 18:20:27 2017 From: ja.geb at me.com (Jannis Gebauer) Date: Fri, 02 Jun 2017 00:20:27 +0200 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: > This makes me remember https://hackernoon.com/building-a-botnet-on-pypi-be1ad280b8d6 on a related note. Yep, that?s basically the same thing. Instead of using package names of builtins, the attacker is using a combination of popular apt/yum packages with a mix of package names with typos. During development, it?s not uncommon to make mistakes like: pip install requirements.txt (forgot the -r) pip install requestd (typo) pip install tkinter (not registered) Or to use the wrong package manager (apt-get install python-dev vs. pip install python-dev). I wonder if it would make sense to build some kind of blacklist for this. According to the blog post there were close to 10k installs over a period of just three days. I believe Debian is running some kind of popularity contest for their packages which could be used to identify problematic packages. This will be a lot of manual work, but I?d work on a list like this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 1 19:26:09 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 19:26:09 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496355241.1540571.995954864.5BB3C5ED@webmail.messagingengine.com> Message-ID: <2376C0C9-30C9-4AAD-B67A-0F3CCA6E6C47@stufft.io> > On Jun 1, 2017, at 6:28 PM, Paul Moore wrote: > > On 1 June 2017 at 23:14, Thomas Kluyver wrote: >> On Thu, Jun 1, 2017, at 10:49 PM, Paul Moore wrote: >>> pip also needs a way to deal with "pip install . In >>> this case, pip (under its current model) copies that directory to a >>> working area. In that area, it runs the build command to create a >>> wheel, and proceeds from there. In principle, there's little change in >>> a PEP 517 world. But again, see below. >> >> I still question whether the copying step is necessary for the frontend. >> Pip does it for setup.py builds (AIUI) because they might modify or >> create files in the working directory, and it wants to keep the source >> directory clean of that. Flit can create a wheel without >> modifying/creating any files in the working directory. > > That's a very fair comment, and I honestly don't know how critical the > copy step is - in the sense that I know we do it to prevent certain > classes of issue, but I don't know what they are, or how serious they > are. Perhaps Donald does? > > It's certainly true that setup.py based builds are particularly > unpleasant for the obvious "running arbitrary code" reasons. But I'm > not sure how happy I am simply saying "backends must ..." what? How > would we word this precisely? It's not just about keeping the sources > clean, it's also about not being affected by unexpected files in the > source directory. Consider that a build using a compiler will have > object files somewhere. Should a backend use existing object files in > preference to sources? What about a backend based on a tool designed > to do precisely that, like waf or make? What if the files came from a > build with different compiler flags? Sure, it's user error or a > backend bug, but it'll be reported to pip as "I tried to install foo > and my program failed when I imported it". We get that sort of bug > report routinely (users reporting bugs in build scripts as pip > problems) and we'll never have a technical solution to all the ways > they can occur, but preventative code like copying the build files to > a clean location can minimise them. (As I say, I'm speculating about > whether that's actually why we build in a temp location, but it's > certainly the sort of thinking that goes into our design). > > Paul I suspect the original reasoning behind copying to a temporary location has been lost to the sands of time. We?ve been doing that in pip for as long as I?ve worked on pip, maybe Jannis or someone remembers why I dunno! From my end, copying the entire directory alleviates a few problems: * In the current environment, it prevents random debris from cluttering up and being written to the current directory, including build files. * This is important, because not only is it unhygienic to allow random bits of crap to crap all over the local directory, but in the current system the build directories are not sufficiently platform dependent (e.g. a Linux build only gets identified as a Linux build, even if it links against two different ABIs because it was mounted inside of a Debian and a CentOS Docker container). * It reduces errors caused by people/tooling editing files while a build is being processed. This can?t ever be fully removed, but by copying to a temporary location we narrow the window down considerably where someone can inadvertently muck up their build mid progress. * It prevents some issues with two builds running at the same time. Narrowing that down to producing a sdist (or some other mechanism for doing a ?copy what you would need? hook) in addition prevents: * Unexpected files changing the behavior of the build. * Misconfigured build tools appearing to ?work? in development but failing when the sdist is released to PyPI or having the sdist and wheels be different because the wheel was produced from a VCS checkout but a build from a sdist wasn?t. Ultimately you?re right, we could just encode this into PEP 517 and say that projects need to *either* give us a way to copy the files they need OR they need hygienic builds that do not modify the current directory at all. I greatly prefer *not* to do that though, because everyone is only human, and there is likely to be build backends that don?t do that? either purposely or accidentally? and it?ll likely be pip that fields those support issues (because they?ll see it as they invoked pip, so it must be pip?s fault). In my mind the cost of *requiring* some mechanism of doing this is pretty low, the project obviously needs to know what files are important to it or else how is it going to know what it?s going to build in the first place. For most projects the amount of data that *needs* copied (versus is just stuff that is sitting there taking up space) is pretty small, so even on a really slow HDD the copy operating should not be a significant amount of time. It?s also not a particularly hard thing to implement I think? certainly it?s much easier than actually building a project in the first place. There?s a principle here at Amazon that goes, ?Good intentions don?t matter?. Which essentially means that simply saying you?re going to do something good doesn?t count because you?re inevitably going to forget or mess up and that instead of just having the intention to do something, you should have a process in place that ensures it is going to happen. Saying that we?re going to make the copying optional and hope that the build tools correctly build in place without an issue feels like a ?good intention? to me, whereas adding the API and step that *mandates* (through technical means) they do it correctly is putting a process in place that ensures it is going to happen. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From bussonniermatthias at gmail.com Thu Jun 1 19:26:31 2017 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Thu, 1 Jun 2017 16:26:31 -0700 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On Thu, Jun 1, 2017 at 3:20 PM, Jannis Gebauer wrote: > This makes me remember > https://hackernoon.com/building-a-botnet-on-pypi-be1ad280b8d6 on a related > note. > > > Yep, that?s basically the same thing. Instead of using package names of > builtins, the attacker is using a combination of popular apt/yum packages > with a mix of package names with typos. > > During development, it?s not uncommon to make mistakes like: > > pip install requirements.txt (forgot the -r) > pip install requestd (typo) > pip install tkinter (not registered) > > Or to use the wrong package manager (apt-get install python-dev vs. pip > install python-dev). > > I wonder if it would make sense to build some kind of blacklist for this. > According to the blog post there were close to 10k installs over a period of > just three days. I believe Debian is running some kind of popularity contest > for their packages which could be used to identify problematic packages. > This will be a lot of manual work, but I?d work on a list like this. > > Does PyPA have a list of the most 404'ed requests for PyPI ? As pip install `doesnotexists` will get pypi's `pypi.python.org/simple/doesnotexist/` we can likely get a quick idea of what is currently unregistered and could potentially be dangerous. That seem more efficient that trying to guess. -- M From donald at stufft.io Thu Jun 1 19:29:45 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 19:29:45 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: > On Jun 1, 2017, at 6:20 PM, Jannis Gebauer wrote: > >> This makes me remember https://hackernoon.com/building-a-botnet-on-pypi-be1ad280b8d6 on a related note. > > > Yep, that?s basically the same thing. Instead of using package names of builtins, the attacker is using a combination of popular apt/yum packages with a mix of package names with typos. > > During development, it?s not uncommon to make mistakes like: > > pip install requirements.txt (forgot the -r) > pip install requestd (typo) > pip install tkinter (not registered) > > Or to use the wrong package manager (apt-get install python-dev vs. pip install python-dev). > > I wonder if it would make sense to build some kind of blacklist for this. According to the blog post there were close to 10k installs over a period of just three days. I believe Debian is running some kind of popularity contest for their packages which could be used to identify problematic packages. This will be a lot of manual work, but I?d work on a list like this. > > Folks have suggested mining the logs from PyPI looking for 404 results on ``/simple/`` to highlight common packages that are being installed which don?t yet exist, then using that data to inform a sort of automatic blacklist for new project names. Other folks have suggested that trying to use some sort of algorithm with existing names to figure out common typos is a solution. Ultimately the thing that?s missing is someone to spend the time to figure out a good solution and implement it. I will get to it eventually, but if someone feels enthused to make it happen sooner, then their contribution would be appreciated. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubernostrum at gmail.com Thu Jun 1 19:22:24 2017 From: ubernostrum at gmail.com (James Bennett) Date: Thu, 1 Jun 2017 16:22:24 -0700 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: A couple of pieces of prior art: In Django, where we supply a command to let users create a new project or application, we ask the user to supply a name. And then as a quick check before proceeding, attempt to import the supplied name; if the import succeeds, the startapp/startproject command bails out telling you not to shadow an existing Python module. Meanwhile, over in django-registration (a Django app for user-account signups), there's a medium-sized list of "usernames" which are marked as reserved by default, and validators which reject on them to avoid registering some potentially-sensitive names. The list there is all manual, and based on this original list: https://ldpreload.com/blog/names-to-reserve Plus some extra stuff that isn't sensitive as a subdomain/mailbox name but might still be a problem (i.e., if you give users account-profile URLs like /users/, you probably don't want someone coming in as /users/signup). -------------- next part -------------- An HTML attachment was scrubbed... URL: From noah at coderanger.net Thu Jun 1 19:17:29 2017 From: noah at coderanger.net (Noah Kantrowitz) Date: Thu, 1 Jun 2017 16:17:29 -0700 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: <9DCB87DB-8DE7-41B2-B138-140FAE756158@coderanger.net> > On Jun 1, 2017, at 4:00 PM, Nick Timkovich wrote: > > This issue was also brought up in January at https://github.com/pypa/pypi-legacy/issues/585 then just as after the initial "typosquatting PyPI" report (June 2016) it's met with resounding silence. Attacking the messenger doesn't seem like a winning move from a security standpoint. > > Can we come up with a plan to address the underlying issue and protect users? If you have a systemic solution I'm sure we would love to hear it :) --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP URL: From fungi at yuggoth.org Thu Jun 1 19:53:20 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 1 Jun 2017 23:53:20 +0000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: <20170601235320.GL12827@yuggoth.org> On 2017-06-01 20:45:53 +0000 (+0000), Brett Cannon wrote: [...] > I think *twine* is the tool that needs a way to specify how to > produce an sdist. If we want to view twine as the tool to upload > artifacts to PyPI then we need twine to know how to produce sdists > and wheels in a PEP 517 world, not pip. [...] Why do you think that? Because traditionally you could call setup.py to upload an sdist as well as build it? One thing I really like about twine, as the tool I trust with my PyPI creds, is that it's a very simple tool unencumbered by unrelated features. While I agree that the tool which retrieves and installs packages doesn't necessarily also need to be the tool which builds packages, I don't see why the tool which securely uploads packages should take on that function either. In the UNIX sense of doing one thing well, I'd much rather see a separate tool for each of these roles. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From matt at nycresistor.com Thu Jun 1 20:15:59 2017 From: matt at nycresistor.com (Matt Joyce) Date: Thu, 1 Jun 2017 20:15:59 -0400 Subject: [Distutils] Malicious packages on PyPI Message-ID: Or start doing signed pgp for package maintainers and build a transitive trust model. On Jun 1, 2017 8:14 PM, wrote: Force packages to match their higher level import namespace in future major Python versions and PEP it. On Jun 1, 2017 7:37 PM, "Noah Kantrowitz" wrote: > On Jun 1, 2017, at 4:00 PM, Nick Timkovich wrote: > > This issue was also brought up in January at https://github.com/pypa/pypi- legacy/issues/585 then just as after the initial "typosquatting PyPI" report (June 2016) it's met with resounding silence. Attacking the messenger doesn't seem like a winning move from a security standpoint. > > Can we come up with a plan to address the underlying issue and protect users? If you have a systemic solution I'm sure we would love to hear it :) --Noah _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at nycresistor.com Thu Jun 1 20:15:59 2017 From: matt at nycresistor.com (Matt Joyce) Date: Thu, 1 Jun 2017 20:15:59 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: <9DCB87DB-8DE7-41B2-B138-140FAE756158@coderanger.net> References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> <9DCB87DB-8DE7-41B2-B138-140FAE756158@coderanger.net> Message-ID: Force packages to match their higher level import namespace in future major Python versions and PEP it. On Jun 1, 2017 7:37 PM, "Noah Kantrowitz" wrote: > > > On Jun 1, 2017, at 4:00 PM, Nick Timkovich > wrote: > > > > This issue was also brought up in January at > https://github.com/pypa/pypi-legacy/issues/585 then just as after the > initial "typosquatting PyPI" report (June 2016) it's met with resounding > silence. Attacking the messenger doesn't seem like a winning move from a > security standpoint. > > > > Can we come up with a plan to address the underlying issue and protect > users? > > If you have a systemic solution I'm sure we would love to hear it :) > > --Noah > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 1 21:06:59 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 21:06:59 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: Message-ID: > On Jun 1, 2017, at 8:15 PM, Matt Joyce wrote: > > Or start doing signed pgp for package maintainers and build a transitive trust model. > PGP is not useful for our use case except as a generic crypto primitive, and there are better generic crypto primitives out there. See https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 1 21:09:57 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 21:09:57 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <20170601235320.GL12827@yuggoth.org> References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <20170601235320.GL12827@yuggoth.org> Message-ID: > On Jun 1, 2017, at 7:53 PM, Jeremy Stanley wrote: > > On 2017-06-01 20:45:53 +0000 (+0000), Brett Cannon wrote: > [...] >> I think *twine* is the tool that needs a way to specify how to >> produce an sdist. If we want to view twine as the tool to upload >> artifacts to PyPI then we need twine to know how to produce sdists >> and wheels in a PEP 517 world, not pip. > [...] > > Why do you think that? Because traditionally you could call setup.py > to upload an sdist as well as build it? > > One thing I really like about twine, as the tool I trust with my > PyPI creds, is that it's a very simple tool unencumbered by > unrelated features. While I agree that the tool which retrieves and > installs packages doesn't necessarily also need to be the tool which > builds packages, I don't see why the tool which securely uploads > packages should take on that function either. In the UNIX sense of > doing one thing well, I'd much rather see a separate tool for each > of these roles. I think a separate tool for each of these roles is somewhat user unfriendly TBH. Splitting things across multiple projects tends to confuse users and increases the conceptual overhead. I sometimes wonder if we should be folding twine into pip itself, although keeping the split between twine == package authoring tool and pip == package installing tool seems like a reasonable enough divide. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Jun 1 21:40:46 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 2 Jun 2017 01:40:46 +0000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <20170601235320.GL12827@yuggoth.org> Message-ID: <20170602014045.GM12827@yuggoth.org> On 2017-06-01 21:09:57 -0400 (-0400), Donald Stufft wrote: [...] > I think a separate tool for each of these roles is somewhat user > unfriendly TBH. [...] I'll do my best not to be offended that you don't consider me a user (or representative of some broader class of users). ;) At any rate, I think it depends on your definition of users. Some users want one shiny kitchen-sink tool that does everything for them, others want composable tools with well-considered bounds of operation. It's possible a modular approach could satisfy both, but then again if twine grows too many features I'm just as likely to write a new lightweight API client instead so I can have something auditable I can trust my credentials to which only knows how to upload. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From glyph at twistedmatrix.com Thu Jun 1 21:35:42 2017 From: glyph at twistedmatrix.com (Glyph) Date: Thu, 1 Jun 2017 18:35:42 -0700 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <20170601235320.GL12827@yuggoth.org> Message-ID: > On Jun 1, 2017, at 6:09 PM, Donald Stufft wrote: > > I sometimes wonder if we should be folding twine into pip itself Yes please. WTB `pip upload`. -g -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 1 21:55:21 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 1 Jun 2017 21:55:21 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <20170602014045.GM12827@yuggoth.org> References: <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <20170601235320.GL12827@yuggoth.org> <20170602014045.GM12827@yuggoth.org> Message-ID: > On Jun 1, 2017, at 9:40 PM, Jeremy Stanley wrote: > > On 2017-06-01 21:09:57 -0400 (-0400), Donald Stufft wrote: > [...] >> I think a separate tool for each of these roles is somewhat user >> unfriendly TBH. > [...] > > I'll do my best not to be offended that you don't consider me a user > (or representative of some broader class of users). ;) I probably should have written out the long form: unfriendly to users who aren?t steeped in packaging lore ;) > > At any rate, I think it depends on your definition of users. Some > users want one shiny kitchen-sink tool that does everything for > them, others want composable tools with well-considered bounds of > operation. It's possible a modular approach could satisfy both, but > then again if twine grows too many features I'm just as likely to > write a new lightweight API client instead so I can have something > auditable I can trust my credentials to which only knows how to > upload. > Largely to me it?s about not throwing a ton of different things at people that they have to both find and learn. It?s easier to keep things consistent in a single code base (lol Unix which has -R and -r for recursive depending on your tool!) and also easier for people to discover the different commands they need to fully manage a project. This can get particularly difficult when the multitude of different tools evolve at different paces (we see this today where pip will support something but setuptools won?t yet, etc) which requires people to have to care about the versions of more different tools. I also think it?s perfectly fine to have another tool that competes with twine (or part of twine) that takes a different set of trade offs. Part of the goals of documenting standards around these things instead of just going ?well setuptools is the thing you need to use and that?s just it? you can go ahead and write your thing that scratches your particular itch better, and they can have a friendly competition ;). At the end of the day though, this is a bit of a tangent since it doesn?t matter wether it?s ``pip sdist``, ``twine sdist``, or ``make-me-and-sdist-plz``, the underlying point of having a command to handle that stands. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at nycresistor.com Thu Jun 1 22:33:52 2017 From: matt at nycresistor.com (Matt Joyce) Date: Thu, 1 Jun 2017 22:33:52 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: Message-ID: I was more pushing for the transitive trust element than signing. That being said, any signing at all would be progress. On Jun 1, 2017 9:07 PM, "Donald Stufft" wrote: On Jun 1, 2017, at 8:15 PM, Matt Joyce wrote: Or start doing signed pgp for package maintainers and build a transitive trust model. PGP is not useful for our use case except as a generic crypto primitive, and there are better generic crypto primitives out there. See https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Fri Jun 2 00:04:12 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Thu, 1 Jun 2017 23:04:12 -0500 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: I suggested on one of those issues to try to auto-blacklist common 404s as that should pose a negligible usability hit. I'd like to start by logging them to collect data, but I'm confused nowadays as to if that should go into pypa/warehouse or pypa/pypi-legacy. How long until warehouse is where most requests go, or do some go there right now, but from which clients...so confuz, plz halp. On Thu, Jun 1, 2017 at 6:29 PM, Donald Stufft wrote: > > On Jun 1, 2017, at 6:20 PM, Jannis Gebauer wrote: > > This makes me remember https://hackernoon.com/building-a-botnet-on-pypi- > be1ad280b8d6 on a related note. > > > Yep, that?s basically the same thing. Instead of using package names of > builtins, the attacker is using a combination of popular apt/yum packages > with a mix of package names with typos. > > During development, it?s not uncommon to make mistakes like: > > pip install requirements.txt (forgot the -r) > pip install requestd (typo) > pip install tkinter (not registered) > > Or to use the wrong package manager (apt-get install python-dev vs. pip > install python-dev). > > I wonder if it would make sense to build some kind of blacklist for this. > According to the blog post there were close to 10k installs over a period > of just three days. I believe Debian is running some kind of popularity > contest for their packages which could be used to identify problematic > packages. This will be a lot of manual work, but I?d work on a list like > this. > > > > > Folks have suggested mining the logs from PyPI looking for 404 results on > ``/simple/`` to highlight common packages that are being installed which > don?t yet exist, then using that data to inform a sort of automatic > blacklist for new project names. > > Other folks have suggested that trying to use some sort of algorithm with > existing names to figure out common typos is a solution. > > Ultimately the thing that?s missing is someone to spend the time to figure > out a good solution and implement it. I will get to it eventually, but if > someone feels enthused to make it happen sooner, then their contribution > would be appreciated. > > ? > Donald Stufft > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 2 03:49:35 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Jun 2017 17:49:35 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 06:45, Brett Cannon wrote: > And so I think in a very wordy way, I just said we need to stop saying "pip > needs a standardized way to produce an sdist" and instead start saying > "twine needs a way to produce an sdist". And that leads to the question > about whether PEP 517 must cover sdist production for twine because we want > to have that solved before we have the consumption side in pip in place. Or > put another way, are we okay with pip consuming things that twine simply > can't make from a pyproject.toml file (yet)? A "yes" means PEP 517 shouldn't > be held up, while a "no" means we need a solution for twine. I largely agree with this phrasing of the problem, but think it's oversimplifying the underlying capabilities needed a bit: 1. sdist creation tools, whether tox, twine, a future "pip publish" command, or something else, need to know how to: 1a. Generate sdist metadata (aka PKG-INFO) 1b. Generate the actual sdist contents 1c. Determine the dependencies needed to handle the above steps 2. Install tools, including pip, need to know how to: 2a. Generate a clean out-of-tree build directory 2b. Generate wheel metadata (including the METADATA file) 2c. Generate the actual wheel contents 2d. Determine the dependencies needed to handle the above steps The agreed-to-be-missing piece that has been identified in PEP 517 is "2a": the part that will let pip (and any other tools doing out-of-tree builds) avoid copying entire VCS checkouts into the build directory. Donald's perspective is that instead of being an independent step, generating an out-of-tree build directory should instead be defined in terms of sdist generation on the following grounds: * it better accounts for local publish-and-install tools like `tox`, which relies on sdist generation to push the code under development into the test venv * defining the "export a build tree" and "generate an sdist tree" steps independently creates the opportunity for future inconsistencies where "VCS -> sdist -> build tree -> wheel" and "VCS -> build tree -> wheel" give different results (just as "pip install project-dir/" and "pip install project-sdist" can give different results today) * `PKG-INFO` in an sdist is essentially the same file as PEP 427's `METADATA`, so any wheel building backend is already required to be able to generate it While the first point doesn't really bother me (I'm OK with folks needing to keep a "setup.py sdist" shim around for now), I have to agree that I find the second point compelling, as it means that any PEP 517 based project will inherently test the correctness of its sdist generation *just by doing a local install*. That means we won't be relying on opt-in pre-release testing with tox, or post-release bug reports from users any more: if the sdist definition is broken, even local installations from the VCS checkout into an active virtual environment won't work. The last point means that requiring an sdist export command shouldn't impose an unreasonable burden on backend developers, as the only differences between "generate an sdist tree" and "export a build tree" would be: - `pyproject.toml` MUST be included unmodified at the root of the sdist tree - `PKG-INFO` MUST be generated at the root of the sdist tree - a `setup.py` shim MAY be generated for backwards compatibility with installation tools that are unaware of PEP 517 Beyond that, both approaches would have the same requirement of "include everything needed to subsequently build a wheel file from the sdist". The mapping back to the originally identified activities would then be: 1. sdist creation tools: 1a. Generate sdist metadata: export the sdist and read PKG-INFO 1b. Generate the actual sdist contents: export the sdist 1c. Determine the dependencies needed to handle the above steps: look at build-system.requires 2. Install tools: 2a. Generate a clean out-of-tree build directory: export the sdist 2b. Generate wheel metadata: export the wheel metadata directory 2c. Generate the actual wheel contents: build the wheel file 2d. Determine the dependencies needed to handle the above steps: look at build-system.requires & get_build_requires() So even though 1a, 1b, and 2a are conceptually different activities, backends would only need to implement one operation to handle all 3: sdist export. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jun 2 04:05:33 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Jun 2017 18:05:33 +1000 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 09:00, Nick Timkovich wrote: > This issue was also brought up in January at > https://github.com/pypa/pypi-legacy/issues/585 then just as after the > initial "typosquatting PyPI" report (June 2016) it's met with resounding > silence. Attacking the messenger doesn't seem like a winning move from a > security standpoint. > > Can we come up with a plan to address the underlying issue and protect > users? I like the suggestion of an auto-generated "common 404" blacklist, where regularly queried-but-nonexistent names can't be registered without prior approval by the PyPI admins or the PSF. Beyond that, one of the biggest challenges we face with the status quo is that it's mainly perceived by commercial redistributors as an opportunity to sell people security scanning and component whitelisting tools, rather than as a shared ecosystem health management problem to be addressed collectively :( Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Jun 2 04:37:31 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Jun 2017 09:37:31 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 08:49, Nick Coghlan wrote: > The last point means that requiring an sdist export command shouldn't > impose an unreasonable burden on backend developers, as the only > differences between "generate an sdist tree" and "export a build tree" > would be: > > - `pyproject.toml` MUST be included unmodified at the root of the sdist tree > - `PKG-INFO` MUST be generated at the root of the sdist tree > - a `setup.py` shim MAY be generated for backwards compatibility with > installation tools that are unaware of PEP 517 Note that a full "build an sdist" process should include a means for authors to add extra files (such as README, LICENSE, ...). But that can be either something that backends deal with themselves or (better) gets standardised in a separate PEP (probably defining a new set of `pyproject.toml` fields for it). It's not inherently something we need right now in PEP 517. Paul From richard at python.org Fri Jun 2 05:42:59 2017 From: richard at python.org (Richard Jones) Date: Fri, 2 Jun 2017 19:42:59 +1000 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 18:05, Nick Coghlan wrote: > On 2 June 2017 at 09:00, Nick Timkovich wrote: > > This issue was also brought up in January at > > https://github.com/pypa/pypi-legacy/issues/585 then just as after the > > initial "typosquatting PyPI" report (June 2016) it's met with resounding > > silence. Attacking the messenger doesn't seem like a winning move from a > > security standpoint. > > > > Can we come up with a plan to address the underlying issue and protect > > users? > > I like the suggestion of an auto-generated "common 404" blacklist, > where regularly queried-but-nonexistent names can't be registered > without prior approval by the PyPI admins or the PSF. > I like it also, but it adds an additional administration burden on top of that which is not being coped with at the moment. 117 open issues in https://github.com/pypa/pypi-legacy/issues 219 open support tickets in https://sourceforge.net/p/pypi/support-requests/ Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Jun 2 08:25:45 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 2 Jun 2017 07:25:45 -0500 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> <9DCB87DB-8DE7-41B2-B138-140FAE756158@coderanger.net> Message-ID: On Thursday, June 1, 2017, Matt Joyce wrote: > Force packages to match their higher level import namespace in future > major Python versions and PEP it. > __import__('siht'[::-1]) Though static analysis would still be great. > > On Jun 1, 2017 7:37 PM, "Noah Kantrowitz" > wrote: > >> >> > On Jun 1, 2017, at 4:00 PM, Nick Timkovich > > wrote: >> > >> > This issue was also brought up in January at >> https://github.com/pypa/pypi-legacy/issues/585 then just as after the >> initial "typosquatting PyPI" report (June 2016) it's met with resounding >> silence. Attacking the messenger doesn't seem like a winning move from a >> security standpoint. >> > >> > Can we come up with a plan to address the underlying issue and protect >> users? >> >> If you have a systemic solution I'm sure we would love to hear it :) >> >> --Noah >> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 2 08:52:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Jun 2017 22:52:20 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 18:37, Paul Moore wrote: > On 2 June 2017 at 08:49, Nick Coghlan wrote: >> The last point means that requiring an sdist export command shouldn't >> impose an unreasonable burden on backend developers, as the only >> differences between "generate an sdist tree" and "export a build tree" >> would be: >> >> - `pyproject.toml` MUST be included unmodified at the root of the sdist tree >> - `PKG-INFO` MUST be generated at the root of the sdist tree >> - a `setup.py` shim MAY be generated for backwards compatibility with >> installation tools that are unaware of PEP 517 > > Note that a full "build an sdist" process should include a means for > authors to add extra files (such as README, LICENSE, ...). But that > can be either something that backends deal with themselves or (better) > gets standardised in a separate PEP (probably defining a new set of > `pyproject.toml` fields for it). It's not inherently something we need > right now in PEP 517. I think we can leave this as a backend level thing, as it really is entirely up to the backend. From a frontend's perspective, "Recursively copy everything that doesn't start with '.' and then generate PKG-INFO" would be an entirely acceptable sdist export implementation, as would "Export all files that are under source control and then generate PKG-INFO". More complex backends may want more sophisticated options than that, but they can still go in a backend dependent configuration file (ala MANIFEST.in for distutils/setuptools) rather than needing to be standardised. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jun 2 09:05:53 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Jun 2017 23:05:53 +1000 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 19:42, Richard Jones wrote: > On 2 June 2017 at 18:05, Nick Coghlan wrote: >> >> On 2 June 2017 at 09:00, Nick Timkovich wrote: >> > This issue was also brought up in January at >> > https://github.com/pypa/pypi-legacy/issues/585 then just as after the >> > initial "typosquatting PyPI" report (June 2016) it's met with resounding >> > silence. Attacking the messenger doesn't seem like a winning move from a >> > security standpoint. >> > >> > Can we come up with a plan to address the underlying issue and protect >> > users? >> >> I like the suggestion of an auto-generated "common 404" blacklist, >> where regularly queried-but-nonexistent names can't be registered >> without prior approval by the PyPI admins or the PSF. > > I like it also, but it adds an additional administration burden on top of > that which is not being coped with at the moment. > > 117 open issues in https://github.com/pypa/pypi-legacy/issues > 219 open support tickets in https://sourceforge.net/p/pypi/support-requests/ Right, to be even remotely viable, any approach would need to be almost entirely automated, and even then, we'd anticipate a potential uptick in support requests asking for particular names to be unblocked. In the meantime, I'm OK with our official answer being "This is why suppliers of component whitelisting and security scanning systems currently have a viable business model - nobody has figured out how to do this sustainably at PyPI's scale with purely volunteer effort, and the PSF's own finances aren't yet in good enough shape to fund it directly on behalf of the wider community". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Fri Jun 2 09:42:30 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 02 Jun 2017 14:42:30 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> Message-ID: <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> As was suggested at some point, I have added a build_sdist hook to my PR, with the following details: - A brief definition of the minimal requirements of an sdist. - I have limited the definition to gzipped tarballs. Zip files also work as sdists, but we're moving towards standardising on tarballs, so I think it's simplest to require that of PEP-517 compliant tools. - The build_sdist hook must be defined, but may not always work (e.g. it may depend on a VCS) - The prepare_build_files hook is optional, and in its absence, frontends can use build_sdist and extract the files to create a build directory. - Backends (like flit) where building an sdist has extra requirements should define prepare_build_files. https://github.com/python/peps/pull/277/files From prometheus235 at gmail.com Fri Jun 2 09:43:30 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Fri, 2 Jun 2017 08:43:30 -0500 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: For a first few passes, if the 404-blacklist is sufficiently lax (strict?), only *extremely* common mistypes like "isx" or "requjests" should be caught. The slow response time would then be good as it would force users to think long and hard about if they really want such strange names and/or make some lazy malicious users give up. ;) On Fri, Jun 2, 2017 at 8:05 AM, Nick Coghlan wrote: > On 2 June 2017 at 19:42, Richard Jones wrote: > > On 2 June 2017 at 18:05, Nick Coghlan wrote: > >> > >> On 2 June 2017 at 09:00, Nick Timkovich > wrote: > >> > This issue was also brought up in January at > >> > https://github.com/pypa/pypi-legacy/issues/585 then just as after the > >> > initial "typosquatting PyPI" report (June 2016) it's met with > resounding > >> > silence. Attacking the messenger doesn't seem like a winning move > from a > >> > security standpoint. > >> > > >> > Can we come up with a plan to address the underlying issue and protect > >> > users? > >> > >> I like the suggestion of an auto-generated "common 404" blacklist, > >> where regularly queried-but-nonexistent names can't be registered > >> without prior approval by the PyPI admins or the PSF. > > > > I like it also, but it adds an additional administration burden on top of > > that which is not being coped with at the moment. > > > > 117 open issues in https://github.com/pypa/pypi-legacy/issues > > 219 open support tickets in https://sourceforge.net/p/ > pypi/support-requests/ > > Right, to be even remotely viable, any approach would need to be > almost entirely automated, and even then, we'd anticipate a potential > uptick in support requests asking for particular names to be > unblocked. > > In the meantime, I'm OK with our official answer being "This is why > suppliers of component whitelisting and security scanning systems > currently have a viable business model - nobody has figured out how to > do this sustainably at PyPI's scale with purely volunteer effort, and > the PSF's own finances aren't yet in good enough shape to fund it > directly on behalf of the wider community". > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Jun 2 10:12:19 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Jun 2017 15:12:19 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 14:42, Thomas Kluyver wrote: > - Backends (like flit) where building an sdist has extra requirements > should define prepare_build_files. I'm struggling to understand why building a sdist in flit should need a VCS. It bothers me that I'm missing something important about how backends will work, that explains why (for example) you can't create a sdist from an export of a VCS branch (i.e., without the VCS metadata). Can you provide a pointer to the docs on flit's "build a sdist" command, that explains the limitations? (I gather that this is in development, so a pointer to the doc files in VCS is fine). Paul From thomas at kluyver.me.uk Fri Jun 2 10:38:52 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 02 Jun 2017 15:38:52 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> Message-ID: <1496414332.1378910.996665408.6C939F0D@webmail.messagingengine.com> On Fri, Jun 2, 2017, at 03:12 PM, Paul Moore wrote: > I'm struggling to understand why building a sdist in flit should need > a VCS. It bothers me that I'm missing something important about how > backends will work, that explains why (for example) you can't create a > sdist from an export of a VCS branch (i.e., without the VCS metadata). It's the best way I've found to answer the question of which files should go in an sdist. The other things that we don't want to do include: 1. Get only the files needed to install and use the library, as we do for a wheel. Bad because people expect sdists to include things like docs and tests (if you put tests outside the package). 2. Tar up the whole directory containing flit.ini (/pyproject.toml). Bad because this will include things like built docs, VCS metadata, and random files you've made, so the sdist will be much bigger than necessary. 3. Hard-coded blacklist/whitelist. Not flexible enough - we can't cover all the different ways you might do docs, for instance. 4. Configurable blacklist/whitelist. This is what MANIFEST.in provides. I think we could come up with a more memorable syntax than MANIFEST.in - probably something like gitignore - but I'm not keen on adding back another boilerplate file. And the big problem I have with MANIFEST.in is that it's easy to forget to update it when you add some files that need to be in the sdist. I think the key realisation for me was that the files I want in an sdist are the same set of files in a fresh checkout of the VCS repo. I want it to be a static snapshot of what was in my VCS when I released (plus, for the sake of other tools, a couple of generated files). So the necessary information to make the sdist is there in the VCS. > Can you provide a pointer to the docs on flit's "build a sdist" > command, that explains the limitations? (I gather that this is in > development, so a pointer to the doc files in VCS is fine). I appreciate your optimism about my docs. ;-) From ncoghlan at gmail.com Fri Jun 2 10:41:23 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 00:41:23 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> Message-ID: On 2 June 2017 at 23:42, Thomas Kluyver wrote: > As was suggested at some point, I have added a build_sdist hook to my > PR, with the following details: > > - A brief definition of the minimal requirements of an sdist. > - I have limited the definition to gzipped tarballs. Zip files also > work as sdists, but we're moving towards standardising on tarballs, so > I think it's simplest to require that of PEP-517 compliant tools. For the sdist case, I'd prefer to leave the actual archive creation in the hands of the frontend as far as the plugin API is concerned. That lets us completely duck the fact that the sdist naming scheme and exact archive format aren't formally defined anywhere, and for pip's local build use case, we want the unpacked tree anyway. In a lot of ways, it's closer in spirit to the wheel metadata generation hook than it is to the wheel building hook. > - The build_sdist hook must be defined, but may not always work (e.g. it > may depend on a VCS) I was going to object to this aspect, but I realised there's a clear marker file that frontends can use to determine if they're working with an already exported sdist tree: PKG-INFO That means the invocation protocol for the additional hook can be: - if PKG-INFO is present, then just copy the full contents of the directory without invoking the backend's sdist export hook - if PKG-INFO is *not* present, then invoke the backend's sdist export hook to do a filtered export that at least omits any VCS bookkeeping files > - The prepare_build_files hook is optional, and in its absence, > frontends can use build_sdist and extract the files to create a build > directory. > - Backends (like flit) where building an sdist has extra requirements > should define prepare_build_files. Having two hooks still leaves us open to "VCS -> sdist -> build tree -> wheel" and "VCS -> build tree -> wheel" giving different answers, and that's specifically the loophole we're aiming to close by including this in PEP 517 rather than leaving it until later. Instead, the flow that I think makes sense is "VCS -> sdist tree [-> sdist tree -> sdist tree -> ...] -> wheel", and the above model where the export filtering is only used when PKG-INFO doesn't exist yet will give us that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Fri Jun 2 10:56:51 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 02 Jun 2017 15:56:51 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> Message-ID: <1496415411.1384073.996691024.69B5CA2E@webmail.messagingengine.com> On Fri, Jun 2, 2017, at 03:41 PM, Nick Coghlan wrote: > Instead, the flow that I think makes sense is "VCS -> sdist tree [-> > sdist tree -> sdist tree -> ...] -> wheel", and the above model where > the export filtering is only used when PKG-INFO doesn't exist yet will > give us that. I still object to conflating 'filter the files needed to build a wheel' with 'build an sdist' - these are different tasks which I would implement differently. And flit cannot do (sdist tree -> sdist tree). The options as I see them: 1. Make it the responsibility of the backend, not the frontend, to build cleanly (except for setup.py builds). Then there's no need for a hook to filter a build tree before building a wheel. 2. Define a hook to filter the files into a build tree, as a separate notion from building sdists. Thomas From donald at stufft.io Fri Jun 2 11:27:11 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 11:27:11 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> Message-ID: <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> > On Jun 2, 2017, at 10:41 AM, Nick Coghlan wrote: > > On 2 June 2017 at 23:42, Thomas Kluyver wrote: >> As was suggested at some point, I have added a build_sdist hook to my >> PR, with the following details: >> >> - A brief definition of the minimal requirements of an sdist. >> - I have limited the definition to gzipped tarballs. Zip files also >> work as sdists, but we're moving towards standardising on tarballs, so >> I think it's simplest to require that of PEP-517 compliant tools. > > For the sdist case, I'd prefer to leave the actual archive creation in > the hands of the frontend as far as the plugin API is concerned. That > lets us completely duck the fact that the sdist naming scheme and > exact archive format aren't formally defined anywhere, and for pip's > local build use case, we want the unpacked tree anyway. > > In a lot of ways, it's closer in spirit to the wheel metadata > generation hook than it is to the wheel building hook. I?d prefer to leave the actual creation of the archive up to the front end in both the sdist and the wheel case. It will allow some cases of pip to avoid having to round trip through a compression -> decompression cycle and can instead just use it directly. Other cases it won?t allow that, but in those cases it doesn?t really add any more time or complexity, it just shifts the time spent compressing from the backend to the frontend. > >> - The build_sdist hook must be defined, but may not always work (e.g. it >> may depend on a VCS) > > I was going to object to this aspect, but I realised there's a clear > marker file that frontends can use to determine if they're working > with an already exported sdist tree: PKG-INFO > > That means the invocation protocol for the additional hook can be: > > - if PKG-INFO is present, then just copy the full contents of the > directory without invoking the backend's sdist export hook > - if PKG-INFO is *not* present, then invoke the backend's sdist export > hook to do a filtered export that at least omits any VCS bookkeeping > files > >> - The prepare_build_files hook is optional, and in its absence, >> frontends can use build_sdist and extract the files to create a build >> directory. >> - Backends (like flit) where building an sdist has extra requirements >> should define prepare_build_files. > > Having two hooks still leaves us open to "VCS -> sdist -> build tree > -> wheel" and "VCS -> build tree -> wheel" giving different answers, > and that's specifically the loophole we're aiming to close by > including this in PEP 517 rather than leaving it until later. > > Instead, the flow that I think makes sense is "VCS -> sdist tree [-> > sdist tree -> sdist tree -> ...] -> wheel", and the above model where > the export filtering is only used when PKG-INFO doesn't exist yet will > give us that. > So my preference is that everything goes through the sdist step as I think that is most likely to provide consistent builds everywhere both from a VCS checkout and from a sdist that was released to PyPI. That being said, I am somewhat sympathetic to the idea that generating a sdist might be a slow process for reasons that are unrelated to actually building a wheel (for example, documentation might get ?compiled? from some kind of source format to a man page, html docs, etc) so I think I am not against the idea of having an optional hook whose job is to just do the copying needed. The requirements would be: * The build_sdist hook is mandatory, but may fail (as any of these commands may fail tbh) if some invariant required by the build backend isn?t satisfied. * The copy_the_files hook is optional, if it exists it SHOULD produce a tree that when the build_wheel hook is called in it, will produce a wheel that is equivalent to one that would have been built had the build_sdist hook been called instead. * If the copy_the_files hook is not defined, then the build frontend is free to just directory call the build_sdist command instead. I think that represents a pretty reasonable trade off, the path of least resistance for a build backend is to just define build_sdist and build_wheel and leave the two optional hooks omitted. I suspect for a lot of pure python packages (although Thomas has said not flit) those two hooks will be fast enough that is all they?ll need to implement. However in cases they?re not we provide both the copy_the_files and the wheel_metadata hook to allow short circuiting a possibly more complex build process to provide a better UX to end users. That kinds of goes against my ?good intentions don?t matter? statement from before, but I also think that practicality beats purity ;) ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Fri Jun 2 12:24:57 2017 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 2 Jun 2017 16:24:57 +0000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> Message-ID: <20170602162457.GN12827@yuggoth.org> On 2017-06-02 15:12:19 +0100 (+0100), Paul Moore wrote: [...] > I'm struggling to understand why building a sdist in flit should need > a VCS. It bothers me that I'm missing something important about how > backends will work, that explains why (for example) you can't create a > sdist from an export of a VCS branch (i.e., without the VCS metadata). [...] Unrelated to flit, but I have similar needs to be able to make sure my sdist version has a 1:1 correspondence to the name of a release tag in my VCS. Making a commit to embed a version number in a file in the VCS and then tagging that commit with the same version number is 1. racy and 2. duplication of information which can (and frequently has) led to confusing mistakes. As a result, I either need the VCS metadata present to be able to construct the version number which will get included in PKG-INFO _or_ I need a complex build wrapper which extracts the metadata prior to the filtered tree copy happening and plumbs it through (with an envvar or maybe spontaneously generating a file on disk) so that the sdist builder will know what version to embed. Present day, this works fine as a setuptools entrypoint which can inspect the VCS metadata at sdist creation time. It would be unfortunate to lose such flexibility in whatever hook implementation we end up with. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 949 bytes Desc: Digital signature URL: From ncoghlan at gmail.com Fri Jun 2 12:26:33 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 02:26:33 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496415411.1384073.996691024.69B5CA2E@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <1496415411.1384073.996691024.69B5CA2E@webmail.messagingengine.com> Message-ID: [Note: I've reverted the PEP to Draft status while this discussion is ongoing: https://github.com/python/peps/blob/master/pep-0517.txt] On 3 June 2017 at 00:56, Thomas Kluyver wrote: > On Fri, Jun 2, 2017, at 03:41 PM, Nick Coghlan wrote: >> Instead, the flow that I think makes sense is "VCS -> sdist tree [-> >> sdist tree -> sdist tree -> ...] -> wheel", and the above model where >> the export filtering is only used when PKG-INFO doesn't exist yet will >> give us that. > > I still object to conflating 'filter the files needed to build a wheel' > with 'build an sdist' - these are different tasks which I would > implement differently This concerns me somewhat, as if a backend implements the two differently, then it means building from an sdist and building from a VCS checkout may give different results (since they may contain different files). Could you provide a little more detail as to what you would do differently in exporting the contents of an sdist that wouldn't apply to export a build tree? (aside from skipping emitting PKG-INFO) >. And flit cannot do (sdist tree -> sdist tree). It wouldn't be required to - since a PKG-INFO would be present in that case, the front end would just copy the directory without bothering the backend about it. > The options as I see them: > > 1. Make it the responsibility of the backend, not the frontend, to build > cleanly (except for setup.py builds). Then there's no need for a hook to > filter a build tree before building a wheel. No, we're not going to do that - build isolation will be the frontend's responsibility. > 2. Define a hook to filter the files into a build tree, as a separate > notion from building sdists. While Donald seems more amenable to this now, I still don't understand the difference you see between the two (aside from PKG-INFO potentially being unneeded in the build tree case, depending on how the backend handles creation of the METADATA file) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Jun 2 12:39:29 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Jun 2017 17:39:29 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: On 2 June 2017 at 16:27, Donald Stufft wrote: > So my preference is that everything goes through the sdist step as I think > that is most likely to provide consistent builds everywhere both from a VCS > checkout and from a sdist that was released to PyPI. Agreed. That's the ideal workflow. The only reason we don't do it now is because... well, I'm not quite sure. I think it's to do with things like setuptools_scm not generating suitable "temporary version numbers" to allow us to work properly with installs that assume that name/version uniquely identifies the code. But regardless, I'd like to say that under PEP 517, we will go source tree -> sdist -> wheel -> install, for everything but editable installs. If projects like setuptools_scm will have an issue with this, they need to feed their requirements into the process of agreeing PEP 517 - the pip developers can't really argue their case for them. > That being said, I am > somewhat sympathetic to the idea that generating a sdist might be a slow > process for reasons that are unrelated to actually building a wheel (for > example, documentation might get ?compiled? from some kind of source format > to a man page, html docs, etc) so I think I am not against the idea of > having an optional hook whose job is to just do the copying needed. The > requirements would be: Agreed. Optimising the process is perfectly OK with me, but I do think we should treat it as just that, an optimisation, and require that backends implementing the optimisation route must ensure that it gives the same results as going via a sdist. Note that there's an implication here - if we define the build process in terms of the effect of "going via a sdist", then we need to at least have an intuitive understanding of what that means in practice. I don't think it's a contentious point (even if the specific term "sdist" is open to debate), as I think repeatable builds are a well-understood idea. (It's at this point that the concerns of people who want incremental builds come in - we should support incremental builds in a way that preserves the "just like going via a sdist" principle. But again, they need to raise their concerns if they think we're missing something key to their use case). > * The build_sdist hook is mandatory, but may fail (as any of these commands > may fail tbh) if some invariant required by the build backend isn?t > satisfied. Agreed. Other tools (or future additions to pip), that want to provide a common interface to the "build a sdist" functionality would use this hook too. They may not be able to fall back in the same way as pip can, but that's their issue to address. > * The copy_the_files hook is optional, if it exists it SHOULD produce a tree > that when the build_wheel hook is called in it, will produce a wheel that is > equivalent to one that would have been built had the build_sdist hook been > called instead. This is precisely the "should look like we built a sdist" principle, so I'm a solid +1 on this, too. It might be worth stating that copy_the_files is only intended to be called after a failed call to build_sdist. I don't know if backends would care, but I don't think we should worry about having to support use of copy_the_files as anything other than a build_sdist fallback. > * If the copy_the_files hook is not defined, then the build frontend is free > to just directory call the build_sdist command instead. Sorry? I assume here that you mean "directly call the build_wheel hook in the original source tree"? That's OK, but I think we should be clear that if this happens, it is the backend's responsibility to ensure that the build is equivalent to building from a sdist. It might even be appropriate for the front end to warn if this happens - "Unable to build out of tree - results may differ from a clean build" (The intent is to remind people that they aren't testing the actual sdist they will be deploying, the issue that Nick pointed out). One thing that is worth flagging here, if only as a note for backends considering this approach, is that the source tree could have arbitrary out of date build artifacts in it (from previous build_wheel calls, possibly with different settings, or from the source tree being installed editable, or even from the user doing something like a manual debug build) and the backend must take responsibility for ensuring that those artifacts don't affect the result. (In case it's not obvious, my personal feeling is that this is a pretty risky option, and I'd strongly prefer backends that implement at least one of the hooks allowing out-of-tree builds). > I think that represents a pretty reasonable trade off, the path of least > resistance for a build backend is to just define build_sdist and build_wheel > and leave the two optional hooks omitted. I suspect for a lot of pure python > packages (although Thomas has said not flit) those two hooks will be fast > enough that is all they?ll need to implement. However in cases they?re not > we provide both the copy_the_files and the wheel_metadata hook to allow > short circuiting a possibly more complex build process to provide a better > UX to end users. That kinds of goes against my ?good intentions don?t > matter? statement from before, but I also think that practicality beats > purity ;) Agreed, this is looking reasonable to me. I think it covers pip's requirements, and I hope it addresses Thomas' needs for flit. I don't honestly have a feel for what other backends might look like, so I'll leave other people to comment on those. Paul From thomas at kluyver.me.uk Fri Jun 2 12:50:14 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 02 Jun 2017 17:50:14 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <1496415411.1384073.996691024.69B5CA2E@webmail.messagingengine.com> Message-ID: <1496422214.2364627.996821928.528A70BF@webmail.messagingengine.com> On Fri, Jun 2, 2017, at 05:26 PM, Nick Coghlan wrote: > Could you provide a little more detail as to what you would do > differently in exporting the contents of an sdist that wouldn't apply > to export a build tree? (aside from skipping emitting PKG-INFO) When creating an sdist, I query the VCS to work out what files to put in it. This is brittle - it depends on the directory being a VCS checkout, and the relevant VCS being available to call. And it's relatively slow, because we have to shell out to another process. I've decided those are acceptable trade-offs for the project maintainer making the release. When exporting a build tree, I would copy only the files that are needed to make the wheel. This is simple, robust and fast. I can even generate PKG-INFO when exporting a build tree if that helps. But I want to keep the idea of a build tree used as an intermediate to generating a wheel separate from that of an sdist, which is a release artifact. Thomas From donald at stufft.io Fri Jun 2 13:02:00 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 13:02:00 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: <843785FA-EAA1-450F-8005-FCC2DA513F4F@stufft.io> > On Jun 2, 2017, at 12:39 PM, Paul Moore wrote: > > On 2 June 2017 at 16:27, Donald Stufft > wrote: >> So my preference is that everything goes through the sdist step as I think >> that is most likely to provide consistent builds everywhere both from a VCS >> checkout and from a sdist that was released to PyPI. > > Agreed. That's the ideal workflow. The only reason we don't do it now > is because... well, I'm not quite sure. I think it's to do with things > like setuptools_scm not generating suitable "temporary version > numbers" to allow us to work properly with installs that assume that > name/version uniquely identifies the code. I?m pretty sure the only reason we don?t do it now is because nobody has had the time to make it happen yet. The problems before weren?t from going via sdist, they were from trying to modify our copy tree implementation to filter out .tox, .git, etc. I don?t think we?ve ever tried going via sdist (other than there is an open PR for it, but it ended up stalling https://github.com/pypa/pip/pull/3722 ). Essentially, volunteer time is finite :( ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Jun 2 13:04:58 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Jun 2017 18:04:58 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <843785FA-EAA1-450F-8005-FCC2DA513F4F@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <843785FA-EAA1-450F-8005-FCC2DA513F4F@stufft.io> Message-ID: On 2 June 2017 at 18:02, Donald Stufft wrote: > > I?m pretty sure the only reason we don?t do it now is because nobody has had > the time to make it happen yet. The problems before weren?t from going via > sdist, they were from trying to modify our copy tree implementation to > filter out .tox, .git, etc. I don?t think we?ve ever tried going via sdist > (other than there is an open PR for it, but it ended up stalling > https://github.com/pypa/pip/pull/3722). Oh, yes - it's the filtering stuff out work that I remembered, plus the fact that we'd discussed going via sdist. > Essentially, volunteer time is finite :( In my case, coding time is very limited, but time to write long email responses seems to be freely available. Sigh :-( Paul From ncoghlan at gmail.com Fri Jun 2 13:05:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 03:05:10 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496422214.2364627.996821928.528A70BF@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <1496415411.1384073.996691024.69B5CA2E@webmail.messagingengine.com> <1496422214.2364627.996821928.528A70BF@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 02:50, Thomas Kluyver wrote: > On Fri, Jun 2, 2017, at 05:26 PM, Nick Coghlan wrote: >> Could you provide a little more detail as to what you would do >> differently in exporting the contents of an sdist that wouldn't apply >> to export a build tree? (aside from skipping emitting PKG-INFO) > > When creating an sdist, I query the VCS to work out what files to put in > it. This is brittle - it depends on the directory being a VCS checkout, > and the relevant VCS being available to call. And it's relatively slow, > because we have to shell out to another process. I've decided those are > acceptable trade-offs for the project maintainer making the release. > > When exporting a build tree, I would copy only the files that are needed > to make the wheel. This is simple, robust and fast. Oh, I get it - for a build tree, you *know* which files you really need (since it's specified as part of flit's settings), whereas for the sdist, you also need to get all the *other* files that the build doesn't need, but the sdist should contain anyway. And this distinction will become even more important if Nathaniel gets his wish and we some day extend the backend API definition to support generating multiple wheel files from the same sdist. > I can even generate PKG-INFO when exporting a build tree if that helps. Now that I understand the point you were making, I don't think that's necessary. > But I want to keep the idea of a build tree used as an intermediate to > generating a wheel separate from that of an sdist, which is a release > artifact. Right, that makes sense to me now. It also means that even when building from a VCS checkout, you may not need the tools to *query* that VCS if things like your version number are specified in a static file. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Fri Jun 2 13:14:30 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 13:14:30 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> > On Jun 2, 2017, at 12:39 PM, Paul Moore wrote: > >> >> * The copy_the_files hook is optional, if it exists it SHOULD produce a tree >> that when the build_wheel hook is called in it, will produce a wheel that is >> equivalent to one that would have been built had the build_sdist hook been >> called instead. > > This is precisely the "should look like we built a sdist" principle, > so I'm a solid +1 on this, too. It might be worth stating that > copy_the_files is only intended to be called after a failed call to > build_sdist. I don't know if backends would care, but I don't think we > should worry about having to support use of copy_the_files as anything > other than a build_sdist fallback. > >> * If the copy_the_files hook is not defined, then the build frontend is free >> to just directory call the build_sdist command instead. > > Sorry? I assume here that you mean "directly call the build_wheel hook > in the original source tree"? That's OK, but I think we should be > clear that if this happens, it is the backend's responsibility to > ensure that the build is equivalent to building from a sdist. It might > even be appropriate for the front end to warn if this happens - > "Unable to build out of tree - results may differ from a clean build" > (The intent is to remind people that they aren't testing the actual > sdist they will be deploying, the issue that Nick pointed out). Should have kept reading before sending my email, sorry! The steps here would basically be (for building from something that isn?t already a .tar.gz or a .whl): # Get our backend using the PEP 517 resolving methods backend = get_the_backend() # Get a copied source tree that is acceptable for using to build a wheel. We # allow copy_files to be used in place of build_sdist to provide an optimization # in cases where build_sdist would be very slow. However the build backend # must ensure that the resulting wheel would not be different than if we had # built it from the sdist instead. If hasattr(backend, ?copy_files?): try: backend.copy_files(?) except Exception: backend.build_sdist(?) else: backend.build_sdist(?) # Determine what depends we need to install the wheel file, we allow the # build tool to optionally give us the deps without actually invoking the wheel build # as an optimization that building the wheel might take awhile, however # the build backend must ensure that the metadata returned here matches the final # wheel built. If hasattr(backend, ?get_wheel_metadata?): backend.get_wheel_metadata(?) has_already_built_wheel = False else: backend.build_wheel(?) has_already_built_wheel = True # Resolve dependencies, etc # Go on to build the wheel if we haven?t already built it. If not has_already_built_wheel: backend.build_wheel(?) # Install the wheel ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 2 13:43:57 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 13:43:57 -0400 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: > On Jun 2, 2017, at 12:04 AM, Nick Timkovich wrote: > > I suggested on one of those issues to try to auto-blacklist common 404s as that should pose a negligible usability hit. I'd like to start by logging them to collect data, but I'm confused nowadays as to if that should go into pypa/warehouse or pypa/pypi-legacy. How long until warehouse is where most requests go, or do some go there right now, but from which clients...so confuz, plz halp. The easiest thing to do would probably be to add this to linehaul (pypa/linehaul on GH). That?s the daemon the processes log lines from Fastly. It?d need some way to dispatch between successful downloads, and a 404 on /simple// (possibly just have it bind to two ports one for downloads one for 404s? I dunno) and it?d need to store the records somewhere (possibly it?d make sense just to make a second set of BigQuery tables, these ones not public and likely expiring after a period of time). There would need to be a little bit of VCL work to actually get Fastly sending those log streams, but that?s pretty easy once the work to ingest them is done. That would get us to the point we can start collecting data and storing it. The next step would be to start processing that data to implement a black list, which would require work to be done in both Warehouse and legacy PyPI. Warehouse you?d want to implement the thing that actually periodically processes the big query data to generate the block list, and then in both Warehouse and Legacy PyPI you?d want to implement the block list support in the upload/register routines. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Jun 2 16:58:02 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 02 Jun 2017 21:58:02 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> Message-ID: <1496437082.4045245.997054632.7F83B2AF@webmail.messagingengine.com> On Fri, Jun 2, 2017, at 06:14 PM, Donald Stufft wrote: > The steps here would basically be (for building from something that > isn?t already a .tar.gz or a .whl): That sounds OK to me. I think the only remaining point of contention is whether the build_sdist hook should make an archive or an unpacked directory. I'm not entirely sold, but Nick's point about not having to specify the archive format is enough that I've changed my PR to specify an unpacked sdist. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 2 22:14:58 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Jun 2017 19:14:58 -0700 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: On Fri, Jun 2, 2017 at 9:39 AM, Paul Moore wrote: > Note that there's an implication here - if we define the build process > in terms of the effect of "going via a sdist", then we need to at > least have an intuitive understanding of what that means in practice. > I don't think it's a contentious point (even if the specific term > "sdist" is open to debate), as I think repeatable builds are a > well-understood idea. (It's at this point that the concerns of people > who want incremental builds come in - we should support incremental > builds in a way that preserves the "just like going via a sdist" > principle. But again, they need to raise their concerns if they think > we're missing something key to their use case). So far my belief is that packages with expensive build processes are going to ignore you and implement, ship, document, and recommend the direct source-tree->wheel path for developer builds. You can force the make-a-wheel-from-a-directory-without-copying-and-then-install-it command have a name that doesn't start with "pip", but it's still going to exist and be used. Why wouldn't it? It's trivial to implement and it works, and I haven't heard any alternative proposals that have either of those properties. [1] Relatedly, the idea of a copy_files hook doesn't make sense to me. The only reason pip wants to force builds through the sdist phase is because it doesn't trust the backend to make clean wheels, and it's willing to make its local directory builds much slower to get that guarantee. When you add copy_files, you lose that guarantee *and* you're still making local directory builds much slower, so what's the point? If the always-via-sdist plan doesn't work for either the simplest cases (flit) or the most complex (incremental builds), then is it really a good plan? So it seems clear that we should do: - add get_build_sdist_requires and build_sdist hooks to PEP 517 or a new PEP, whichever (and for clarity rename the current get_build_requires -> get_build_wheel_requires) Beyond that, my preference is: - 'pip install local-directory/' switches to building in-place If the pip devs don't trust build systems in general, but (as suggested by copy_files discussion) are ok with trusting them if they promise to be super trustworthy, alternate proposal: - add a 'in_place_build_safe = True' hook, which indicates that the build system has been carefully written so that this will generate the same result as building an sdist and then building that; pip checks for this to decide whether to build in place or to build an sdist first. In principle this is a little silly (who doesn't think themselves trustworthy?), but it would let us continue to do build isolation for setuptools builds and any build system that hasn't put some thought into this, makes it clear where the responsibility lies if someone screws up, and backends that don't want to deal with building sdists from sdists could make this False for VCS checkouts and True for unpacked sdists. If the pip devs are set on 'pip install local-directory/' always making some sort of copy, then I suggest: - 'pip install local-directory/' makes an sdist and then builds it - we also add something like 'pip devinstall local-directory/' that does an in-place build and then installs it, so that I don't have to make a separate 'devinstall' script and ship it separately - 'flit' adds code to make sdist-from-sdist work. (One way: when building an sdist from a VCS checkout, make a list of all the ancillary files and include it in the resulting sdist. Or possibly just a list of all files + hashes. When asked to make an sdist from an arbitrary directory, check for this file, and if present use it as the list of ancillary files to include, and possibly check if any hashes have changed, and if so change the version number of the resulting sdist by appending "+dirty" or something; otherwise, use the current VCS-based system.) One thing that's not clear to me: a crucial use case for sdists is (1) download, (2) unpack, (3) patch the source, possibly adding new files, (4) build and install. (After all, the whole reason we insist on distributing sdists is that open source software should be modifiable by the recipient.) Does flit currently support this, given the reliance on VCS metadata? Other unresolved issues: - Donald had some concerns about get_wheel_metadata and they've led to several suggestions, none of which has made everyone go "oh yeah obviously that's the solution". To me this suggests we should go ahead and drop it from PEP 517 and add it back later if/when the need is more obvious. It's optional anyway, so adding it later doesn't hurt anything. - It sounds like there's some real question about how exactly a build frontend should handle the output from build_wheel; in particular, the PEP should say what happens if there are multiple files deposited into the output dir. My original idea when writing the PEP was that the build frontend would know the name/version of the wheel it was looking for, and so it would ignore any other files found in the output dir, which would be forward compatible with a future PEP allowing build_wheel to drop multiple wheels into the output dir (i.e., old pip's would just ignore them). It's clear from the discussion that this isn't how others were imagining it. Which is fine, I don't think this is a huge problem, but we should nail it down so we're not surprised later. -n [1] Donald's suggestion of silently caching intermediate files in some global cache dir is unreasonably difficult to implement in a user-friendly way ? cache management is Hard, and I frankly I still don't think users will accept individual package's build systems leaving hundreds of megabytes of random gunk inside hidden directories. We could debate the details here, but basically, if this were a great idea to do by default, then surely one of cmake/autoconf/... would already do it? Also, my understanding is the main reason pip wants to copy files in the first place is to avoid accidental pollution between different builds using the same local tree; but if a build system implements a global cache like this then surprise, now you can get pollution between arbitrary builds using different trees, or between builds that don't even use a local tree at all (e.g. running 'pip install numpy==1.12.0' can potentially cause a later run of 'pip install numpy==1.12.1' to be corrupted). And, it assumes that all build systems can easily support out-of-tree incremental builds, which is often true but not guaranteed when your wheel build has to wrap some random third party C library's build system. -- Nathaniel J. Smith -- https://vorpus.org From donald at stufft.io Fri Jun 2 23:38:34 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Jun 2017 23:38:34 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> > On Jun 2, 2017, at 10:14 PM, Nathaniel Smith wrote: > > On Fri, Jun 2, 2017 at 9:39 AM, Paul Moore wrote: >> Note that there's an implication here - if we define the build process >> in terms of the effect of "going via a sdist", then we need to at >> least have an intuitive understanding of what that means in practice. >> I don't think it's a contentious point (even if the specific term >> "sdist" is open to debate), as I think repeatable builds are a >> well-understood idea. (It's at this point that the concerns of people >> who want incremental builds come in - we should support incremental >> builds in a way that preserves the "just like going via a sdist" >> principle. But again, they need to raise their concerns if they think >> we're missing something key to their use case). > > So far my belief is that packages with expensive build processes are > going to ignore you and implement, ship, document, and recommend the > direct source-tree->wheel path for developer builds. You can force the > make-a-wheel-from-a-directory-without-copying-and-then-install-it > command have a name that doesn't start with "pip", but it's still > going to exist and be used. Why wouldn't it? It's trivial to implement > and it works, and I haven't heard any alternative proposals that have > either of those properties. [1] If someone wants to implement a direct-to-wheel build tool and have it compete with ``pip install .`` they?re more than welcome to. Competition is healthy and at the very worst case it could validate either the idea that direct-to-wheel is important enough that people will gladly overcome the relatively small barrier of having to install another tool and then we have data to indicate maybe we need to rethink things or it could validate the idea that it?s not important enough and leave things as they are. I went and looked through all 105 pages of pip?s issues (open and closed) and made several searches using any keyword I could think of looking for any issue where someone asked for this. The only times I can find anyone asking for this were you and Ralf Gommers as part of the extended discussion around this set of PEPs and I?ve not been able to find a single other person asking for it or complaining about it. However, what I was able to find was what appears to be the original reason pip started copying the directory to begin with, https://github.com/pypa/pip/issues/178 which was caused by the build system reusing the build directory between two different virtual environments and causing an invalid installation to happen. The ticket is old enough that I can get at specifics it because it was migrated over from bitbucket. However the fact that we *used* to do exactly what you want and it caused exactly one of problem I was worried about seems to suggest to me that pip is absolutely correct in keeping this behavior. > > Relatedly, the idea of a copy_files hook doesn't make sense to me. The > only reason pip wants to force builds through the sdist phase is > because it doesn't trust the backend to make clean wheels, and it's > willing to make its local directory builds much slower to get that > guarantee. When you add copy_files, you lose that guarantee *and* > you're still making local directory builds much slower, so what's the > point? If the always-via-sdist plan doesn't work for either the > simplest cases (flit) or the most complex (incremental builds), then > is it really a good plan? It?s not that I don?t trust the backend, it?s that I believe in putting in systems that make it harder to do the wrong thing than the right thing. As it is now building in place correctly requires the build backend to do extra work to ensure that some file that wouldn?t be included in the sdist doesn?t influence the build in some way. Given that I?m pretty sure literally every build tool in existence for Python currently fails this test, I think that is a pretty reasonable statement to say that it might continue to be a problem into the future. Copying the files makes that harder to do (but still easier than always going through the sdist). If you want to argue that we should always go through the sdist and we shouldn?t have a copy_files hook, I?m ok with that. I?m only partially in favor of it as a performance trade off because I think it passes a high enough bar that it?s unlikely enough for mistakes to be made (and when they do, they?ll be more obvious). > > ? ... > > - 'flit' adds code to make sdist-from-sdist work. (One way: when > building an sdist from a VCS checkout, make a list of all the > ancillary files and include it in the resulting sdist. Or possibly > just a list of all files + hashes. When asked to make an sdist from an > arbitrary directory, check for this file, and if present use it as the > list of ancillary files to include, and possibly check if any hashes > have changed, and if so change the version number of the resulting > sdist by appending "+dirty" or something; otherwise, use the current > VCS-based system.) This seems reasonable to me. > > One thing that's not clear to me: a crucial use case for sdists is (1) > download, (2) unpack, (3) patch the source, possibly adding new files, > (4) build and install. (After all, the whole reason we insist on > distributing sdists is that open source software should be modifiable > by the recipient.) Does flit currently support this, given the > reliance on VCS metadata? > > Other unresolved issues: > > - Donald had some concerns about get_wheel_metadata and they've led to > several suggestions, none of which has made everyone go "oh yeah > obviously that's the solution". To me this suggests we should go ahead > and drop it from PEP 517 and add it back later if/when the need is > more obvious. It's optional anyway, so adding it later doesn't hurt > anything. My main concern is the metadata diverging between the get_wheel_metadata and the building wheel phase. The current PEP solves that in a reasonable enough way (and in a way I can assert against). My other concerns are mostly just little API niggles to make it harder to mess up. I think this one is important to support because we do not to be able to get at the dependencies, and invoking the entire build chain to do that seems like it will be extraordinarily slow. > > - It sounds like there's some real question about how exactly a build > frontend should handle the output from build_wheel; in particular, the > PEP should say what happens if there are multiple files deposited into > the output dir. My original idea when writing the PEP was that the > build frontend would know the name/version of the wheel it was looking > for, and so it would ignore any other files found in the output dir, > which would be forward compatible with a future PEP allowing > build_wheel to drop multiple wheels into the output dir (i.e., old > pip's would just ignore them). It's clear from the discussion that > this isn't how others were imagining it. Which is fine, I don't think > this is a huge problem, but we should nail it down so we're not > surprised later. How do you determine the name/version for ``pip install .`` except by running get_wheel_metadata or build_wheel or build_sdist? > > -n > > [1] Donald's suggestion of silently caching intermediate files in some > global cache dir is unreasonably difficult to implement in a > user-friendly way ? cache management is Hard, and I frankly I still > don't think users will accept individual package's build systems > leaving hundreds of megabytes of random gunk inside hidden > directories. We could debate the details here, but basically, if this > were a great idea to do by default, then surely one of > cmake/autoconf/... would already do it? Also, my understanding is the > main reason pip wants to copy files in the first place is to avoid > accidental pollution between different builds using the same local > tree; but if a build system implements a global cache like this then > surprise, now you can get pollution between arbitrary builds using > different trees, or between builds that don't even use a local tree at > all (e.g. running 'pip install numpy==1.12.0' can potentially cause a > later run of 'pip install numpy==1.12.1' to be corrupted). And, it > assumes that all build systems can easily support out-of-tree > incremental builds, which is often true but not guaranteed when your > wheel build has to wrap some random third party C library's build > system. > Make it opt-in and build a hash of the directory into the cache key so different file contents mean different cache objects then. I?m not really sold on the idea that the fact some developers haven?t decided to do it then it is a bad idea. Perhaps those build systems are operating under different constraints than we are (I?m almost certainly sure this is the case). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Fri Jun 2 23:53:17 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Fri, 2 Jun 2017 22:53:17 -0500 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496437082.4045245.997054632.7F83B2AF@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> <1496437082.4045245.997054632.7F83B2AF@webmail.messagingengine.com> Message-ID: On Fri, Jun 2, 2017 at 3:58 PM, Thomas Kluyver wrote: > On Fri, Jun 2, 2017, at 06:14 PM, Donald Stufft wrote: > > The steps here would basically be (for building from something that isn?t > already a .tar.gz or a .whl): > > > That sounds OK to me. I think the only remaining point of contention is > whether the build_sdist hook should make an archive or an unpacked > directory. I'm not entirely sold, but Nick's point about not having to > specify the archive format is enough that I've changed my PR to specify an > unpacked sdist. > This isn't a question directed at what I've quoted here, but seems as good of place as any. I want to make sure I understand what I'd need to do, as a user, in a post PEP 517 world. Say I wanted to accomplish the following three things: * Generate version info from my VCS * Generate .h and .c from .pyx or cffi's out-of-line API mode * Use an alternative build process for a platform distutils isn't behaving with (ignore this requirement if it leads to problems answering, we can just as well assume distutils here) Since I can only define one entrypoint for build-system.build-backend, I can't point directly at setuptools_scm (or equiv) as it only achieves the first, I can't point directly at cython or cffi (or equiv) as it only achieves the second, and I can't point directly at my-custom-backend or distutils as it only achieves the third... right? If I added setuptools_scm and cffi to build-system.requires, will it have an opportunity to somehow do something useful before my-custom-backend begins its work? Does this mean, for many situations, unless a single backed supports all the things I want to do automatically to my source tree prior to building, I need to write a local wrapper build-system backend, in my project, that farms out build API calls to each of the backends above, myself? Can I call it setup.py? I'm not suggesting the above is good or bad, or even suggesting it wouldn't be an improvement over what we have now, I'm just trying to think through how it works and how I'd end up doing things. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 3 01:02:40 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 15:02:40 +1000 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: <1496337528.1454126.995666696.7CCDF273@webmail.messagingengine.com> <1496337875.1455799.995679408.69485FA9@webmail.messagingengine.com> <1496338850.1459729.995689848.71E6A0C6@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 03:43, Donald Stufft wrote: > That would get us to the point we can start collecting data and storing it. > The next step would be to start processing that data to implement a black > list, which would require work to be done in both Warehouse and legacy PyPI. > Warehouse you?d want to implement the thing that actually periodically > processes the big query data to generate the block list, and then in both > Warehouse and Legacy PyPI you?d want to implement the block list support in > the upload/register routines. Something I was wondering was whether it might make sense to store the blocklist as a separate table in the database, and enforce it using a PostgreSQL constraint trigger when a project name is used for the first time (i.e. when there are no previously recorded releases using that name). You'd still want explicit support at least in the Warehouse frontend to provide a better UX when someone attempts to register a blocked name and to provide admins with the ability to override the auto-generated blocklist settings, but actually *enforcing* the blocklist would become the database's job. Not an urgent concern (since it's only a hypothetical design question until the blocklist generation is implemented), but I figured it was worth mentioning as a potential way to avoid having to update the legacy PyPI codebase with block list support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 3 01:21:39 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 15:21:39 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> Message-ID: On 3 June 2017 at 13:38, Donald Stufft wrote: > However, what I was able to find was what appears to be the original reason > pip started copying the directory to begin with, > https://github.com/pypa/pip/issues/178 which was caused by the build system > reusing the build directory between two different virtual environments and > causing an invalid installation to happen. The ticket is old enough that I > can get at specifics it because it was migrated over from bitbucket. However > the fact that we *used* to do exactly what you want and it caused exactly > one of problem I was worried about seems to suggest to me that pip is > absolutely correct in keeping this behavior. FWIW, I'll also note that in-place builds play merry hell with containerised build tools, volume mounts, and SELinux filesystem labels. In-place builds *can* be made to work, and when you invest the time to make them work, they give you all sorts of useful benefits (incremental builds, etc), but out-of-tree builds inherently avoid a lot of potential problems (especially in a world where virtual environments are a thing). As far as "out-of-tree caching" is concerned, all the build systems I'm personally familiar with *except* the C/C++ ones use some form of out of band caching location, even if that's dependency caching rather than build artifact caching. As an example of the utility of that approach, Atlassian recently updated the Alpha version of their Pipelines feature to automatically manage cache directories and allow them to be shared between otherwise independent builds: https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Sat Jun 3 01:40:43 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Jun 2017 22:40:43 -0700 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> Message-ID: On Fri, Jun 2, 2017 at 8:38 PM, Donald Stufft wrote: > > On Jun 2, 2017, at 10:14 PM, Nathaniel Smith wrote: > >> So far my belief is that packages with expensive build processes are >> going to ignore you and implement, ship, document, and recommend the >> direct source-tree->wheel path for developer builds. You can force the >> make-a-wheel-from-a-directory-without-copying-and-then-install-it >> command have a name that doesn't start with "pip", but it's still >> going to exist and be used. Why wouldn't it? It's trivial to implement >> and it works, and I haven't heard any alternative proposals that have >> either of those properties. [1] > > > If someone wants to implement a direct-to-wheel build tool and have it > compete with ``pip install .`` they?re more than welcome to. Competition is > healthy and at the very worst case it could validate either the idea that > direct-to-wheel is important enough that people will gladly overcome the > relatively small barrier of having to install another tool and then we have > data to indicate maybe we need to rethink things or it could validate the > idea that it?s not important enough and leave things as they are. > > I went and looked through all 105 pages of pip?s issues (open and closed) > and made several searches using any keyword I could think of looking for any > issue where someone asked for this. The only times I can find anyone asking > for this were you and Ralf Gommers as part of the extended discussion around > this set of PEPs and I?ve not been able to find a single other person asking > for it or complaining about it. That's because until now, the message that everyone has received over and over is that the way you install a package from a directory on disk is: cd directory python setup.py install and this does incremental builds. (My experience is that even today, most people are surprised to learn that 'pip install' accepts directory paths.) In our glorious PEP 517 future, we have to teach everyone to stop using 'setup.py install' and instead use 'pip install .'. This switch enables a glorious new future of non-distutils-based build systems and fixes a bunch of other brokenness at the same time, hooray, BUT currently switching to 'pip install' also causes a regression for everyone who's used to incremental builds working. Ralf and I noticed this because we were looking at getting a head start on the glorious future by making 'pip install' mandatory for numpy and scipy. The reason no-one else has noticed is that we're among the few people that have tried using 'pip install' as their standard install-from-working-tree command. But soon there will be more. > However, what I was able to find was what appears to be the original reason > pip started copying the directory to begin with, > https://github.com/pypa/pip/issues/178 which was caused by the build system > reusing the build directory between two different virtual environments and > causing an invalid installation to happen. The ticket is old enough that I > can get at specifics it because it was migrated over from bitbucket. However > the fact that we *used* to do exactly what you want and it caused exactly > one of problem I was worried about seems to suggest to me that pip is > absolutely correct in keeping this behavior. Hmm, it looks to me like that bug is saying that at the time, if you ran 'python setup.py install' *inside the pip source tree*, and then tried to run pip's test suite (possibly via 'setup.py test'), then it broke. I don't think this is related to the behavior of 'pip install .', and I feel like we would know if it were currently true that running 'setup.py install' twice in the same directory produced broken shebang lines. (Again, most people who install from source directories are currently using setup.py install!) The source tree copying was originally added in: https://github.com/pypa/pip/commit/57bd8163e4483b7138342da93f5f6bb8460f0e4a (which is dated ~2 months before that bug you found, and if I'm reading it right tweaks a code path that previously only worked for 'pip install foo.zip' so it also works for 'pip install foo/'). AFAICT the reason it was written this way is that pip started out with the assumption that it was always going to be downloading and unpacking archives, so the logic went: 1) make a temporary directory 2) unpack the sdist into this temporary directory 3) build from this temporary directory Then, when it came time to add support for building from directories, the structure of the logic meant that by the time pip got to step (2) and realized that it already had a source directory, it was too late -- it was already committed to using the selected temporary directory. So instead of refactoring all this code, they made the minimal change of implementing the "unpack this sdist into this directory" operation for source directories by using shutil.copytree. I think this chain of reasoning will feel very familiar to anyone working with the modern pip source 5 years later... It's absolutely true that there are cases where incremental builds can screw things up, especially when using distutils/setuptools. But I don't think this is why pip does things this way originally :-). > It?s not that I don?t trust the backend, it?s that I believe in putting in > systems that make it harder to do the wrong thing than the right thing. As > it is now building in place correctly requires the build backend to do extra > work to ensure that some file that wouldn?t be included in the sdist doesn?t > influence the build in some way. Given that I?m pretty sure literally every > build tool in existence for Python currently fails this test, I think that > is a pretty reasonable statement to say that it might continue to be a > problem into the future. > > Copying the files makes that harder to do (but still easier than always > going through the sdist). If you want to argue that we should always go > through the sdist and we shouldn?t have a copy_files hook, I?m ok with that. > I?m only partially in favor of it as a performance trade off because I think > it passes a high enough bar that it?s unlikely enough for mistakes to be > made (and when they do, they?ll be more obvious). What do you think of letting build backends opt-in to in-place builds? >> Other unresolved issues: >> >> - Donald had some concerns about get_wheel_metadata and they've led to >> several suggestions, none of which has made everyone go "oh yeah >> obviously that's the solution". To me this suggests we should go ahead >> and drop it from PEP 517 and add it back later if/when the need is >> more obvious. It's optional anyway, so adding it later doesn't hurt >> anything. > > My main concern is the metadata diverging between the get_wheel_metadata and > the building wheel phase. The current PEP solves that in a reasonable enough > way (and in a way I can assert against). My other concerns are mostly just > little API niggles to make it harder to mess up. > > I think this one is important to support because we do not to be able to get > at the dependencies, and invoking the entire build chain to do that seems > like it will be extraordinarily slow. It's only slow in the case where (a) there's no wheel (obviously), and (b) after getting the dependencies we decide we don't want to install this sdist after all. I imagine numpy for example won't bother implementing get_wheel_metadata because we provide wheels for all the platforms we support and because we have no dependencies, so it is doubly useless AFAICT. But yeah in other cases it could matter. I'm not opposed to including it in general, just thought this might be a way to help get the minimal PEP 517 out the door. >> - It sounds like there's some real question about how exactly a build >> frontend should handle the output from build_wheel; in particular, the >> PEP should say what happens if there are multiple files deposited into >> the output dir. My original idea when writing the PEP was that the >> build frontend would know the name/version of the wheel it was looking >> for, and so it would ignore any other files found in the output dir, >> which would be forward compatible with a future PEP allowing >> build_wheel to drop multiple wheels into the output dir (i.e., old >> pip's would just ignore them). It's clear from the discussion that >> this isn't how others were imagining it. Which is fine, I don't think >> this is a huge problem, but we should nail it down so we're not >> surprised later. > > > How do you determine the name/version for ``pip install .`` except by > running get_wheel_metadata or build_wheel or build_sdist? Well, I was imagining that the semantics of 'pip install .' in a multi-wheel world would be to install all the generated wheels :-). But yeah, it's not really well-specified as currently written. Possibly the simplest solution is to say that build_wheel has to return a string which names the wheel, and then in the future we could add build_wheel2 which is identical but returns a list of strings, and backwards compatibility would be: def build_wheel2(...): return build_wheel(...)[0] > -n > >> [1] Donald's suggestion of silently caching intermediate files in some >> global cache dir is unreasonably difficult to implement in a >> user-friendly way ? cache management is Hard, and I frankly I still >> don't think users will accept individual package's build systems >> leaving hundreds of megabytes of random gunk inside hidden >> directories. We could debate the details here, but basically, if this >> were a great idea to do by default, then surely one of >> cmake/autoconf/... would already do it? Also, my understanding is the >> main reason pip wants to copy files in the first place is to avoid >> accidental pollution between different builds using the same local >> tree; but if a build system implements a global cache like this then >> surprise, now you can get pollution between arbitrary builds using >> different trees, or between builds that don't even use a local tree at >> all (e.g. running 'pip install numpy==1.12.0' can potentially cause a >> later run of 'pip install numpy==1.12.1' to be corrupted). And, it >> assumes that all build systems can easily support out-of-tree >> incremental builds, which is often true but not guaranteed when your >> wheel build has to wrap some random third party C library's build >> system. > > > > Make it opt-in If it's opt-in, then I might as well tell people to run 'pip devinstall .' or 'in-place-install .' or whatever instead, and it'll be much easier all around. But instead of making it opt-in, I'd much rather it Just Work. It's frustrating that at the same time we're moving to the glorious simplified future, we're also picking up a new piece of arcane wisdom that devs will need to be taught, and another place where numerical Python devs will roll their eyes at how the standard Python tooling doesn't care about them. (And I totally understand that the motivation on your end is also to make things Just Work, but I feel like in the specific case where someone is *repeatedly* building out of the *same source directory* ? which is the one in dispute here ? we should optimize for developer experience.) > and build a hash of the directory into the cache key so > different file contents mean different cache objects then. I?m not really > sold on the idea that the fact some developers haven?t decided to do it then > it is a bad idea. Perhaps those build systems are operating under different > constraints than we are (I?m almost certainly sure this is the case). I think the way the constraints differ is just that they don't have this imposed constraint that they *must* build out of an ephemeral tempdir. If we're operating under that constraint, then your idea is certainly worth considering. My point is that it's much easier to remove that constraint (by switching to using a non-pip tool) than it is to work around it, so that's my prediction for what will happen. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Sat Jun 3 01:53:28 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Jun 2017 22:53:28 -0700 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> Message-ID: On Fri, Jun 2, 2017 at 10:21 PM, Nick Coghlan wrote: > On 3 June 2017 at 13:38, Donald Stufft wrote: >> However, what I was able to find was what appears to be the original reason >> pip started copying the directory to begin with, >> https://github.com/pypa/pip/issues/178 which was caused by the build system >> reusing the build directory between two different virtual environments and >> causing an invalid installation to happen. The ticket is old enough that I >> can get at specifics it because it was migrated over from bitbucket. However >> the fact that we *used* to do exactly what you want and it caused exactly >> one of problem I was worried about seems to suggest to me that pip is >> absolutely correct in keeping this behavior. > > FWIW, I'll also note that in-place builds play merry hell with > containerised build tools, volume mounts, and SELinux filesystem > labels. > > In-place builds *can* be made to work, and when you invest the time to > make them work, they give you all sorts of useful benefits > (incremental builds, etc), but out-of-tree builds inherently avoid a > lot of potential problems (especially in a world where virtual > environments are a thing). > > As far as "out-of-tree caching" is concerned, all the build systems > I'm personally familiar with *except* the C/C++ ones use some form of > out of band caching location, even if that's dependency caching rather > than build artifact caching. > > As an example of the utility of that approach, Atlassian recently > updated the Alpha version of their Pipelines feature to automatically > manage cache directories and allow them to be shared between otherwise > independent builds: > https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html Oh sure, if you have a piece of build *infrastructure*, then all kinds of things make sense. Set up ccache, distcc, cache dependencies, go wild. Mozilla's got a cute version of ccache that puts the cache in s3 so it can be shared among ephemeral build VMs. That's not what I'm talking about. The case I'm talking about is, like, a baby dev taking their first steps, or someone trying to get a build of a package working on an unusual system: git clone ..../numpy.git cd numpy # edit some file, maybe a config file saying which fortran compiler this weird machine uses # build and run tests In this case it would be extremely rude to silently dump all our intermediate build artifacts into ~/.something, but I also don't want to require every new dev opt-in to some special infrastructure and learn new commands ? I want there to be gentle onramp from blindly installing packages as a user to hacking on them. Making 'pip install' automatically do incremental builds when run repeatedly on the same working directory accomplishes this better than anything else. It's not clear to me what cases you're concerned about breaking with "containerised build tools, ...". Are you thinking about, like, 'docker run some-volume -v $PWD:/io pip install /io'? Surely for anything involving containers there should be an explicit wheel built somewhere in there? -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sat Jun 3 03:09:37 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 17:09:37 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> Message-ID: On 3 June 2017 at 15:53, Nathaniel Smith wrote: > That's not what I'm talking about. The case I'm talking about is, > like, a baby dev taking their first steps, or someone trying to get a > build of a package working on an unusual system: > > git clone ..../numpy.git > cd numpy > # edit some file, maybe a config file saying which fortran compiler > this weird machine uses > # build and run tests It's come up a couple of times before, but this example makes me realise that we should be *explicitly* using "tox" as our reference implementation for "local developer experience", to avoid letting ourselves fall into the trap of optimising too much for pip specifically as the reference installer. The reason I say that is I actually looked at the tox docs yesterday, and *completely missed* the relevance of one of their config settings to PEP 517 (the one that lets you skip the sdist creation step when its too slow): https://tox.readthedocs.io/en/latest/example/general.html#avoiding-expensive-sdist However, I'm not sure doing that leads to the conclusion that we need to support in-place builds in PEP 517, as tox's approach to skipping the sdist step in general is to require the user to specify a custom build command. The only "in-place" option it supports directly is editable installs. So from that perspective, the PEP 517 answer to "How do I do an in-place build?" would be "Use whatever command your backend provides for that purpose". This makes sense, as this particular abstraction layer isn't meant to hide the build backend from the people *working on* a project - it's only meant to hide it from the people *using* the project. So as a NumPy or SciPy developer, it's entirely reasonable to have to know that the command for an in-place build is "python setup.py ...". > In this case it would be extremely rude to silently dump all our > intermediate build artifacts into ~/.something, but I also don't want > to require every new dev opt-in to some special infrastructure and > learn new commands ? I want there to be gentle onramp from blindly > installing packages as a user to hacking on them. Making 'pip install' > automatically do incremental builds when run repeatedly on the same > working directory accomplishes this better than anything else. "Build this static snapshot of a thing someone else published so I can use it" and "Build this thing I'm working on so I can test it" are closely related, but not the same, so I think the latter concerns are likely to be better handled through an effort to replace the setuptools specific "pip install -e" with a more general "pip devinstall". That would then map directly to the existing logic in tox, allowing that to migrate from running "pip install -e" when usedevelop=True to instead running "pip devinstall". > It's not clear to me what cases you're concerned about breaking with > "containerised build tools, ...". Are you thinking about, like, > 'docker run some-volume -v $PWD:/io pip install /io'? Surely for > anything involving containers there should be an explicit wheel built > somewhere in there? With live re-loading support in web servers, it's really handy to volume mount your working directory into the container. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sat Jun 3 03:47:26 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 3 Jun 2017 03:47:26 -0400 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> Message-ID: > On Jun 3, 2017, at 1:40 AM, Nathaniel Smith wrote: > > On Fri, Jun 2, 2017 at 8:38 PM, Donald Stufft wrote: >> >> On Jun 2, 2017, at 10:14 PM, Nathaniel Smith wrote: >> >>> So far my belief is that packages with expensive build processes are >>> going to ignore you and implement, ship, document, and recommend the >>> direct source-tree->wheel path for developer builds. You can force the >>> make-a-wheel-from-a-directory-without-copying-and-then-install-it >>> command have a name that doesn't start with "pip", but it's still >>> going to exist and be used. Why wouldn't it? It's trivial to implement >>> and it works, and I haven't heard any alternative proposals that have >>> either of those properties. [1] >> >> >> If someone wants to implement a direct-to-wheel build tool and have it >> compete with ``pip install .`` they?re more than welcome to. Competition is >> healthy and at the very worst case it could validate either the idea that >> direct-to-wheel is important enough that people will gladly overcome the >> relatively small barrier of having to install another tool and then we have >> data to indicate maybe we need to rethink things or it could validate the >> idea that it?s not important enough and leave things as they are. >> >> I went and looked through all 105 pages of pip?s issues (open and closed) >> and made several searches using any keyword I could think of looking for any >> issue where someone asked for this. The only times I can find anyone asking >> for this were you and Ralf Gommers as part of the extended discussion around >> this set of PEPs and I?ve not been able to find a single other person asking >> for it or complaining about it. > > That's because until now, the message that everyone has received over > and over is that the way you install a package from a directory on > disk is: > > cd directory > python setup.py install > > and this does incremental builds. (My experience is that even today, > most people are surprised to learn that 'pip install' accepts > directory paths.) I?m not sure this is true? I mean I?m sure it?s true that for *some* people that?s the message they get, but we see a reasonable volume of bug reports and the like from people passing a path to pip that it?s not hardly an un(der)used feature in the slightest. We have zero metrics so I suspect there is no way to answer which one is used more than the other though. I think writing either one off as ?nobody uses that method? is likely to be the wrong answer. Pip is generally used widely enough in enough different scenarios that it is unusual for any feature it has (even ones that are completely undocumented and requires diving in the source code!) to not be used by a decent chunk of people. I?m not saying that zero people exist that would want this (indeed, there is at least two!) but that it has obviously not been pressing enough to need it that someone felt the need to open a ticket or ask for it before. > > In our glorious PEP 517 future, we have to teach everyone to stop > using 'setup.py install' and instead use 'pip install .'. This switch > enables a glorious new future of non-distutils-based build systems and > fixes a bunch of other brokenness at the same time, hooray, BUT > currently switching to 'pip install' also causes a regression for > everyone who's used to incremental builds working. > > Ralf and I noticed this because we were looking at getting a head > start on the glorious future by making 'pip install' mandatory for > numpy and scipy. The reason no-one else has noticed is that we're > among the few people that have tried using 'pip install' as their > standard install-from-working-tree command. But soon there will be > more. One thing I?d note is that as far as I can tell, neither the current copy of the PEP, nor my PR or Thomas? PR or any of the discussions here have talked about having the interface itself mandate calling build_sdist prior to calling build_wheel. I?m personally fine if we want the interface to allow it (and I assumed we would TBH) and explicitly calling that out. That means that whether or not you call build_sdist (or some copy files hook) first effectively ends up being an implementation detail of the frontend in question. That would allow pip to build the sdist (or do the copy thing) first but another tool could implement it the other way (and installing a local wheel is so simple that you could pretty easily implement that and just shell out to pip for the rest if you wanted). That also means that we can adjust our answer to it in the future. If such a tool gets built and a lot of people end up using it and asking for it in pip, we can revisit that decision in a future version of pip. Part of the stand off here is the pip developers view it as a regression if we stop building in isolation and you view it as a regression if incremental/inplace builds are not supported. Both can be true! It?s the opinion of the pip developers who have spoken so far that for us, the risk of our regressions is high enough we don?t currently feel comfortable changing that behavior. > >> However, what I was able to find was what appears to be the original reason >> pip started copying the directory to begin with, >> https://github.com/pypa/pip/issues/178 which was caused by the build system >> reusing the build directory between two different virtual environments and >> causing an invalid installation to happen. The ticket is old enough that I >> can get at specifics it because it was migrated over from bitbucket. However >> the fact that we *used* to do exactly what you want and it caused exactly >> one of problem I was worried about seems to suggest to me that pip is >> absolutely correct in keeping this behavior. > > Hmm, it looks to me like that bug is saying that at the time, if you > ran 'python setup.py install' *inside the pip source tree*, and then > tried to run pip's test suite (possibly via 'setup.py test'), then it > broke. I don't think this is related to the behavior of 'pip install > .', and I feel like we would know if it were currently true that > running 'setup.py install' twice in the same directory produced broken > shebang lines. (Again, most people who install from source directories > are currently using setup.py install!) Meh yea, I misread the bug report. Reading 105 pages of reports will do that I guess :P > > The source tree copying was originally added in: > > https://github.com/pypa/pip/commit/57bd8163e4483b7138342da93f5f6bb8460f0e4a > > (which is dated ~2 months before that bug you found, and if I'm > reading it right tweaks a code path that previously only worked for > 'pip install foo.zip' so it also works for 'pip install foo/'). AFAICT > the reason it was written this way is that pip started out with the > assumption that it was always going to be downloading and unpacking > archives, so the logic went: > > 1) make a temporary directory > 2) unpack the sdist into this temporary directory > 3) build from this temporary directory > > Then, when it came time to add support for building from directories, > the structure of the logic meant that by the time pip got to step (2) > and realized that it already had a source directory, it was too late > -- it was already committed to using the selected temporary directory. > So instead of refactoring all this code, they made the minimal change > of implementing the "unpack this sdist into this directory" operation > for source directories by using shutil.copytree. > > I think this chain of reasoning will feel very familiar to anyone > working with the modern pip source 5 years later... > > It's absolutely true that there are cases where incremental builds can > screw things up, especially when using distutils/setuptools. But I > don't think this is why pip does things this way originally :-). > >> It?s not that I don?t trust the backend, it?s that I believe in putting in >> systems that make it harder to do the wrong thing than the right thing. As >> it is now building in place correctly requires the build backend to do extra >> work to ensure that some file that wouldn?t be included in the sdist doesn?t >> influence the build in some way. Given that I?m pretty sure literally every >> build tool in existence for Python currently fails this test, I think that >> is a pretty reasonable statement to say that it might continue to be a >> problem into the future. >> >> Copying the files makes that harder to do (but still easier than always >> going through the sdist). If you want to argue that we should always go >> through the sdist and we shouldn?t have a copy_files hook, I?m ok with that. >> I?m only partially in favor of it as a performance trade off because I think >> it passes a high enough bar that it?s unlikely enough for mistakes to be >> made (and when they do, they?ll be more obvious). > > What do you think of letting build backends opt-in to in-place builds? I think the PEP already allows the interface to be used for in-place builds, though it could use some language specifying that and making sure it?s explicit that is an option the PEP allows. I think I?m neutral on allowing backends to express a preference for in-place builds, but again I don?t think I would be comfortable listening to that to allow a backend to do an in-place build, at least not by default or at first. Could I see us get to a place where we did allow that? Maybe. I wouldn?t promise anything one way or another (and TBH how pip chooses to implement a PEP isn?t really something that the PEP or distutils-sig gets to choose) but I?m fine with leaving the mechanisms in place to allow that in the future. > >>> Other unresolved issues: >>> >>> - Donald had some concerns about get_wheel_metadata and they've led to >>> several suggestions, none of which has made everyone go "oh yeah >>> obviously that's the solution". To me this suggests we should go ahead >>> and drop it from PEP 517 and add it back later if/when the need is >>> more obvious. It's optional anyway, so adding it later doesn't hurt >>> anything. >> >> My main concern is the metadata diverging between the get_wheel_metadata and >> the building wheel phase. The current PEP solves that in a reasonable enough >> way (and in a way I can assert against). My other concerns are mostly just >> little API niggles to make it harder to mess up. >> >> I think this one is important to support because we do not to be able to get >> at the dependencies, and invoking the entire build chain to do that seems >> like it will be extraordinarily slow. > > It's only slow in the case where (a) there's no wheel (obviously), and > (b) after getting the dependencies we decide we don't want to install > this sdist after all. I imagine numpy for example won't bother > implementing get_wheel_metadata because we provide wheels for all the > platforms we support and because we have no dependencies, so it is > doubly useless AFAICT. But yeah in other cases it could matter. I'm > not opposed to including it in general, just thought this might be a > way to help get the minimal PEP 517 out the door. Yea I don?t think this is really a sticking point, I think it?s mostly just around: 1) Do we include a build_sdist hook? A) Can pip use this as part of the process of going from a VCS to a wheel 2) Do we include a copy_the_files hook? 3) Minor decisions like unpacked vs packed wheels/sdists and cwd vs pass in a path. > >>> - It sounds like there's some real question about how exactly a build >>> frontend should handle the output from build_wheel; in particular, the >>> PEP should say what happens if there are multiple files deposited into >>> the output dir. My original idea when writing the PEP was that the >>> build frontend would know the name/version of the wheel it was looking >>> for, and so it would ignore any other files found in the output dir, >>> which would be forward compatible with a future PEP allowing >>> build_wheel to drop multiple wheels into the output dir (i.e., old >>> pip's would just ignore them). It's clear from the discussion that >>> this isn't how others were imagining it. Which is fine, I don't think >>> this is a huge problem, but we should nail it down so we're not >>> surprised later. >> >> >> How do you determine the name/version for ``pip install .`` except by >> running get_wheel_metadata or build_wheel or build_sdist? > > Well, I was imagining that the semantics of 'pip install .' in a > multi-wheel world would be to install all the generated wheels :-). > But yeah, it's not really well-specified as currently written. > > Possibly the simplest solution is to say that build_wheel has to > return a string which names the wheel, and then in the future we could > add build_wheel2 which is identical but returns a list of strings, and > backwards compatibility would be: > > def build_wheel2(...): > return build_wheel(...)[0] That seems reasonable to me. > >> -n >> >>> [1] Donald's suggestion of silently caching intermediate files in some >>> global cache dir is unreasonably difficult to implement in a >>> user-friendly way ? cache management is Hard, and I frankly I still >>> don't think users will accept individual package's build systems >>> leaving hundreds of megabytes of random gunk inside hidden >>> directories. We could debate the details here, but basically, if this >>> were a great idea to do by default, then surely one of >>> cmake/autoconf/... would already do it? Also, my understanding is the >>> main reason pip wants to copy files in the first place is to avoid >>> accidental pollution between different builds using the same local >>> tree; but if a build system implements a global cache like this then >>> surprise, now you can get pollution between arbitrary builds using >>> different trees, or between builds that don't even use a local tree at >>> all (e.g. running 'pip install numpy==1.12.0' can potentially cause a >>> later run of 'pip install numpy==1.12.1' to be corrupted). And, it >>> assumes that all build systems can easily support out-of-tree >>> incremental builds, which is often true but not guaranteed when your >>> wheel build has to wrap some random third party C library's build >>> system. >> >> >> >> Make it opt-in > > If it's opt-in, then I might as well tell people to run 'pip > devinstall .' or 'in-place-install .' or whatever instead, and it'll > be much easier all around. But instead of making it opt-in, I'd much > rather it Just Work. It's frustrating that at the same time we're > moving to the glorious simplified future, we're also picking up a new > piece of arcane wisdom that devs will need to be taught, and another > place where numerical Python devs will roll their eyes at how the > standard Python tooling doesn't care about them. (And I totally > understand that the motivation on your end is also to make things Just > Work, but I feel like in the specific case where someone is > *repeatedly* building out of the *same source directory* ? which is > the one in dispute here ? we should optimize for developer > experience.) People repeatably build out of the same source directory for lots of different reasons. I just closed a ticket the other day about someone who was trying to do a ``-e .`` in a read only directory because they had it mounted inside of a container or a VM or so and were editing it from another machine and it didn?t work (because -e obviously builds in place). One more specific example I?m concerned about with regressions here is a case like: docker run -it ubuntu:latest -v $PWD:/app/ pip install /app/ docker run -it centos:latest -v $PWD:/app/ pip install /app/ I?ve seen people doing similar things in the wild today, and if an in place build directory just uses the normal platform tuples to differentiate build directories, there is a good chance that is going to fail miserably, likely with some sort of god awful linking error or segfault. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Jun 3 04:59:07 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Jun 2017 09:59:07 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: On 3 June 2017 at 03:14, Nathaniel Smith wrote: > So far my belief is that packages with expensive build processes are > going to ignore you and implement, ship, document, and recommend the > direct source-tree->wheel path for developer builds. You can force the > make-a-wheel-from-a-directory-without-copying-and-then-install-it > command have a name that doesn't start with "pip", but it's still > going to exist and be used. Why wouldn't it? It's trivial to implement > and it works, and I haven't heard any alternative proposals that have > either of those properties. [1] I may be misunderstanding you, but it's deeply concerning if you're saying "as a potential backend developer, I'm sitting here listening to the discussion about PEP 517 and I've decided not to raise my concerns but simply to let it be implemented and then ignore it". OTOH, I'm not sure how you plan on ignoring it - are you suggesting that projects like numpy won't support "pip install numpy" except for wheel installs[1]? > One thing that's not clear to me: a crucial use case for sdists is (1) > download, (2) unpack, (3) patch the source, possibly adding new files, > (4) build and install. (After all, the whole reason we insist on > distributing sdists is that open source software should be modifiable > by the recipient.) Does flit currently support this, given the > reliance on VCS metadata? That's a reasonable concern, and a reservation I have about flit's sdist support, but as a comment about flit, it probably belongs more on the flit tracker than here. As a point for PEP 517, I think it's valid though. I'd suggest that the PEP add a few more lines on what constitutes a "source tree", by offering some examples. It seems to me that the two key examples of a "source tree" that the PEP must support are 1. A VCS checkout of a project under development. 2. An unpacked sdist that the user has made edits to before building. (The second of these being the one we're talking about here). Paul [1] By "won't support" consider that in a PEP 517 world, pip issues stating "pip install takes too long" or similar will be passed to the backend developer with the suggestion that they implement the build_sdist or copy_files hook. Saying numpy won't support PEP 517 means that such requests will be denied. From p.f.moore at gmail.com Sat Jun 3 05:12:41 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Jun 2017 10:12:41 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: On 3 June 2017 at 09:59, Paul Moore wrote: > On 3 June 2017 at 03:14, Nathaniel Smith wrote: >> So far my belief is that packages with expensive build processes are >> going to ignore you and implement, ship, document, and recommend the >> direct source-tree->wheel path for developer builds. You can force the >> make-a-wheel-from-a-directory-without-copying-and-then-install-it >> command have a name that doesn't start with "pip", but it's still >> going to exist and be used. Why wouldn't it? It's trivial to implement >> and it works, and I haven't heard any alternative proposals that have >> either of those properties. [1] > > I may be misunderstanding you, but it's deeply concerning if you're > saying "as a potential backend developer, I'm sitting here listening > to the discussion about PEP 517 and I've decided not to raise my > concerns but simply to let it be implemented and then ignore it". > OTOH, I'm not sure how you plan on ignoring it - are you suggesting > that projects like numpy won't support "pip install numpy" except for > wheel installs[1]? > >> One thing that's not clear to me: a crucial use case for sdists is (1) >> download, (2) unpack, (3) patch the source, possibly adding new files, >> (4) build and install. (After all, the whole reason we insist on >> distributing sdists is that open source software should be modifiable >> by the recipient.) Does flit currently support this, given the >> reliance on VCS metadata? > > That's a reasonable concern, and a reservation I have about flit's > sdist support, but as a comment about flit, it probably belongs more > on the flit tracker than here. > > As a point for PEP 517, I think it's valid though. I'd suggest that > the PEP add a few more lines on what constitutes a "source tree", by > offering some examples. It seems to me that the two key examples of a > "source tree" that the PEP must support are > > 1. A VCS checkout of a project under development. > 2. An unpacked sdist that the user has made edits to before building. > > (The second of these being the one we're talking about here). > > Paul > > [1] By "won't support" consider that in a PEP 517 world, pip issues > stating "pip install takes too long" or > similar will be passed to the backend developer with the suggestion > that they implement the build_sdist or copy_files hook. Saying numpy > won't support PEP 517 means that such requests will be denied. Apologies, gmail's web interface split the conversation thread so I wrote this before reading through to the end, so some of what I say is out of date given subsequent emails. Bad gmail, sorry :-) Paul From p.f.moore at gmail.com Sat Jun 3 05:29:42 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Jun 2017 10:29:42 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <7F40EDDB-51AE-4F7D-B672-4B79A45B4134@stufft.io> Message-ID: On 3 June 2017 at 08:47, Donald Stufft wrote: > > That also means that we can adjust our answer to it in the future. If such a > tool gets built and a lot of people end up using it and asking for it in > pip, we can revisit that decision in a future version of pip. Part of the > stand off here is the pip developers view it as a regression if we stop > building in isolation and you view it as a regression if incremental/inplace > builds are not supported. Both can be true! It?s the opinion of the pip > developers who have spoken so far that for us, the risk of our regressions > is high enough we don?t currently feel comfortable changing that behavior. In summary (for my own benefit as much as anything): Currently: 1. pip provides out-of-tree builds for directories 2. the backend (setup.py) provides in-place builds Under PEP 517: 1. pip provides out-of-tree builds for directories (optimised via the build_sdist and or copy_files hooks) 2. the backend (see below) provides in-place builds The PEP 517 backend for setup.py builds may be something new, or it may be setup.py plus some "support legacy source trees" code in pip. It largely depends on who wants to write and maintain PEP 517 adapter code for setup.py. That's not clear to me yet. OTOH, numpy retaining setup.py and relying on legacy support for that format will work for some time yet, so there's no rush to move to a new backend. Future: 1. the numpy developers can make a case for a new pip feature, to do in-place builds, PEP 517 has the build_wheel hook that allows this to be implemented Did I miss anything? Based on this, in-place builds don't seem like they are a big deal (as long as we can agree that "pip doesn't provide in-place builds" isn't a huge issue that's suddenly appeared). Paul From thomas at kluyver.me.uk Sat Jun 3 05:45:40 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 03 Jun 2017 10:45:40 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> On Sat, Jun 3, 2017, at 03:14 AM, Nathaniel Smith wrote: > If the pip devs don't trust build systems in general, but (as > suggested by copy_files discussion) are ok with trusting them if they > promise to be super trustworthy, alternate proposal: > - add a 'in_place_build_safe = True' hook, which indicates that the > build system has been carefully written so that this will generate the > same result as building an sdist and then building that; pip checks > for this to decide whether to build in place or to build an sdist > first. I would use this for flit if it becomes part of the spec. I can see the rationale for not trusting build systems from the frontend's point of view, but it does feel like all potential build systems are being subjected to the same constraints we need for distutils/setuptools. > One thing that's not clear to me: a crucial use case for sdists is (1) > download, (2) unpack, (3) patch the source, possibly adding new files, > (4) build and install. (After all, the whole reason we insist on > distributing sdists is that open source software should be modifiable > by the recipient.) Does flit currently support this, given the > reliance on VCS metadata? Flit does support that, so long as step 4 never needs to build an sdist. Producing the sdist is the only operation for which flit needs the VCS. This is why I'm doggedly arguing that building and installing should be possible without invoking any 'sdist' hook. ;-) Thomas From p.f.moore at gmail.com Sat Jun 3 05:47:09 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Jun 2017 10:47:09 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> <1496437082.4045245.997054632.7F83B2AF@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 04:53, C Anthony Risinger wrote: > I want to make sure I understand what I'd need to do, as a user, in a post > PEP 517 world. Say I wanted to accomplish the following three things: > > * Generate version info from my VCS > * Generate .h and .c from .pyx or cffi's out-of-line API mode > * Use an alternative build process for a platform distutils isn't behaving > with (ignore this requirement if it leads to problems answering, we can just > as well assume distutils here) > > Since I can only define one entrypoint for build-system.build-backend, I > can't point directly at setuptools_scm (or equiv) as it only achieves the > first, I can't point directly at cython or cffi (or equiv) as it only > achieves the second, and I can't point directly at my-custom-backend or > distutils as it only achieves the third... right? If I added setuptools_scm > and cffi to build-system.requires, will it have an opportunity to somehow do > something useful before my-custom-backend begins its work? I agree, this isn't something that's been clearly set out yet. Obviously setuptools support isn't going away any time soon, so you can just carry on as you do now - but I understand that's not what you're asking. Let's assume there's a new PEP 517 backend, call it setuptools_tng, that handles building C files using the platform compiler. That's basically the replacement for setup.py. Then it would be down to that backend to provide a means of hooking into setuptools_scm and cython, much like at the moment you specify that you're using them in setup.py. Obviously, you might want to use setuptools_scm with other backends. In that case, maybe backend developers would need to come up with a unified plugin system. But it could just be a case of saying to the flit (or other backend) developers, "please add support for setuptools_scm". So, basically the answer is that this is an open question for the new backend development ecosystem. But it's not something PEP 517 needs to take a view on. Hope that helps, Paul From p.f.moore at gmail.com Sat Jun 3 05:55:45 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Jun 2017 10:55:45 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 10:45, Thomas Kluyver wrote: >> One thing that's not clear to me: a crucial use case for sdists is (1) >> download, (2) unpack, (3) patch the source, possibly adding new files, >> (4) build and install. (After all, the whole reason we insist on >> distributing sdists is that open source software should be modifiable >> by the recipient.) Does flit currently support this, given the >> reliance on VCS metadata? > > Flit does support that, so long as step 4 never needs to build an sdist. > Producing the sdist is the only operation for which flit needs the VCS. > > This is why I'm doggedly arguing that building and installing should be > possible without invoking any 'sdist' hook. ;-) This is getting very off-topic, but what if I wanted to patch the source and then build a sdist to put into my local PyPI index? I presume the answer is that I either have to checkout the original sources from VCS or I have to build only wheels and maintain my source patches some other way. I can think of realistic reasons why neither of these 2 options are practical, but it is of course your prerogative to not support those cases. Also, I typically have a lot of stuff (working notes, utility scripts, etc) checked into VCS that I don't want to be in the sdist. I don't know if flit has a way to exclude such files - and if it does, why can't it use that method to also allow me to say "exclude everything *except* this list of files" if I want to? This is basically why I'm persistently unable to see why you won't even consider a fallback for building sdists when the VCS isn't present. Paul From thomas at kluyver.me.uk Sat Jun 3 06:09:14 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 03 Jun 2017 11:09:14 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> Message-ID: <1496484554.1563620.997434072.526F619E@webmail.messagingengine.com> On Sat, Jun 3, 2017, at 10:55 AM, Paul Moore wrote: > This is getting very off-topic, but what if I wanted to patch the > source and then build a sdist to put into my local PyPI index? I > presume the answer is that I either have to checkout the original > sources from VCS or I have to build only wheels and maintain my source > patches some other way. I can think of realistic reasons why neither > of these 2 options are practical, but it is of course your prerogative > to not support those cases. Another alternative would be to unpack the sdist, carefully modify the files you want, and tar it back up manually. If this is needed a lot, it shouldn't be hard to write tools to help with this. I'm not going to claim this is a perfect system, but I prefer it to every alternative I've thought of so far. ;-) > Also, I typically have a lot of stuff (working notes, utility scripts, > etc) checked into VCS that I don't want to be in the sdist. I don't > know if flit has a way to exclude such files - and if it does, why > can't it use that method to also allow me to say "exclude everything > *except* this list of files" if I want to? One thing you could do is to have a subfolder containing the package as you want it to be in an sdist, including the flit.ini/pyproject.toml file. When flit makes an sdist, it will only use the folder where that file is found - so that you can e.g. have multiple packages in one repository. More generally, though, I'd question why you don't want those files to be in an sdist? Why should an sdist be any different to a snapshot of your VCS at release time, including all of your thoughts and tools used in development? Installation will usually use a wheel, so download size shouldn't be a major concern. I might consider adding an additional way to exclude files from an sdist, but I'll leave it a while to see if a compelling need emerges before adding more complexity. Thomas From ncoghlan at gmail.com Sat Jun 3 06:38:47 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 20:38:47 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 19:45, Thomas Kluyver wrote: > On Sat, Jun 3, 2017, at 03:14 AM, Nathaniel Smith wrote: >> If the pip devs don't trust build systems in general, but (as >> suggested by copy_files discussion) are ok with trusting them if they >> promise to be super trustworthy, alternate proposal: >> - add a 'in_place_build_safe = True' hook, which indicates that the >> build system has been carefully written so that this will generate the >> same result as building an sdist and then building that; pip checks >> for this to decide whether to build in place or to build an sdist >> first. > > I would use this for flit if it becomes part of the spec. I can see the > rationale for not trusting build systems from the frontend's point of > view, but it does feel like all potential build systems are being > subjected to the same constraints we need for distutils/setuptools. Think of it in terms of fire doors in a building: those exist not because we expect all buildings to always be on fire, but because if there *is* a fire, we want the building to be set up to limit the ability of smoke and the fire itself to spread. Breaking up a build pipeline is a similar notion: every stage of the process is a potential location for bugs, but if you clearly delineate the steps, you get better natural debugging support simply based on the ability to identify which step in the pipeline is failing. If there's only one "build it" hook (e.g. "python setup.py install"), then folks debugging an installation failure are completely dependent on the build backend for their ability to diagnose what's going on. By contrast, if there's a certain process breakdown that backend developers are obliged to support, then it raises the minimum required level of debuggability, since it allows a frontend to execute the build step-by-step and go: 1. Did the sdist generation or build tree extraction work? Yes, good. 2. Did the wheel metadata generation work? Yes, good. 3. Did the actual wheel build work? Yes, good. If any one of those steps fails, the frontend (and hence the frontend's users) naturally have more information than if the only command available was "Build a wheel directly from this VCS checkout". That kind of thing is less important for publishers and application integrators (the majority of whom are dealing with dozens or maybe hundreds of projects at most) than it is for platform integrators (who are more likely to be dealing with thousands or tens of thousands of different projects, and hence benefit more from systematic improvements in fault isolation support). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 3 06:44:22 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Jun 2017 20:44:22 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496484554.1563620.997434072.526F619E@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> <1496484554.1563620.997434072.526F619E@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 20:09, Thomas Kluyver wrote: > On Sat, Jun 3, 2017, at 10:55 AM, Paul Moore wrote: >> This is getting very off-topic, but what if I wanted to patch the >> source and then build a sdist to put into my local PyPI index? I >> presume the answer is that I either have to checkout the original >> sources from VCS or I have to build only wheels and maintain my source >> patches some other way. I can think of realistic reasons why neither >> of these 2 options are practical, but it is of course your prerogative >> to not support those cases. > > Another alternative would be to unpack the sdist, carefully modify the > files you want, and tar it back up manually. If this is needed a lot, it > shouldn't be hard to write tools to help with this. > > I'm not going to claim this is a perfect system, but I prefer it to > every alternative I've thought of so far. ;-) This is why I haven't understood your insistence that flit can't support sdist export from an sdist tree: if PKG-INFO is already present, there's nothing to generate, just copy the entire directory again. If any modifications have been made they'll get captured automatically. I like the idea of backends being able to modify PKG-INFO to indicate that the sdist has been modified since the original export (if they want to do so), so it does make sense to me to leave that check up to the backend, rather than telling frontends to only call the sdist export if PKG-INFO is missing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Jun 3 06:46:15 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Jun 2017 11:46:15 +0100 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: <1496484554.1563620.997434072.526F619E@webmail.messagingengine.com> References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <1496483140.1558584.997425424.3199DA56@webmail.messagingengine.com> <1496484554.1563620.997434072.526F619E@webmail.messagingengine.com> Message-ID: On 3 June 2017 at 11:09, Thomas Kluyver wrote: > More generally, though, I'd question why you don't want those files to > be in an sdist? Why should an sdist be any different to a snapshot of > your VCS at release time, including all of your thoughts and tools used > in development? Installation will usually use a wheel, so download size > shouldn't be a major concern. Because that's how my project is set up, and because that's how I tend to work. Honestly, I'm not trying to debate what's right or wrong here, just pointing out that flit feels to me pretty opinionated on this matter, and its opinions don't match mine. Sadly, that means that I don't use flit - even though as a tool for building wheels, it's ideal for me. Maybe one day I'll make a PR, or fork flit for my own use - it's so close that doing so seems easier than writing a competing tool. In the context of *this* discussion, it does mean that your arguments about not wanting to support a general "build a sdist" hook come across to me as "because I don't want to work that way" rather than as objective constraints. But it's not that important now, as we seem to have come to a compromise on what the PEP requires anyway. Paul From pepoluan at gmail.com Thu Jun 1 23:46:34 2017 From: pepoluan at gmail.com (Pandu Poluan) Date: Fri, 2 Jun 2017 10:46:34 +0700 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: Message-ID: +1 for transitive trust. At the base/simplest level, `pip` would trust any packages trusted by PyPI. More advanced users / more security-oriented installation can add additional "required trusts". Maybe another special "PyPI Curator" pseudo-user. All packages whose signing key is trusted by PyPI *and* PyPI Curator can be deemed trustworthy. And if in a highly secure environment, probably internal curators. Which means that installation of packages will require three (or more) trusts: PyPI, PyPI Curator, curator-1 at example.com, curator-2 at example.com, etc. (The relationship need not be simple boolean AND, but can also be implemented as a score system. For examply, PyPI has weight 0.5, PyPI Curator has weight 1.0, internal company curators have weights 2.0 (> PyPI + PyPI Curator), and minimum acceptable score is 5.5, meaning that the package must be trusted by PyPI, PyPI Curator, and at least 2 internal company curators.) We can even create multiple levels of "PyPI Curator": * PyPI Trusted Authors -- automagically trust well-known 'authors' * PyPI Voted Trust -- packages voted by a committee (or by minimum N users) to be trustworthy * PyPI Audited Trust -- packages that had gone through a more thorough code audit / code review Rgds, -- FdS Pandu E Poluan ~ IT Optimizer ~ ? LOPSA Member #15248 ? Blog : http://pandu.poluan.info/blog/ ? Linked-In : http://id.linkedin.com/in/pepoluan On Fri, Jun 2, 2017 at 9:33 AM, Matt Joyce wrote: > I was more pushing for the transitive trust element than signing. That > being said, any signing at all would be progress. > > On Jun 1, 2017 9:07 PM, "Donald Stufft" wrote: > > > On Jun 1, 2017, at 8:15 PM, Matt Joyce wrote: > > Or start doing signed pgp for package maintainers and build a transitive > trust model. > > > > PGP is not useful for our use case except as a generic crypto primitive, > and there are better generic crypto primitives out there. See > https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ > > > ? > Donald Stufft > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Jun 3 13:01:23 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 3 Jun 2017 12:01:23 -0500 Subject: [Distutils] Malicious packages on PyPI In-Reply-To: References: Message-ID: On Thu, Jun 1, 2017 at 10:46 PM, Pandu Poluan wrote: > +1 for transitive trust. > > At the base/simplest level, `pip` would trust any packages trusted by PyPI. > > More advanced users / more security-oriented installation can add > additional "required trusts". > > Maybe another special "PyPI Curator" pseudo-user. All packages whose > signing key is trusted by PyPI *and* PyPI Curator can be deemed trustworthy. > Ha! "Implement "hook" support for package signature verification" https://github.com/pypa/pip/issues/1035 - TUF (The Update Framework) - (How does this work with DevPi / mirroring?) - https://github.com/pypa/pip/issues/1035#issuecomment-302766580 - re: blockchain, **blockcerts** (because which keyserver is up right now?) Note that: - Projects define { install_requires, extras_require, } in setup.py - setup.py is not declarative: you must execute setup.py to determine the value of install_requires for the current platform - so, PyPi **can't** know which dependencies setup.py might list when run on a given machine - this essentially requires recursive eval of remote code (it must run each and every setup.py) - which is basically the exercise: download lots of executable source from hopefully trusted third parties over a hopefully secure channel - this is relevant to a trust framework because the whole objective would be to "maximally verify" the trust chain for the whole dependency graph; but the dependency graph isn't known until after things are unpacked and executed - ... which eventually brings us to: "we could/should teach devs to sign VCS commits and releases (e.g. before they're repackaged, patche, and then signed again by e.g. a linux distro) - Retrieiving the dependency metadata for a project which MAY define conditional dependencies in a setup.py does require executing the setup.py (which does indeed complicate efforts to solve a dependency graph (and a trust graph) > > And if in a highly secure environment, probably internal curators. Which > means that installation of packages will require three (or more) trusts: > PyPI, PyPI Curator, curator-1 at example.com, curator-2 at example.com, etc. > So, if this (curator approval / LikeAction / AssessAction ) does or does not occur, where should the metadata regarding who and why be stored? From http://markmail.org/search/?q=list%3Aorg.python+LikeAction#query:list%3Aorg.python%20LikeAction+page:1+mid:e2xbsb2guxagtshc+state:results In terms of schema.org, a Django Packages resource has: * [ ] a unique URI > * [ ] typed features (predicates with ranges) > * [ ] http://schema.org/review > * [ ] http://schema.org/VoteAction > * [ ] http://schema.org/LikeAction - https://schema.org/AssessAction - https://schema.org/ReviewAction ... "curated" - "[Distutils] Maintaining a curated set of Python packages" http://markmail.org/search/?q=list%3Aorg.python+curated#" query:list%3Aorg.python%20curated+page:1+mid:ibnoqnovjxp3gavi+state:results > > (The relationship need not be simple boolean AND, but can also be > implemented as a score system. For examply, PyPI has weight 0.5, PyPI > Curator has weight 1.0, internal company curators have weights 2.0 (> PyPI > + PyPI Curator), and minimum acceptable score is 5.5, meaning that the > package must be trusted by PyPI, PyPI Curator, and at least 2 internal > company curators.) > > We can even create multiple levels of "PyPI Curator": > > * PyPI Trusted Authors -- automagically trust well-known 'authors' > * PyPI Voted Trust -- packages voted by a committee (or by minimum N > users) to be trustworthy > * PyPI Audited Trust -- packages that had gone through a more thorough > code audit / code review > This data would be most useful if it could be merged into one graph and linked to a stable per-package URI. > > > Rgds, > -- > > > FdS Pandu E Poluan > ~ IT Optimizer ~ > > ? LOPSA Member #15248 > ? Blog : http://pandu.poluan.info/blog/ > ? Linked-In : http://id.linkedin.com/in/pepoluan > > On Fri, Jun 2, 2017 at 9:33 AM, Matt Joyce wrote: > >> I was more pushing for the transitive trust element than signing. That >> being said, any signing at all would be progress. >> >> On Jun 1, 2017 9:07 PM, "Donald Stufft" wrote: >> >> >> On Jun 1, 2017, at 8:15 PM, Matt Joyce wrote: >> >> Or start doing signed pgp for package maintainers and build a transitive >> trust model. >> >> >> >> PGP is not useful for our use case except as a generic crypto primitive, >> and there are better generic crypto primitives out there. See >> https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ >> >> >> ? >> Donald Stufft >> >> >> >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Sat Jun 3 20:09:45 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Sat, 3 Jun 2017 19:09:45 -0500 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> <1496437082.4045245.997054632.7F83B2AF@webmail.messagingengine.com> Message-ID: On Jun 3, 2017 4:47 AM, "Paul Moore" wrote: On 3 June 2017 at 04:53, C Anthony Risinger wrote: > I want to make sure I understand what I'd need to do, as a user, in a post > PEP 517 world. Say I wanted to accomplish the following three things: > > * Generate version info from my VCS > * Generate .h and .c from .pyx or cffi's out-of-line API mode > * Use an alternative build process for a platform distutils isn't behaving > with (ignore this requirement if it leads to problems answering, we can just > as well assume distutils here) [...] So, basically the answer is that this is an open question for the new backend development ecosystem. But it's not something PEP 517 needs to take a view on. Fair enough. It seems like there will almost certainly emerge some way of chaining small "source tree mutators" (leading to an sdist) with truly custom build backends (that may ultimately terminate on either setuptools/distutils like you mention, or a completely separate toolchain [I want to say something like waf could be this alternate]). This wrapper/pipeline layer could be baked into pip/flit/whatever as a plugin system, but ideally it would just be a small and blessed pypa tool I'd think... then I suppose to make use of multiple transformations, a user would pass a list of actual build-system backends via tool.BLESSED-CHAINER-APP.build-backends in pyproject.toml or something. Is it unreasonable to request right now that build-system.build-backend be a repeatable key in pyproject.toml? Then I could just list them in order. Might be easy to add later without breakage though too. As long as all backends understand the hard separation between build_sdist "prepare the redistributable source tree" and build_wheel "construct an installable" they can be called in proper order and in phases. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Jun 3 20:36:17 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 3 Jun 2017 17:36:17 -0700 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> <84555DC0-43DF-4386-9DB7-4A55147BEFDF@stufft.io> <1496437082.4045245.997054632.7F83B2AF@webmail.messagingengine.com> Message-ID: On Sat, Jun 3, 2017 at 5:09 PM, C Anthony Risinger wrote: > Fair enough. It seems like there will almost certainly emerge some way of > chaining small "source tree mutators" (leading to an sdist) with truly > custom build backends (that may ultimately terminate on either > setuptools/distutils like you mention, or a completely separate toolchain [I > want to say something like waf could be this alternate]). > > This wrapper/pipeline layer could be baked into pip/flit/whatever as a > plugin system, but ideally it would just be a small and blessed pypa tool > I'd think... then I suppose to make use of multiple transformations, a user > would pass a list of actual build-system backends via > tool.BLESSED-CHAINER-APP.build-backends in pyproject.toml or something. > > Is it unreasonable to request right now that build-system.build-backend be a > repeatable key in pyproject.toml? Then I could just list them in order. > Might be easy to add later without breakage though too. > > As long as all backends understand the hard separation between build_sdist > "prepare the redistributable source tree" and build_wheel "construct an > installable" they can be called in proper order and in phases. That's a neat idea, but I'd prefer that this be handled by someone (you?) writing a metabuild system that implements whatever layering logic they prefer and putting it up on pypi, so it can be used like: [build-system] build-backend = metabuild [tool.metabuild] backends = ["backend1", "backend2", "backend3"] As you can see, standards are hard; it takes hundreds of emails to standardize the API of a function that just takes a source tree and outputs a wheel :-). So the idea of PEP 517 is to push as much actual semantics as possible into the build backend, where they can be easily changed and evolved and experimented with. It's not at all obvious to me right now whether monolithic or plugin-style build backends will predominate, and it's also not at all obvious to me how *exactly* you're imagining the layering/phases would work, but the great thing is that PEP 517 is already flexible enough to let you do what you want without having to convince me first :-) -n -- Nathaniel J. Smith -- https://vorpus.org From ralf.gommers at gmail.com Sat Jun 3 20:39:45 2017 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Sun, 4 Jun 2017 12:39:45 +1200 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: On Sat, Jun 3, 2017 at 8:59 PM, Paul Moore wrote: > On 3 June 2017 at 03:14, Nathaniel Smith wrote: > > So far my belief is that packages with expensive build processes are > > going to ignore you and implement, ship, document, and recommend the > > direct source-tree->wheel path for developer builds. You can force the > > make-a-wheel-from-a-directory-without-copying-and-then-install-it > > command have a name that doesn't start with "pip", but it's still > > going to exist and be used. Why wouldn't it? It's trivial to implement > > and it works, and I haven't heard any alternative proposals that have > > either of those properties. [1] > > I may be misunderstanding you, but it's deeply concerning if you're > saying "as a potential backend developer, I'm sitting here listening > to the discussion about PEP 517 and I've decided not to raise my > concerns but simply to let it be implemented and then ignore it". > I think you partly misunderstood - "ignore you" should mean "ignore pip" or "ignore the mandatory sdist part of PEP 517" not "ignore all of PEP 517". And concerns have been raised (just rejected as less important than the possibility of more bug reports to pip)? And I agree with Nathaniel's view in the paragraph above. > OTOH, I'm not sure how you plan on ignoring it - are you suggesting > that projects like numpy won't support "pip install numpy" except for > wheel installs[1]? > Of course not, that will always be supported. It's just that where the developer/build docs now say "python setup.py ..." we want them to say "pip install . -v" and with sdist generation that won't happen - they will instead say "somenewtool install ." where somenewtool is a utility that does something like: 1. invoke backend directly to build a wheel (using PEP 517 build_wheel interface) 2. install the wheel with pip and probably also 1. invoke backend directly for in-place build 2. make the inplace build visible (may involve telling pip to uninstall the project if it's installed elsewhere, and messing with PYTHONPATH or pip metadata) Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 3 23:28:48 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Jun 2017 13:28:48 +1000 Subject: [Distutils] Provisionally accepting PEP 517's declarative build system interface In-Reply-To: References: <43CEE2F8-3115-44AE-84BF-256430C26492@stufft.io> <3B482B33-2FA3-42A6-8616-DFF545DE202F@stufft.io> <93245B80-BCB4-402D-9EA3-7D7684DA7C44@stufft.io> <9A657E25-BD4E-435B-9917-7A0A5212D7AE@stufft.io> <1496237945.428843.994137600.0F49BA4A@webmail.messagingengine.com> <1496239217.434572.994170256.65D1515E@webmail.messagingengine.com> <1496239512.435685.994175464.27BB5FD6@webmail.messagingengine.com> <0CFA4795-02E6-4574-A3E1-DC1ECFE0FB05@stufft.io> <9F734CDB-59D4-4805-8267-FFE8C9101C6F@stufft.io> <1496257947.754070.994532544.49214D71@webmail.messagingengine.com> <8B7C8C8F-3CEB-4AE1-832D-58FA7E1C5001@stufft.io> <1496259927.763203.994563560.15C47384@webmail.messagingengine.com> <1496265209.1644205.994657696.35CDED47@webmail.messagingengine.com> <1496410950.1365928.996612800.6564664A@webmail.messagingengine.com> <74077720-267D-4A21-A4CF-CCC3F3F3C367@stufft.io> Message-ID: On 4 June 2017 at 10:39, Ralf Gommers wrote: > > > On Sat, Jun 3, 2017 at 8:59 PM, Paul Moore wrote: >> >> On 3 June 2017 at 03:14, Nathaniel Smith wrote: >> > So far my belief is that packages with expensive build processes are >> > going to ignore you and implement, ship, document, and recommend the >> > direct source-tree->wheel path for developer builds. You can force the >> > make-a-wheel-from-a-directory-without-copying-and-then-install-it >> > command have a name that doesn't start with "pip", but it's still >> > going to exist and be used. Why wouldn't it? It's trivial to implement >> > and it works, and I haven't heard any alternative proposals that have >> > either of those properties. [1] >> >> I may be misunderstanding you, but it's deeply concerning if you're >> saying "as a potential backend developer, I'm sitting here listening >> to the discussion about PEP 517 and I've decided not to raise my >> concerns but simply to let it be implemented and then ignore it". > > > I think you partly misunderstood - "ignore you" should mean "ignore pip" or > "ignore the mandatory sdist part of PEP 517" not "ignore all of PEP 517". > And concerns have been raised (just rejected as less important than the > possibility of more bug reports to pip)? > > And I agree with Nathaniel's view in the paragraph above. > >> >> OTOH, I'm not sure how you plan on ignoring it - are you suggesting >> that projects like numpy won't support "pip install numpy" except for >> wheel installs[1]? > > > Of course not, that will always be supported. It's just that where the > developer/build docs now say "python setup.py ..." we want them to say "pip > install . -v" and with sdist generation that won't happen - they will > instead say "somenewtool install ." Allowing software consumption frontends like pip to completely replace setup.py as a *development* tool isn't one of the goals of PEP 517 - while we want to enable the use of both simpler (e.g flit) and more capable (e.g. Scons, meson, waf) alternatives without breaking end user installation processes and other tooling, it's also entirely acceptable for projects that are happy with setuptools/distutils as their build toolchain to keep using it. The subtitle of the packaging panel at PyCon US 2013 ended up being "'./setup.py install' must die", *not* "./setup.py must die" or "./setup.py sdist must die", and that focus specifically on software consumption wasn't an accident. Personally, I'm thoroughly opposed to commingling software publishing tasks and software consumption tasks in the same toolchain the way distutils/setuptools do (since you can't optimise for the needs of both software publishers *and* software consumers at the same time outside relatively narrow domains like the Chandler plugin system PJE originally wrote setuptools to handle), so I think the mismatched perspectives here may be due at least in part to my failing to communicate that adequately. When it appears otherwise, it's because I'm conceding that a "pip publish" front-end to a software publishing tool like twine may be beneficial for discoverability and learning purposes when folks are making the leap from "open source consumer" to "open source publisher", and because "retrieve & unpack sdist, apply downstream patches as needed, build & publish wheel" is a fairly common step in automated build pipelines that inherently blurs the line between software consumption and software publication :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From robin at reportlab.com Mon Jun 5 05:00:52 2017 From: robin at reportlab.com (Robin Becker) Date: Mon, 5 Jun 2017 10:00:52 +0100 Subject: [Distutils] manylinux1 testing Message-ID: <06727ac2-1d43-ba75-c643-f9eadec340fe@chamonix.reportlab.co.uk> Are there docker images suitable for testing manylinux packages. I am guessing the manylinux1 images might be suitable, provided I drop all building of libraries inside them and then just use the pythons I want to test against. Reason I'm asking is that I noticed that I was building images where the final grafting didn't seem to be happening I assume something must have been wrong with the repair_wheehouse command. My manylinux1 images were 4 weeks old. The situation improved after I updated today. However, I have been testing these 'bad' wheels and they all appeared to work. Problem is that I'm only bundling freetype, libpng and libz and I guess my test systems all have those. So is it reasonable for me to use the manylinux1 images as a null linux so I can pip load my stuff from the local file system and test it all runs on a system with almost no libraries built in? -- Robin Becker From matthew.brett at gmail.com Mon Jun 5 05:34:08 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 5 Jun 2017 10:34:08 +0100 Subject: [Distutils] manylinux1 testing In-Reply-To: <06727ac2-1d43-ba75-c643-f9eadec340fe@chamonix.reportlab.co.uk> References: <06727ac2-1d43-ba75-c643-f9eadec340fe@chamonix.reportlab.co.uk> Message-ID: Hi, On Mon, Jun 5, 2017 at 10:00 AM, Robin Becker wrote: > Are there docker images suitable for testing manylinux packages. I am > guessing the manylinux1 images might be suitable, provided I drop all > building of libraries inside them and then just use the pythons I want to > test against. > > Reason I'm asking is that I noticed that I was building images where the > final grafting didn't seem to be happening I assume something must have been > wrong with the repair_wheehouse command. > > My manylinux1 images were 4 weeks old. The situation improved after I > updated today. > > However, I have been testing these 'bad' wheels and they all appeared to > work. Problem is that I'm only bundling freetype, libpng and libz and I > guess my test systems all have those. > > So is it reasonable for me to use the manylinux1 images as a null linux so I > can pip load my stuff from the local file system and test it all runs on a > system with almost no libraries built in? You could do that, they will test whether the wheels run a minimal case in terms of versions and dependencies. You may hit inconveniences building packages, because the toolchains are rather old. Alternatively you could use a bare but more recent distribution. For example, my multibuild framework for manylinux testing uses manylinux containers to build with, and bare Ubuntu Trusty images for testing: https://github.com/matthew-brett/multibuild https://hub.docker.com/r/matthewbrett/trusty Cheers, Matthew From robin at reportlab.com Mon Jun 5 07:11:12 2017 From: robin at reportlab.com (Robin Becker) Date: Mon, 5 Jun 2017 12:11:12 +0100 Subject: [Distutils] manylinux1 testing In-Reply-To: References: <06727ac2-1d43-ba75-c643-f9eadec340fe@chamonix.reportlab.co.uk> Message-ID: On 05/06/2017 10:34, Matthew Brett wrote: > Hi, > ........... > > You could do that, they will test whether the wheels run a minimal > case in terms of versions and dependencies. You may hit > inconveniences building packages, because the toolchains are rather > old. > I assumed that everything I need would be in the wheels I have built. The manylinux docker needs to be able to build pythons anyhow so I guess the tools are good enough for that. > Alternatively you could use a bare but more recent distribution. For > example, my multibuild framework for manylinux testing uses manylinux > containers to build with, and bare Ubuntu Trusty images for testing: > > https://github.com/matthew-brett/multibuild > https://hub.docker.com/r/matthewbrett/trusty > > Cheers, > > Matthew > -- Robin Becker From matthew.brett at gmail.com Mon Jun 5 07:23:01 2017 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 5 Jun 2017 12:23:01 +0100 Subject: [Distutils] manylinux1 testing In-Reply-To: References: <06727ac2-1d43-ba75-c643-f9eadec340fe@chamonix.reportlab.co.uk> Message-ID: Hi, On Mon, Jun 5, 2017 at 12:11 PM, Robin Becker wrote: > On 05/06/2017 10:34, Matthew Brett wrote: >> >> Hi, >> > ........... >> >> >> You could do that, they will test whether the wheels run a minimal >> case in terms of versions and dependencies. You may hit >> inconveniences building packages, because the toolchains are rather >> old. >> > I assumed that everything I need would be in the wheels I have built. The > manylinux docker needs to be able to build pythons anyhow so I guess the > tools are good enough for that. Sure - the building I was think of was for any dependencies where you didn't have a built wheel. You'll likely be able to get by with the tools on the Manylinux docker image, but you might have to pay attention to stuff like which version of cmake you have and so on. Cheers, Matthew From catafest at yahoo.com Wed Jun 7 10:48:09 2017 From: catafest at yahoo.com (Catalin George Festila) Date: Wed, 7 Jun 2017 14:48:09 +0000 (UTC) Subject: [Distutils] PyQT python module . References: <1703035759.3680235.1496846889367.ref@mail.yahoo.com> Message-ID: <1703035759.3680235.1496846889367@mail.yahoo.com> Dear python team . I used python 2.7 and 3.4 version and I saw your simple way Installing Python Modules ? Python 3.6.1 documentation Installing Python Modules ? Python 3.6.1 documentation Hope this mail will help you and your changes will help python users. I think is need to fix some python modules, like: PyQt 4 and 5 with SIP. The main reason is the hard way to install this python modules. Also some python package PyAudio can be install into python 2.7 version but not into 3.4. This python module ( PyAudio ) is need by SpeechRecognition python module. I try also the wheel way , but nothing . For windows I used this : pip-review.exe --auto --verbose and working well with some packages but I got some errors: Rolling back uninstall of mysql-connector Command "C:\Python34\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\mythcat\\AppData\\Local\\Temp\\pip-build-21khqzin\\mysql-connector\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\mythcat\AppData\Local\Temp\pip-80abjmph-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\mythcat\AppData\Local\Temp\pip-build-21khqzin\mysql-connector\ Thank you. Best regards. C?t?lin George Fe?til? https://python-catalin.blogspot.ro/ From brett at python.org Thu Jun 8 15:28:37 2017 From: brett at python.org (Brett Cannon) Date: Thu, 08 Jun 2017 19:28:37 +0000 Subject: [Distutils] PyQT python module . In-Reply-To: <1703035759.3680235.1496846889367@mail.yahoo.com> References: <1703035759.3680235.1496846889367.ref@mail.yahoo.com> <1703035759.3680235.1496846889367@mail.yahoo.com> Message-ID: So this list is about discussing the development of the tools for packaging, not individual packages themselves (e.g. we have nothing to do with PyAudio). If you need help installing a package then I would ask on the python-list or tutor mailing lists. On Wed, 7 Jun 2017 at 07:54 Catalin George Festila via Distutils-SIG < distutils-sig at python.org> wrote: > Dear python team . > > I used python 2.7 and 3.4 version and I saw your simple way Installing > Python Modules ? Python 3.6.1 documentation > > Installing Python Modules ? Python 3.6.1 documentation > > Hope this mail will help you and your changes will help python users. > > I think is need to fix some python modules, like: PyQt 4 and 5 with SIP. > The main reason is the hard way to install this python modules. > Also some python package PyAudio can be install into python 2.7 version > but not into 3.4. > This python module ( PyAudio ) is need by SpeechRecognition python module. > > I try also the wheel way , but nothing . > For windows I used this : > > pip-review.exe --auto --verbose > > and working well with some packages but I got some errors: > > Rolling back uninstall of mysql-connector > Command "C:\Python34\python.exe -u -c "import setuptools, > tokenize;__file__='C:\\Users\\mythcat\\AppData\\Local\\Temp\\pip-build-21khqzin\\mysql-connector\\setup.py';f=getattr(tokenize, > 'open', open)(__file__);code=f.read().replace('\r\n', > '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record > C:\Users\mythcat\AppData\Local\Temp\pip-80abjmph-record\install-record.txt > --single-version-externally-managed --compile" failed with error code 1 in > C:\Users\mythcat\AppData\Local\Temp\pip-build-21khqzin\mysql-connector\ > > > Thank you. > Best regards. > > C?t?lin George Fe?til? > > > https://python-catalin.blogspot.ro/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 9 23:32:02 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 10 Jun 2017 13:32:02 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories Message-ID: Hi folks, Thanks for the recent discussions around the PEP 517 build hooks PEP - I think they've clarified and improved the PEP in several areas and are setting us up well for an interoperability API that is both flexible and straightforward for frontends and backends to support. However, I think they've also raised some new still open questions, which I'd like to discuss specifically in this thread. 1. Should backends create artifacts directly, or simply export their contents, and let frontends assemble them into redistributable archives? 2. Are backends responsible in any way for naming artifacts, or should that responsibility lie entirely with frontends? For the first question, in the latest PEP draft, we have: - the sdist hook is `export_sdist`, with the expected output being a filesystem directory - the metadata hook is `get_wheel_metadata`, with the expected output being the contents of the metadata directory - the wheel hook is `build_wheel`, with the expected output being an already assembled binary wheel archive - the build preparation hook is `prepare_build_files`, with the expected output being the subset of files specfically needed to assemble a full wheel file with `build_wheel` For the second question: - the sdist hook is currently unclear on whether the backend should write the sdist contents directly into the frontend supplied directory, or into a suitably named subdirectory - the metadata hook is explicit that the backend should write the metadata directly into the given directory - the wheel hook is explicit that the backend should create already assembled wheel archives in the given directory - the build preparation hook is explicit that the backend should write the metadata directly into the given directory When considered as a unified specification this seems confusingly inconsistent to me, although understandable given that we started out covering only the well specified wheel format, and eventually expanded it to cover the substantially less well specified notions of sdist archives and temporary build directories. While I'm open to being persuaded otherwise, my current thinking is that these concerns could be suitably addressed by making the following amendments: 1. Move all artifact naming responsibilities to the frontend. All backend hooks would just write files in the defined format into the directory supplied by the frontend (and those directories would have no defined naming scheme - they're just a staging area for the backend to pass content to the frontend) 2. Change the names of the artifact hooks to be "export_sdist_content" and "export_wheel_content" to emphasise that backends are responsible for generating the artifact *contents*, not the artifacts themselves 3. Change the optional metadata preparation hook to be "prepare_wheel_metadata" for consistency with "prepare_build_files". This gives us a consistent naming scheme where the "export_*" hooks are required, but each has an optional "prepare_*" counterpart that backends may implement as a more efficient alternative when the full artifact contents aren't (or may not be) needed. Assuming we add support for building multiple wheels from a single source archive in the future, we'd do this through some kind of declarative API based mechanism like adding a new optional "wheel_name" parameter to `prepare_wheel_metadata` and `export_wheel_content` (to tell the backend which wheel the frontend is interested in), along with an additional field in [build-system] to tell frontends which secondary wheels are defined by that project. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Jun 10 05:05:14 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Jun 2017 10:05:14 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: Message-ID: On 10 June 2017 at 04:32, Nick Coghlan wrote: > > While I'm open to being persuaded otherwise, my current thinking is > that these concerns could be suitably addressed by making the > following amendments: > > 1. Move all artifact naming responsibilities to the frontend. All > backend hooks would just write files in the defined format into the > directory supplied by the frontend (and those directories would have > no defined naming scheme - they're just a staging area for the backend > to pass content to the frontend) > > 2. Change the names of the artifact hooks to be "export_sdist_content" > and "export_wheel_content" to emphasise that backends are responsible > for generating the artifact *contents*, not the artifacts themselves > > 3. Change the optional metadata preparation hook to be > "prepare_wheel_metadata" for consistency with "prepare_build_files". > This gives us a consistent naming scheme where the "export_*" hooks > are required, but each has an optional "prepare_*" counterpart that > backends may implement as a more efficient alternative when the full > artifact contents aren't (or may not be) needed. +1 In particular, I think that by making frontends responsible for packaging and naming wheels and sdists, we centralise responsibility for following those parts of the relevant standards, which is one less thing for (multiple) backends to track. My only reservation is that if anyone is looking to create a PEP 517 wrapper around legacy setuptools/distutils (at the moment, we haven't really touched on how we maintain support for setuptools-based projects, other than an assumed "keep all the code round until no-one uses setuptools any more") then that wrapper will need to build the artifacts as files (because that's what setuptools does), then unpack them only to have the frontend repack them. But that's a relatively minor price to pay. Paul From thomas at kluyver.me.uk Sat Jun 10 05:12:17 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 10 Jun 2017 10:12:17 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: Message-ID: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> On Sat, Jun 10, 2017, at 04:32 AM, Nick Coghlan wrote: > 1. Move all artifact naming responsibilities to the frontend. All > backend hooks would just write files in the defined format into the > directory supplied by the frontend (and those directories would have > no defined naming scheme - they're just a staging area for the backend > to pass content to the frontend) If the 'make a wheel' hook needs to make an unpacked wheel, flit will probably implement this as building a wheel zip file, and then unpacking it into the target directory again. There isn't currently code for building an 'unpacked wheel', and I don't see any of this as performance criticial, since most users will be installing from a pre-built wheel anyway. This is OK if this is what we decide, but I wanted to highlight that at least in flit's case, there's no simplicity or performance benefit from asking for an unpacked wheel. > Assuming we add support for building multiple wheels from a single > source archive in the future, we'd do this through some kind of > declarative API based mechanism like adding a new optional > "wheel_name" parameter to `prepare_wheel_metadata` and > `export_wheel_content` (to tell the backend which wheel the frontend > is interested in), along with an additional field in [build-system] to > tell frontends which secondary wheels are defined by that project. This isn't really my area of expertise, but I would assume that we'd want to ask the backend to build all the wheels it knows about in one shot, not one-by-one. Thomas From ncoghlan at gmail.com Sat Jun 10 12:23:28 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Jun 2017 02:23:28 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> Message-ID: On 10 June 2017 at 19:12, Thomas Kluyver wrote: > On Sat, Jun 10, 2017, at 04:32 AM, Nick Coghlan wrote: >> 1. Move all artifact naming responsibilities to the frontend. All >> backend hooks would just write files in the defined format into the >> directory supplied by the frontend (and those directories would have >> no defined naming scheme - they're just a staging area for the backend >> to pass content to the frontend) > > If the 'make a wheel' hook needs to make an unpacked wheel, flit will > probably implement this as building a wheel zip file, and then unpacking > it into the target directory again. There isn't currently code for > building an 'unpacked wheel', and I don't see any of this as performance > criticial, since most users will be installing from a pre-built wheel > anyway. > > This is OK if this is what we decide, but I wanted to highlight that at > least in flit's case, there's no simplicity or performance benefit from > asking for an unpacked wheel. The fact this is also true for both "setup.py bdist_wheel" and for "enscons" would then be the strongest argument in favour of keeping the current "build_wheel" API: existing backends are already building wheels for publication directly, so it makes sense to standardise on what they're already doing, rather than requiring them to do extra work just to support PEP 517. We also expect even "build for installation" use cases to want the wheel as an archive for caching purposes. The rationale for `export_sdist_content` being inconsistent with that would then be: - the sdist naming and archive format isn't fully specified, so we want to avoid depending on those details - we specifically want `export_sdist_content` to be usable as a fallback for `prepare_build_files` when backends don't define the latter - tools don't tend to cache locally generated sdists the way they do wheel archives That means the only suggested name changes would be: - export_sdist -> export_sdist_content - get_wheel_metadata -> prepare_wheel_metadata The description of `build_wheel` would specifically mention the fact that, unlikely the other hooks, it is the backend's responsibility to actually generate the artifacts. >> Assuming we add support for building multiple wheels from a single >> source archive in the future, we'd do this through some kind of >> declarative API based mechanism like adding a new optional >> "wheel_name" parameter to `prepare_wheel_metadata` and >> `export_wheel_content` (to tell the backend which wheel the frontend >> is interested in), along with an additional field in [build-system] to >> tell frontends which secondary wheels are defined by that project. > > This isn't really my area of expertise, but I would assume that we'd > want to ask the backend to build all the wheels it knows about in one > shot, not one-by-one. Aye, that's how rpmbuild works, which would be another point in favour of keeping the current `build_wheel` API. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sat Jun 10 12:59:13 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 Jun 2017 12:59:13 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> Message-ID: <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> > On Jun 10, 2017, at 12:23 PM, Nick Coghlan wrote: > > The fact this is also true for both "setup.py bdist_wheel" and for > "enscons" would then be the strongest argument in favour of keeping > the current "build_wheel" API: existing backends are already building > wheels for publication directly, so it makes sense to standardise on > what they're already doing, rather than requiring them to do extra > work just to support PEP 517. > > We also expect even "build for installation" use cases to want the > wheel as an archive for caching purposes. We have a few possible cases where the build-the-wheel backend is going to be called: 1) We?re creating a wheel ahead of time to publish somewhere. 2) We?re creating a wheel JIT that we?re going to cache. 3) We?re creating a wheel JIT that we?re not going to cache. For (1) it mostly doesn?t matter if this happens in the front end or the backend, except doing it in the front end allows a small performance benefit in the case that someone is using a frontend that can be told to ?build + upload this wheel?, because the frontend will need to read some files (particularly the METADATA file) to process the upload and if it has the wheel unzipped already it doesn?t need to roundtrip the METADATA file through a zip+unzip process. However this speedup is only minor and it only matters in cases where you have a singular command that does build+upload in a single go. For (2) as the wheel cache is currently implemented it doesn?t matter if the front end does it or not. However, I could see us in the future possibly start caching unzipped wheels rather than zipped wheels. It would provide a speed up on every subsequent install since we would stop having to unzip the file over and over again on each subsequent install. I don?t *know* that we?re going to do that, it would take doing some figuring if storing the contents decompressed (which would take up more space and take up more inodes) is worth the tradeoff of removing that processing on each install. Another possible option along this line is still storing it as zip files, but using ZIP_STORED instead of ZIP_DEFLATE, so that we still keep individual .whl files, but ones that don?t store their members compressed to reduce time spent doing compression. For (3), it allows us to skip work entirely, instead of having the backend zip up the work and then immediately unzip it again in the front end we can just not zip it up to start with. I haven?t explicitly tested it with bdist_wheel, but with the sdist command removing the step where it actually creates the .tar.gz removes the bulk of the processing time of ``python setup.py sdist`` and I would guess that is true for pure python wheels too. Ultimately allowing the front end to do this gives us flexibility, the backend doesn?t know why we?re building the wheel so it can?t make decisions to try and optimize the workflow that it goes through, it can only do the same thing every time. However if we allow the frontend to manage that, then the frontend, which does know why we?re building things, can alter what it does with those results depending on what it expects to be done with them. Keeping it in the backend doesn?t really buy us much of anything, except that a handful of backend authors don?t have to make relatively minor adjustments to their code base. In a vacuum I can?t see any compelling reason to have the backend do the archiving at all and the only reason I think we?re talking about it is that?s just how the backends work now? but again, changing that is almost certainly going to be extremely trivial to do. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 10 13:14:07 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Jun 2017 03:14:07 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> Message-ID: On 11 June 2017 at 02:59, Donald Stufft wrote: > We have a few possible cases where the build-the-wheel backend is going to > be called: > > 1) We?re creating a wheel ahead of time to publish somewhere. > 2) We?re creating a wheel JIT that we?re going to cache. > 3) We?re creating a wheel JIT that we?re not going to cache. [snip] > Ultimately allowing the front end to do this gives us flexibility, the > backend doesn?t know why we?re building the wheel so it can?t make decisions > to try and optimize the workflow that it goes through, it can only do the > same thing every time. However if we allow the frontend to manage that, then > the frontend, which does know why we?re building things, can alter what it > does with those results depending on what it expects to be done with them. > > Keeping it in the backend doesn?t really buy us much of anything, except > that a handful of backend authors don?t have to make relatively minor > adjustments to their code base. In a vacuum I can?t see any compelling > reason to have the backend do the archiving at all and the only reason I > think we?re talking about it is that?s just how the backends work now? but > again, changing that is almost certainly going to be extremely trivial to > do. Thanks for spelling those out - that's the intuition that prompted me to make the suggestion, but without working through the specifics I wasn't confident enough in the idea to specifically request that Thomas make the update. Thomas - I agree with Donald's reasoning here, so would you mind updating the PEP accordingly? I think it's OK if that means that the initial PEP 517 support in the existing backends builds the archive and then unpacks it: - future PEP 517 backends still won't need to include support for their own wheel building - existing backends can do pack-and-unpack if that's easier for them, and defer refactoring until later - some backends may simply never be refactored, and that's between the backend developers and their users - for setuptools, `bdist_wheel` is actually provided by Daniel Holth's wheel project, and that could potentially gain a `bdist_wheel_unpacked` command Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Jun 10 13:31:45 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Jun 2017 18:31:45 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> Message-ID: On 10 June 2017 at 18:14, Nick Coghlan wrote: >> Keeping it in the backend doesn?t really buy us much of anything, except >> that a handful of backend authors don?t have to make relatively minor >> adjustments to their code base. In a vacuum I can?t see any compelling >> reason to have the backend do the archiving at all and the only reason I >> think we?re talking about it is that?s just how the backends work now? but >> again, changing that is almost certainly going to be extremely trivial to >> do. > > Thanks for spelling those out - that's the intuition that prompted me > to make the suggestion, but without working through the specifics I > wasn't confident enough in the idea to specifically request that > Thomas make the update. > > Thomas - I agree with Donald's reasoning here, so would you mind > updating the PEP accordingly? I think it's OK if that means that the > initial PEP 517 support in the existing backends builds the archive > and then unpacks it: > > - future PEP 517 backends still won't need to include support for > their own wheel building Unless they want to act as standalone tools as well as backends - much like flit does. I don't personally have a feel for how other backends will look, will developers want to expose their own UI or will they be happy to act just as a plugin to pip/twine/etc? > - existing backends can do pack-and-unpack if that's easier for them, > and defer refactoring until later > - some backends may simply never be refactored, and that's between the > backend developers and their users Again, it's probably less of a refactoring, and more of a decision about what UI (if any) they want to expose. > - for setuptools, `bdist_wheel` is actually provided by Daniel Holth's > wheel project, and that could potentially gain a > `bdist_wheel_unpacked` command One thing that we've never really discussed in any detail is what will happen to setuptools in a post-PEP 517 world. At the moment, I'm sort of working on the assumption that we'll simply have to retain the existing setuptools support code until everyone's stopped using setuptools. But the other option is that someone writes a backend that wraps setuptools. Such a backend would rely on the sdist/egg_info/bdist_wheel commands from setuptools and expose PEP 517 interfaces based on them. We could code pip to assume that backend if there's no pyproject.toml, much like with the recent change to pip that assumes setuptools is a build dependency unless there's a pyproject.toml that explicitly states otherwise. But I don't know if anyone is thinking of doing this. Paul From donald at stufft.io Sat Jun 10 13:40:05 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 Jun 2017 13:40:05 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> Message-ID: <2B319990-70D6-45DF-AE93-AB594DE79ABA@stufft.io> > On Jun 10, 2017, at 1:31 PM, Paul Moore wrote: > > On 10 June 2017 at 18:14, Nick Coghlan wrote: >>> Keeping it in the backend doesn?t really buy us much of anything, except >>> that a handful of backend authors don?t have to make relatively minor >>> adjustments to their code base. In a vacuum I can?t see any compelling >>> reason to have the backend do the archiving at all and the only reason I >>> think we?re talking about it is that?s just how the backends work now? but >>> again, changing that is almost certainly going to be extremely trivial to >>> do. >> >> Thanks for spelling those out - that's the intuition that prompted me >> to make the suggestion, but without working through the specifics I >> wasn't confident enough in the idea to specifically request that >> Thomas make the update. >> >> Thomas - I agree with Donald's reasoning here, so would you mind >> updating the PEP accordingly? I think it's OK if that means that the >> initial PEP 517 support in the existing backends builds the archive >> and then unpacks it: >> >> - future PEP 517 backends still won't need to include support for >> their own wheel building > > Unless they want to act as standalone tools as well as backends - much > like flit does. I don't personally have a feel for how other backends > will look, will developers want to expose their own UI or will they be > happy to act just as a plugin to pip/twine/etc? If they want to act as a standalone tool, they?re free to do that as well. This doesn?t require them to only support unpacked creation. They have a few options here, either they could implement their standalone path as simply consuming the same directory that twine would and just do a shutil.make_archive(?) on that directory OR they could just support a flag to switch their internal logic from copying to a target directory to building an in memory wheel. The file signatures are close enough that it should be pretty easy to do with functools.partial. > > >> - for setuptools, `bdist_wheel` is actually provided by Daniel Holth's >> wheel project, and that could potentially gain a >> `bdist_wheel_unpacked` command > > One thing that we've never really discussed in any detail is what will > happen to setuptools in a post-PEP 517 world. At the moment, I'm sort > of working on the assumption that we'll simply have to retain the > existing setuptools support code until everyone's stopped using > setuptools. But the other option is that someone writes a backend that > wraps setuptools. Such a backend would rely on the > sdist/egg_info/bdist_wheel commands from setuptools and expose PEP 517 > interfaces based on them. We could code pip to assume that backend if > there's no pyproject.toml, much like with the recent change to pip > that assumes setuptools is a build dependency unless there's a > pyproject.toml that explicitly states otherwise. But I don't know if > anyone is thinking of doing this. > I think the best long thing here is to roll bdist_wheel into setuptools (it?s kind of silly we haven?t done this already since we?re standardizing on it now) and then have setuptools implement the PEP 517 interface. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Mon Jun 12 16:01:21 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 12 Jun 2017 21:01:21 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> Message-ID: <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote: > Thomas - I agree with Donald's reasoning here, so would you mind > updating the PEP accordingly? I've done so here: https://github.com/python/peps/pull/290 There are still a couple of questions on which I wasn't quite sure what the consensus is: - Do we want to rename the build_wheel hook now that it makes an unpacked wheel, e.g. export_wheel_contents to match export_sdist_contents? - I have assumed that the wheel hook puts its contents in the directory it's passed, rather than creating a subfolder. This is in keeping with the structure of wheels, which do not have a single top-level directory (unlike sdists), but it wouldn't fit with a future hypothetical extension to build multiple wheels at once; we would need a separate hook for that. From donald at stufft.io Mon Jun 12 16:14:56 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 16:14:56 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> Message-ID: <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> > On Jun 12, 2017, at 4:01 PM, Thomas Kluyver wrote: > > On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote: >> Thomas - I agree with Donald's reasoning here, so would you mind >> updating the PEP accordingly? > > I've done so here: > https://github.com/python/peps/pull/290 > > There are still a couple of questions on which I wasn't quite sure what > the consensus is: > > - Do we want to rename the build_wheel hook now that it makes an > unpacked wheel, e.g. export_wheel_contents to match > export_sdist_contents? I?m neutral on this, this is just a total bike shed I think so I?m happy to go with whatever you prefer. > - I have assumed that the wheel hook puts its contents in the > directory it's passed, rather than creating a subfolder. This is in > keeping with the structure of wheels, which do not have a single > top-level directory (unlike sdists), but it wouldn't fit with a future > hypothetical extension to build multiple wheels at once; we would need a > separate hook for that. I don?t think having a separate hook is a bad thing here since we don?t really know specifically what that would look like. However I also don?t think doing something like what we?ve done with prepare_wheel_metadata is out of the question either? One thing I notice is that prepare_wheel_metadata still doesn?t provide a way for the backend to communicate to the frontend what .dist-info folder it should be looking for but it?s currently possible for (mistakeningly or not) to end up with one or more .dist-info files in that directory, so you can?t just glob looking for any dist-info. Perhaps the answer for both of these hooks is to just put the contents into the passed in directory (so remove the {name}-{version}.dist-info directory from prepare_wheel_metadata, and leave the build_wheel/export_wheel_contents, just putting things in the root of the directory and only build this API to handle a single wheel at a time. If/when we add support for multiple wheels at a time, we can then add a new hook to handle that which we can make sure actually supports everything we need at that point, rather than trying to guess what that might look like today? ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 12 16:41:23 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 16:41:23 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> Message-ID: > On Jun 12, 2017, at 4:36 PM, Daniel Holth wrote: > > It's certainly easier to build a zipfile correctly than to build a directory tree. Might even be faster if your filesystem is slow. Surely if there are multiple *.dist-info it is an error? > Sure, but it?s an error that we currently run into and it tends to occur for the people who are least able to do anything about it. Why not design the interface to make the error impossible? ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Jun 12 16:42:11 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Jun 2017 20:42:11 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> Message-ID: Once I thought it might be useful to define 'wheel internal manifest' (whim) format which would just be a list or mapping that looks like [('category', source path or filelike, destination path), ... ], since that is basically what wheel is doing, putting paths in categories. Then you get the data model without worrying about the file format. On Mon, Jun 12, 2017 at 4:36 PM Daniel Holth wrote: > It's certainly easier to build a zipfile correctly than to build a > directory tree. Might even be faster if your filesystem is slow. Surely if > there are multiple *.dist-info it is an error? > > On Mon, Jun 12, 2017 at 4:15 PM Donald Stufft wrote: > >> On Jun 12, 2017, at 4:01 PM, Thomas Kluyver wrote: >> >> On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote: >> >> Thomas - I agree with Donald's reasoning here, so would you mind >> updating the PEP accordingly? >> >> >> I've done so here: >> https://github.com/python/peps/pull/290 >> >> There are still a couple of questions on which I wasn't quite sure what >> the consensus is: >> >> - Do we want to rename the build_wheel hook now that it makes an >> unpacked wheel, e.g. export_wheel_contents to match >> export_sdist_contents? >> >> >> >> I?m neutral on this, this is just a total bike shed I think so I?m happy >> to go with whatever you prefer. >> >> >> - I have assumed that the wheel hook puts its contents in the >> directory it's passed, rather than creating a subfolder. This is in >> keeping with the structure of wheels, which do not have a single >> top-level directory (unlike sdists), but it wouldn't fit with a future >> hypothetical extension to build multiple wheels at once; we would need a >> separate hook for that. >> >> >> I don?t think having a separate hook is a bad thing here since we don?t >> really know specifically what that would look like. However I also don?t >> think doing something like what we?ve done with prepare_wheel_metadata is >> out of the question either? >> >> One thing I notice is that prepare_wheel_metadata still doesn?t provide a >> way for the backend to communicate to the frontend what .dist-info folder >> it should be looking for but it?s currently possible for (mistakeningly or >> not) to end up with one or more .dist-info files in that directory, so you >> can?t just glob looking for any dist-info. >> >> Perhaps the answer for both of these hooks is to just put the contents >> into the passed in directory (so remove the {name}-{version}.dist-info >> directory from prepare_wheel_metadata, and leave the >> build_wheel/export_wheel_contents, just putting things in the root of the >> directory and only build this API to handle a single wheel at a time. >> If/when we add support for multiple wheels at a time, we can then add a new >> hook to handle that which we can make sure actually supports everything we >> need at that point, rather than trying to guess what that might look like >> today? >> >> >> ? >> >> Donald Stufft >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Jun 12 16:36:00 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Jun 2017 20:36:00 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> Message-ID: It's certainly easier to build a zipfile correctly than to build a directory tree. Might even be faster if your filesystem is slow. Surely if there are multiple *.dist-info it is an error? On Mon, Jun 12, 2017 at 4:15 PM Donald Stufft wrote: > On Jun 12, 2017, at 4:01 PM, Thomas Kluyver wrote: > > On Sat, Jun 10, 2017, at 06:14 PM, Nick Coghlan wrote: > > Thomas - I agree with Donald's reasoning here, so would you mind > updating the PEP accordingly? > > > I've done so here: > https://github.com/python/peps/pull/290 > > There are still a couple of questions on which I wasn't quite sure what > the consensus is: > > - Do we want to rename the build_wheel hook now that it makes an > unpacked wheel, e.g. export_wheel_contents to match > export_sdist_contents? > > > > I?m neutral on this, this is just a total bike shed I think so I?m happy > to go with whatever you prefer. > > > - I have assumed that the wheel hook puts its contents in the > directory it's passed, rather than creating a subfolder. This is in > keeping with the structure of wheels, which do not have a single > top-level directory (unlike sdists), but it wouldn't fit with a future > hypothetical extension to build multiple wheels at once; we would need a > separate hook for that. > > > I don?t think having a separate hook is a bad thing here since we don?t > really know specifically what that would look like. However I also don?t > think doing something like what we?ve done with prepare_wheel_metadata is > out of the question either? > > One thing I notice is that prepare_wheel_metadata still doesn?t provide a > way for the backend to communicate to the frontend what .dist-info folder > it should be looking for but it?s currently possible for (mistakeningly or > not) to end up with one or more .dist-info files in that directory, so you > can?t just glob looking for any dist-info. > > Perhaps the answer for both of these hooks is to just put the contents > into the passed in directory (so remove the {name}-{version}.dist-info > directory from prepare_wheel_metadata, and leave the > build_wheel/export_wheel_contents, just putting things in the root of the > directory and only build this API to handle a single wheel at a time. > If/when we add support for multiple wheels at a time, we can then add a new > hook to handle that which we can make sure actually supports everything we > need at that point, rather than trying to guess what that might look like > today? > > > ? > > Donald Stufft > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Jun 12 16:45:08 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Jun 2017 20:45:08 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> Message-ID: On Mon, Jun 12, 2017 at 4:41 PM Donald Stufft wrote: > > On Jun 12, 2017, at 4:36 PM, Daniel Holth wrote: > > It's certainly easier to build a zipfile correctly than to build a > directory tree. Might even be faster if your filesystem is slow. Surely if > there are multiple *.dist-info it is an error? > > > > Sure, but it?s an error that we currently run into and it tends to occur > for the people who are least able to do anything about it. Why not design > the interface to make the error impossible? > Live dangerously. I think all my wheel generators except bdist_wheel build the zipfile directly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Mon Jun 12 16:57:23 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 12 Jun 2017 21:57:23 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> Message-ID: <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: > I think all my wheel generators except bdist_wheel build the zipfile > directly. There is a certain appeal to using the zipped .whl file as the canonical format for all tools that produce or consume wheels, rather than defining a closely related but distinct 'unpacked wheel' format. A directory and a zip file do not have 100% identical features (filename encodings may differ, entries in a zip file are ordered, there may be metadata in one format that's not present in the other, and so on). I think we're also making this change in the assumption that frontends will be few and backends numerous, so it makes sense to shift more work into frontends. That may not necessarily be true - I could imagine more frontends emerging while people standardise on just a few backends. Jupyter's frontend/kernel separation was initially designed in the belief that it would support one kernel and many frontends, but we've ended up getting a lot of kernels with just a handful of popular backends. I don't feel strongly about this - I can build a wheel and then unzip it again if that's what the spec says. But given the choice, I'd specify it in terms of a zipped .whl file rather than a directory. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Jun 12 17:05:24 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Jun 2017 21:05:24 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: Yes, and I worry that certain front ends will generate the zipfile incorrectly. Better to do it in the back end. On Mon, Jun 12, 2017 at 4:57 PM Thomas Kluyver wrote: > On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: > > I think all my wheel generators except bdist_wheel build the zipfile > directly. > > > There is a certain appeal to using the zipped .whl file as the canonical > format for all tools that produce or consume wheels, rather than defining a > closely related but distinct 'unpacked wheel' format. A directory and a zip > file do not have 100% identical features (filename encodings may differ, > entries in a zip file are ordered, there may be metadata in one format > that's not present in the other, and so on). > > I think we're also making this change in the assumption that frontends > will be few and backends numerous, so it makes sense to shift more work > into frontends. That may not necessarily be true - I could imagine more > frontends emerging while people standardise on just a few backends. > Jupyter's frontend/kernel separation was initially designed in the belief > that it would support one kernel and many frontends, but we've ended up > getting a lot of kernels with just a handful of popular backends. > > I don't feel strongly about this - I can build a wheel and then unzip it > again if that's what the spec says. But given the choice, I'd specify it in > terms of a zipped .whl file rather than a directory. > > Thomas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 12 17:09:40 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 17:09:40 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: <8ABB808B-9A58-4783-9256-C6ED2237F309@stufft.io> > On Jun 12, 2017, at 5:05 PM, Daniel Holth wrote: > > Yes, and I worry that certain front ends will generate the zipfile incorrectly. Better to do it in the back end. It?s pretty hard to screw up zipping up a directory. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Mon Jun 12 17:21:13 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 12 Jun 2017 22:21:13 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <8ABB808B-9A58-4783-9256-C6ED2237F309@stufft.io> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <8ABB808B-9A58-4783-9256-C6ED2237F309@stufft.io> Message-ID: <1497302473.2035658.1007153176.03C1BEEC@webmail.messagingengine.com> On Mon, Jun 12, 2017, at 10:09 PM, Donald Stufft wrote: > It?s pretty hard to screw up zipping up a directory. If you want reproducible builds, it's very easy to screw up, and your response doesn't inspire confidence that frontends will do it carefully. But I see flit mainly as something you use directly to build and publish wheels, and these hooks of secondary interest for e.g. installing from a VCS URL. So I'll keep my zip-archive-building code in flit, and let frontend tools duplicate the part of that functionality they care about. It's also, of course, hard to screw up unzipping to a directory. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 12 17:23:37 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 17:23:37 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497302473.2035658.1007153176.03C1BEEC@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <8ABB808B-9A58-4783-9256-C6ED2237F309@stufft.io> <1497302473.2035658.1007153176.03C1BEEC@webmail.messagingengine.com> Message-ID: > On Jun 12, 2017, at 5:21 PM, Thomas Kluyver wrote: > > On Mon, Jun 12, 2017, at 10:09 PM, Donald Stufft wrote: >> It?s pretty hard to screw up zipping up a directory. > > If you want reproducible builds, it's very easy to screw up, and your response doesn't inspire confidence that frontends will do it carefully. But I see flit mainly as something you use directly to build and publish wheels, and these hooks of secondary interest for e.g. installing from a VCS URL. So I'll keep my zip-archive-building code in flit, and let frontend tools duplicate the part of that functionality they care about. > > It's also, of course, hard to screw up unzipping to a directory. ;-) > > Sure! I?m not advocating for unpacked wheels because I think either one is harder or easier to screw up, but rather because it allows the front ends more flexibility in how they use the wheels and allows us to avoid doing work, making the process involved faster for everyone. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Mon Jun 12 18:10:41 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Mon, 12 Jun 2017 23:10:41 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <8ABB808B-9A58-4783-9256-C6ED2237F309@stufft.io> <1497302473.2035658.1007153176.03C1BEEC@webmail.messagingengine.com> Message-ID: <1497305441.2869508.1007196960.545512FB@webmail.messagingengine.com> On Mon, Jun 12, 2017, at 10:23 PM, Donald Stufft wrote: > because it allows the front ends more flexibility in how they use > the wheels I don't get this? Why is it more flexible? > and allows us to avoid doing work, making the process involved faster > for everyone. This is true so long as backends build a directory and then skip zipping it up. If backends are building a zip file and then unpacking it (which, from what Daniel and I have described, may be common), then for tasks which want a zip file, you're now unpacking and repacking it. So it hinges both on what backends do and on what tasks are common for frontends using this interface. You might assume that the most common task will be installation, which uses the files unpacked. But most installs will use a pre-built wheel, so it's not obvious to me that the typical use of the build interface will be to install a package. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 12 18:30:43 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 18:30:43 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497305441.2869508.1007196960.545512FB@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <8ABB808B-9A58-4783-9256-C6ED2237F309@stufft.io> <1497302473.2035658.1007153176.03C1BEEC@webmail.messagingengine.com> <1497305441.2869508.1007196960.545512FB@webmail.messagingengine.com> Message-ID: <9DE9F4A8-9A1E-46B0-B283-D3607E1BD7E8@stufft.io> > On Jun 12, 2017, at 6:10 PM, Thomas Kluyver wrote: > > On Mon, Jun 12, 2017, at 10:23 PM, Donald Stufft wrote: >> because it allows the front ends more flexibility in how they use the wheels > > I don't get this? Why is it more flexible? I went into some detail here: https://mail.python.org/pipermail/distutils-sig/2017-June/030684.html > >> and allows us to avoid doing work, making the process involved faster for everyone. > > This is true so long as backends build a directory and then skip zipping it up. If backends are building a zip file and then unpacking it (which, from what Daniel and I have described, may be common), then for tasks which want a zip file, you're now unpacking and repacking it. We can?t force backends (or frontend) to implement things in the most performant way, we can only provide the tools to make it possible for them to do so. They could just as easily choose to add a time.sleep(5) just for kicks too. However, even in that case where the author of a backend tool chooses to implement it less efficiently than they possibly could, that?s fine because in one of the cases we?re in no worse off a condition than we would be if the backend was doing the zipping and in two conditions we are worse off, but they?re also conditions where the work is done once and then shared many times, so the amortized cost is relatively low anyways. > > So it hinges both on what backends do and on what tasks are common for frontends using this interface. You might assume that the most common task will be installation, which uses the files unpacked. But most installs will use a pre-built wheel, so it's not obvious to me that the typical use of the build interface will be to install a package. > No, I assume that for 2/3 of the cases *today* it basically does not matter if the front end or the backend does the ziping, the wiping is going to happen either way. For 1 of those 2/3 cases there is a possibility of changing that in the future to optimize a common path more (via the wheel cache) where moving the zipping into the front end shaved off time. For 1/3 of those cases today it will cut off processing time. So to sum up: Today: 1/3 of the cases will be faster, 2/3 of the cases it will make no difference. Possible Future: 2/3 of the cases will be faster, 1/3 of the cases it will make no difference. There is basically no benefit to having the backend do it, there is no case where it is faster to do so unless you assume a modern CPU with an ancient and slow hard drive to the point that copying a file is significantly slower than compressing it. Just for kicks I went and did a quick test on pip, when generating a wheel, ~50% of the time taken running ``python setup.py bdist_wheel`` on my computer is taken up by compressing the file and another ~25% of that time is taken up by decompressing the file again. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jun 12 18:34:22 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Jun 2017 23:34:22 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On 12 June 2017 at 21:57, Thomas Kluyver wrote: > There is a certain appeal to using the zipped .whl file as the canonical > format for all tools that produce or consume wheels, rather than defining a > closely related but distinct 'unpacked wheel' format. A directory and a zip > file do not have 100% identical features (filename encodings may differ, > entries in a zip file are ordered, there may be metadata in one format > that's not present in the other, and so on). This is a reasonable point. As I understand it, zipfiles are guaranteed to support the full Unicode range for filenames, via UTF-8. But it's not impossible for filesystems to only support a limited subset (for example, I believe the encoding used for FAT32 filesystems is not clearly defined, but is probably some 8-bit codepage, and Unix systems rely on whatever encoding the user has specified via the locale settings). I can't offhand imagine a practical situation where the filesystem encoding of the build system would cause wheel generation to fail, but would work for the rest of the build chain - but on the other hand, this whole question is pretty borderline in any case, as far as I can tell, so I'm somewhat inclined to go for using the format that *doesn't* have potential encoding issues built in (I'm pretty sick of dealing with encoding issues with pip...) But honestly, I think we're at the point where someone just needs to make a decision - there's very little compelling evidence either way. Paul From njs at pobox.com Mon Jun 12 18:36:23 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 15:36:23 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On Mon, Jun 12, 2017 at 1:57 PM, Thomas Kluyver wrote: > On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: > > I think all my wheel generators except bdist_wheel build the zipfile > directly. > > > There is a certain appeal to using the zipped .whl file as the canonical > format for all tools that produce or consume wheels, rather than defining a > closely related but distinct 'unpacked wheel' format. A directory and a zip > file do not have 100% identical features (filename encodings may differ, > entries in a zip file are ordered, there may be metadata in one format > that's not present in the other, and so on). I find the reproducible builds argument to be a pretty compelling argument for generating wheels directly. (It also applies to sdists.) Another point is that tools that you might have in your build pipeline -- like auditwheel -- currently use wheel files as their interchange format, so you might end up having to zip, run auditwheel, unzip for pip, and the pip zips again to cache the wheel... The whole conversation feels a bit like we're falling into the developer trap of "oo there's a thing that might be optimizable therefore we MUST optimize it" without any real assessment of the benefits (I'm as guilty as anyone!). It's not even clear to me that copying a tree twice *is* faster than packing and then unpacking a wheel in general ? if your tree consists of lots of small files and you're IO-bound, then the wheel version might well be faster. (E.g. on an underprovisioned virtual server, especially if using spinning media - while of course we're all benchmarking on laptops with fast SSD and everything in cache :-).) And in any case, I'm generally very skeptical of moving away from the well-specified wheel format that already has lots of tooling and consensus around it towards anything ad hoc, when AFAICT no-one has even identified this as an important bottleneck. -n -- Nathaniel J. Smith -- https://vorpus.org From donald at stufft.io Mon Jun 12 18:42:48 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 18:42:48 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: > On Jun 12, 2017, at 6:34 PM, Paul Moore wrote: > > On 12 June 2017 at 21:57, Thomas Kluyver wrote: >> There is a certain appeal to using the zipped .whl file as the canonical >> format for all tools that produce or consume wheels, rather than defining a >> closely related but distinct 'unpacked wheel' format. A directory and a zip >> file do not have 100% identical features (filename encodings may differ, >> entries in a zip file are ordered, there may be metadata in one format >> that's not present in the other, and so on). > > This is a reasonable point. As I understand it, zipfiles are > guaranteed to support the full Unicode range for filenames, via UTF-8. > But it's not impossible for filesystems to only support a limited > subset (for example, I believe the encoding used for FAT32 filesystems > is not clearly defined, but is probably some 8-bit codepage, and Unix > systems rely on whatever encoding the user has specified via the > locale settings). As always, it?s complicated ? https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/ ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 12 18:49:56 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 18:49:56 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: > On Jun 12, 2017, at 6:36 PM, Nathaniel Smith wrote: > > On Mon, Jun 12, 2017 at 1:57 PM, Thomas Kluyver wrote: >> On Mon, Jun 12, 2017, at 09:45 PM, Daniel Holth wrote: >> >> I think all my wheel generators except bdist_wheel build the zipfile >> directly. >> >> >> There is a certain appeal to using the zipped .whl file as the canonical >> format for all tools that produce or consume wheels, rather than defining a >> closely related but distinct 'unpacked wheel' format. A directory and a zip >> file do not have 100% identical features (filename encodings may differ, >> entries in a zip file are ordered, there may be metadata in one format >> that's not present in the other, and so on). > > I find the reproducible builds argument to be a pretty compelling > argument for generating wheels directly. (It also applies to sdists.) We?re not preventing backends from having a stand alone tool that produces reproducible wheels if they?re able/willing to do that. > Another point is that tools that you might have in your build pipeline > -- like auditwheel -- currently use wheel files as their interchange > format, so you might end up having to zip, run auditwheel, unzip for > pip, and the pip zips again to cache the wheel? How is that different from today? In the hypothetical build_wheel producing a zip file? you produce a zip file, run auditwheel which unzips it, which presumably has to zip it back up again for pip, and then pip unzips it again on every single install. If auditwheel doesn?t start to accept unzipped wheels, then nothing changes, if it does then suddenly we skip some round trips through zip/unzip and things get faster for everyone. > > The whole conversation feels a bit like we're falling into the > developer trap of "oo there's a thing that might be optimizable > therefore we MUST optimize it" without any real assessment of the > benefits (I'm as guilty as anyone!). It's not even clear to me that > copying a tree twice *is* faster than packing and then unpacking a > wheel in general ? if your tree consists of lots of small files and > you're IO-bound, then the wheel version might well be faster. (E.g. on > an underprovisioned virtual server, especially if using spinning media > - while of course we're all benchmarking on laptops with fast SSD and > everything in cache :-).) And in any case, I'm generally very > skeptical of moving away from the well-specified wheel format that > already has lots of tooling and consensus around it towards anything > ad hoc, when AFAICT no-one has even identified this as an important > bottleneck. > I?ve measured that 50%-75% of the time taken by ``python setup.py bdist_wheel`` + unzipping the resulting wheel can be eliminated for ``pip install ./pip``. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jun 12 18:55:59 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 15:55:59 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On Mon, Jun 12, 2017 at 3:34 PM, Paul Moore wrote: > But honestly, I think we're at the point where someone just needs to > make a decision - there's very little compelling evidence either way. I was the original PEP author, but Thomas has mostly taken it over at this point, so I'm not sure how much you should listen to me :-). But if you pointed at me and told me to make some decisions, then right now this is what I'd do: 1) Go back to having the wheel generation hook generate a .whl file Rationale: the optimization benefits of generating an unpacked wheel are unclear, but we know that reproducible builds are important, and filename encoding is tricky and important, and that having a common well-understood standard with tooling around it is important, and on those axes .whl unambiguously wins. And if there do later turn out to be compelling optimization benefits to generating unpacked wheels, then we can add an optional generate_unpacked_wheel hook in the future. 2) Specify that the wheel generation hook, metadata hook, and sdist hook return the name of path that they created as a unicode string. Rationale: fixes a point of ambiguity in the current spec. 3) Go back to having the sdist generation hook generate an archive file instead of an unpacked directory Rationale: Essentially the same as for point (1). The additional complication here is that we know that pip plans to use this as part of its standard build-from-unpacked-source pipeline, so if we're trying to optimize this case for developers with their fast SSD drives etc. then the compress/decompress cycle actually does matter. However... from discussion upthread it sounds like the decision was that for developers doing repeated rebuilds, plain 'pip install .' is just not the tool to use; they should be using something like develop installs (which make re-installation instantaneous), or backend-specific mechanisms (that can do incremental builds). These give an order of magnitude improvement over even the 'optimized' version of 'pip install .' where the backend generates an unpacked tree, and "special cases aren't special enough to break the rules". 4) Specify that the sdist archive file must be .tar.gz Rationale: we only need one and it reduces variation, and dstufft likes .tar.gz better than .zip. 5) Drop the prepare_files hook Rationale: it's purpose is somewhat unclear at this point, and it gives up the main advantage of going through sdist (the guarantee that building from sdist and building from unpacked tree give the same results) while still having the main disadvantages (you have to copy everything and lose incremental rebuilds). -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Mon Jun 12 19:18:18 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 16:18:18 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On Mon, Jun 12, 2017 at 3:49 PM, Donald Stufft wrote: > > On Jun 12, 2017, at 6:36 PM, Nathaniel Smith wrote > > Another point is that tools that you might have in your build pipeline > -- like auditwheel -- currently use wheel files as their interchange > format, so you might end up having to zip, run auditwheel, unzip for > pip, and the pip zips again to cache the wheel? > > > How is that different from today? In the hypothetical build_wheel > producing a zip file? you produce a zip file, run auditwheel which unzips > it, which presumably has to zip it back up again for pip, and then pip > unzips it again on every single install. > > If auditwheel doesn?t start to accept unzipped wheels, then nothing > changes, if it does then suddenly we skip some round trips through > zip/unzip and things get faster for everyone. > I would strongly prefer auditwheel not have to accept unzipped wheel or generate unzipped wheels, because that just multiples the number of cases that need to be supported, and as you've pointed out many times, more potential paths = more chances for bugs. So if you have auditwheel as the last step in your pipeline, that means that at the end of the build what you have is a zipped wheel. If pip accepts zipped wheels, then we can just hand this over and pip drops it in its cache and unzips it into the final location. If pip requires unpacked wheels, then first the backend has to unzip it, and then pip has to do something with the unpacked directory (either copy it file-by-file, or possibly even zip it up again to cache it). > > > The whole conversation feels a bit like we're falling into the > developer trap of "oo there's a thing that might be optimizable > therefore we MUST optimize it" without any real assessment of the > benefits (I'm as guilty as anyone!). It's not even clear to me that > copying a tree twice *is* faster than packing and then unpacking a > wheel in general ? if your tree consists of lots of small files and > you're IO-bound, then the wheel version might well be faster. (E.g. on > an underprovisioned virtual server, especially if using spinning media > - while of course we're all benchmarking on laptops with fast SSD and > everything in cache :-).) And in any case, I'm generally very > skeptical of moving away from the well-specified wheel format that > already has lots of tooling and consensus around it towards anything > ad hoc, when AFAICT no-one has even identified this as an important > bottleneck. > > > I?ve measured that 50%-75% of the time taken by ``python setup.py > bdist_wheel`` + unzipping the resulting wheel can be eliminated for ``pip > install ./pip``. > Sure, but no-one noticed or cared about this until we started talking about unpacked wheels for other reasons, and then we went hunting for benchmarks to justify the idea :-). And even so your benchmark is a bit cherry-picked -- that %age will go down if you include the 'setup.py sdist' / 'setup.py unpacked_sdist' step that you want 'pip install ./pip' to do, and even more so if you test on some system with a less robust IO layer than your fancy developer laptop. (I heard a rumor recently that the reason Travis-CI's MacOS builds are so terribly behind all the time is that their hosting provider has plenty of CPU but their SAN is at its absolute limit in terms of IOPS, so they can't add any more capacity. Packed wheels are much friendlier than unpacked ones when it comes to IOPS...) -n -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jun 12 19:41:56 2017 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Jun 2017 19:41:56 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: <82221A5A-31E7-46A6-9C32-4725AF07D987@stufft.io> > On Jun 12, 2017, at 7:18 PM, Nathaniel Smith wrote: > > On Mon, Jun 12, 2017 at 3:49 PM, Donald Stufft > wrote: > >> On Jun 12, 2017, at 6:36 PM, Nathaniel Smith > wrote >> Another point is that tools that you might have in your build pipeline >> -- like auditwheel -- currently use wheel files as their interchange >> format, so you might end up having to zip, run auditwheel, unzip for >> pip, and the pip zips again to cache the wheel? > > How is that different from today? In the hypothetical build_wheel producing a zip file? you produce a zip file, run auditwheel which unzips it, which presumably has to zip it back up again for pip, and then pip unzips it again on every single install. > > If auditwheel doesn?t start to accept unzipped wheels, then nothing changes, if it does then suddenly we skip some round trips through zip/unzip and things get faster for everyone. > > I would strongly prefer auditwheel not have to accept unzipped wheel or generate unzipped wheels, because that just multiples the number of cases that need to be supported, and as you've pointed out many times, more potential paths = more chances for bugs. So if you have auditwheel as the last step in your pipeline, that means that at the end of the build what you have is a zipped wheel. If pip accepts zipped wheels, then we can just hand this over and pip drops it in its cache and unzips it into the final location. If pip requires unpacked wheels, then first the backend has to unzip it, and then pip has to do something with the unpacked directory (either copy it file-by-file, or possibly even zip it up again to cache it). Unless audit wheel is calling this backend directly, or is trying to implement this API to be called by pip, then it never has to do that. This isn?t really meant to be an end user exposed UX, this is strictly for two tools to talk to each other. Thus auditwheel is free to continue to work as it does today and it can completely ignore this spec by just continuing to expect someone to invoke a command that builds a wheel first. > > > >> >> The whole conversation feels a bit like we're falling into the >> developer trap of "oo there's a thing that might be optimizable >> therefore we MUST optimize it" without any real assessment of the >> benefits (I'm as guilty as anyone!). It's not even clear to me that >> copying a tree twice *is* faster than packing and then unpacking a >> wheel in general ? if your tree consists of lots of small files and >> you're IO-bound, then the wheel version might well be faster. (E.g. on >> an underprovisioned virtual server, especially if using spinning media >> - while of course we're all benchmarking on laptops with fast SSD and >> everything in cache :-).) And in any case, I'm generally very >> skeptical of moving away from the well-specified wheel format that >> already has lots of tooling and consensus around it towards anything >> ad hoc, when AFAICT no-one has even identified this as an important >> bottleneck. >> > > I?ve measured that 50%-75% of the time taken by ``python setup.py bdist_wheel`` + unzipping the resulting wheel can be eliminated for ``pip install ./pip``. > > Sure, but no-one noticed or cared about this until we started talking about unpacked wheels for other reasons, and then we went hunting for benchmarks to justify the idea :-). And even so your benchmark is a bit cherry-picked -- that %age will go down if you include the 'setup.py sdist' / 'setup.py unpacked_sdist' step that you want 'pip install ./pip' to do, and even more so if you test on some system with a less robust IO layer than your fancy developer laptop. Well generating an unpacked sdist rather than a packed sdist saves roughly 50% of the time there too. I wouldn?t specifically say nobody cared, but rather the nature of things meant nobody was in a position to do anything about it until now. It?s not like any of the tooling provided a way to do it natively, so it wasn?t worth the cost of monkey patching. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jun 12 20:15:00 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Jun 2017 17:15:00 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <82221A5A-31E7-46A6-9C32-4725AF07D987@stufft.io> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <82221A5A-31E7-46A6-9C32-4725AF07D987@stufft.io> Message-ID: On Mon, Jun 12, 2017 at 4:41 PM, Donald Stufft wrote: > > On Jun 12, 2017, at 7:18 PM, Nathaniel Smith wrote: > > On Mon, Jun 12, 2017 at 3:49 PM, Donald Stufft wrote: > >> >> On Jun 12, 2017, at 6:36 PM, Nathaniel Smith wrote >> >> Another point is that tools that you might have in your build pipeline >> -- like auditwheel -- currently use wheel files as their interchange >> format, so you might end up having to zip, run auditwheel, unzip for >> pip, and the pip zips again to cache the wheel? >> >> >> How is that different from today? In the hypothetical build_wheel >> producing a zip file? you produce a zip file, run auditwheel which unzips >> it, which presumably has to zip it back up again for pip, and then pip >> unzips it again on every single install. >> >> If auditwheel doesn?t start to accept unzipped wheels, then nothing >> changes, if it does then suddenly we skip some round trips through >> zip/unzip and things get faster for everyone. >> > > I would strongly prefer auditwheel not have to accept unzipped wheel or > generate unzipped wheels, because that just multiples the number of cases > that need to be supported, and as you've pointed out many times, more > potential paths = more chances for bugs. So if you have auditwheel as the > last step in your pipeline, that means that at the end of the build what > you have is a zipped wheel. If pip accepts zipped wheels, then we can just > hand this over and pip drops it in its cache and unzips it into the final > location. If pip requires unpacked wheels, then first the backend has to > unzip it, and then pip has to do something with the unpacked directory > (either copy it file-by-file, or possibly even zip it up again to cache it). > > > Unless audit wheel is calling this backend directly, or is trying to > implement this API to be called by pip, then it never has to do that. This > isn?t really meant to be an end user exposed UX, this is strictly for two > tools to talk to each other. Thus auditwheel is free to continue to work as > it does today and it can completely ignore this spec by just continuing to > expect someone to invoke a command that builds a wheel first. > Yeah, I'm talking about the currently-hypothetical situation where the build backend wants to call auditwheel as part of its build. Auditwheel's current design as a secondary tool you run after the "real" build is expedient, but it would be nice if someday build systems could generate working wheels directly... -n -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jun 12 21:27:34 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Jun 2017 11:27:34 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On 13 June 2017 at 08:55, Nathaniel Smith wrote: > On Mon, Jun 12, 2017 at 3:34 PM, Paul Moore wrote: >> But honestly, I think we're at the point where someone just needs to >> make a decision - there's very little compelling evidence either way. > > I was the original PEP author, but Thomas has mostly taken it over at > this point, so I'm not sure how much you should listen to me :-). But > if you pointed at me and told me to make some decisions, then right > now this is what I'd do: > > 1) Go back to having the wheel generation hook generate a .whl file > Rationale: the optimization benefits of generating an unpacked wheel > are unclear, but we know that reproducible builds are important, and > filename encoding is tricky and important, and that having a common > well-understood standard with tooling around it is important, and on > those axes .whl unambiguously wins. And if there do later turn out to > be compelling optimization benefits to generating unpacked wheels, > then we can add an optional generate_unpacked_wheel hook in the > future. Despite being the one to originally propose standardisation on passing directory paths around, I'm starting to lean back towards this approach. My rationale for this doesn't really have a lot to do with topics we've discussed so far, and instead asks the question: what would work best for an installation frontend that wanted to keep the actual build tools off the system doing the installation, while still allowing for transparent "from sdist" installations? And that's where cross-platform archive formats really shine: since they're incredibly convenient for network transfers, they make fewer implicit assumptions about *where* the work is being done. Whereas if we rely on directories in the baseline interface specification, we're making a lot of assumptions about how all future frontends are going to work. It should also be relatively straightforward to add cross-platform validators to frontends that check that archives are well-formed, but it's harder to write such validators that work cross-platform on arbitrary directories. > 2) Specify that the wheel generation hook, metadata hook, and sdist > hook return the name of path that they created as a unicode string. > Rationale: fixes a point of ambiguity in the current spec. And still leaves the door open to supporting multiple wheels in the future by returning a list instead of string. > 3) Go back to having the sdist generation hook generate an archive > file instead of an unpacked directory > Rationale: Essentially the same as for point (1). The additional > complication here is that we know that pip plans to use this as part > of its standard build-from-unpacked-source pipeline, so if we're > trying to optimize this case for developers with their fast SSD drives > etc. then the compress/decompress cycle actually does matter. I think you also raise a good point in bringing up the "First make it work, then make it right, then make it fast" priority order. We're currently in the "make it right" phase, and I think you're correct that that's much easier to achieve cross platform with archive based data exchange - otherwise we run the risk of running into surprises with NTFS, HDFS+, etc (let alone folks running builds over NFS or CIFS). So the core "make it right" API would be "build_sdist" and "build_wheel" as mandatory backend hooks, with "prepare_wheel_metadata" and "prepare_build_files" as optional "make it faster" helpers. We'd then reserve "prepare_sdist_content" and "prepare_wheel_content" as possible future "make it faster" APIs, rather than adding them now. > 4) Specify that the sdist archive file must be .tar.gz > Rationale: we only need one and it reduces variation, and dstufft > likes .tar.gz better than .zip. +1 > 5) Drop the prepare_files hook > Rationale: it's purpose is somewhat unclear at this point, and it > gives up the main advantage of going through sdist (the guarantee that > building from sdist and building from unpacked tree give the same > results) while still having the main disadvantages (you have to copy > everything and lose incremental rebuilds). The main reason this made it into the PEP is that build_sdist may have additional dependencies that prepare_build_files doesn't (e.g. Thomas has indicated that flit needs to call out to VCS tools to build a full sdist, but can export its own input files directly). So while it's technically a "make it faster" hook rather than an essential "make it right" hook, I think just smoothing out the interaction between pip & flit specifically provides sufficient value to make it worth including in the initial version of the specification. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Jun 13 03:59:40 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Jun 2017 08:59:40 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On 12 June 2017 at 23:42, Donald Stufft wrote: > > As always, it?s complicated ? > https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/ Lol. But I still consider falling over on normalisation rules an improvement over falling over on unrepresentable characters ;-) And I assume that the stdlib zipfile module at least does the right thing with Unicode filenames (UTF-8 encoding, recorded properly in the zip). Paul From njs at pobox.com Tue Jun 13 04:22:26 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Jun 2017 01:22:26 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On Tue, Jun 13, 2017 at 12:59 AM, Paul Moore wrote: > > On 12 June 2017 at 23:42, Donald Stufft wrote: > > > > As always, it?s complicated ? > > https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/ > > Lol. But I still consider falling over on normalisation rules an > improvement over falling over on unrepresentable characters ;-) And I > assume that the stdlib zipfile module at least does the right thing > with Unicode filenames (UTF-8 encoding, recorded properly in the zip). Yeah, it looks like the stdlib (at least in master) does like: - if the filename is valid ascii, then flag it as being in cp437 - otherwise, use utf-8 and flag it as such I guess the idea is that this way, the resulting zip files can be read correctly by correct readers that pay attention to the flags, and by readers that ignore the flags and assume everything is utf-8, and (if the filenames happen to fit in ascii) by ancient readers that only know cp437 and blow up when they see the utf-8 flag. Though - we should probably mandate sometime/somewhere that the filenames in a wheel are always in UTF-8 and also that they follow some particular normalization (NFC?). And likewise for sdists, though actually this is a place where zip is technically superior to tar: AFAIK tar has no way to indicate the filename encoding at all. I guess this could be an argument for making zip the standard sdist format as well, but I really don't care either way so I'm not going to fuss about it. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Tue Jun 13 04:45:23 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Jun 2017 01:45:23 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On Tue, Jun 13, 2017 at 1:22 AM, Nathaniel Smith wrote: > Though - we should probably mandate sometime/somewhere that the > filenames in a wheel are always in UTF-8 and also that they follow > some particular normalization (NFC?). It looks like PEP 3131 says that Python identifiers get normalized to NFKC while parsing, so using NFKC for wheel filenames (which often include package/module names) probably makes sense as well. -n -- Nathaniel J. Smith -- https://vorpus.org From thomas at kluyver.me.uk Tue Jun 13 05:44:58 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 13 Jun 2017 10:44:58 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: > Despite being the one to originally propose standardisation on passing > directory paths around, I'm starting to lean back towards this > approach. > > My rationale for this doesn't really have a lot to do with topics > we've discussed so far, and instead asks the question: what would work > best for an installation frontend that wanted to keep the actual build > tools off the system doing the installation, while still allowing for > transparent "from sdist" installations? I think that we're discovering a variety of reasons why an unpacked distribution may not be clearly preferable to an archive. In the absence of a clear benefit, I think it's advantageous to say that the archive is the canonical interchange format which different tools produce and consume. This is more compelling for wheels than for sdists, since the wheel format is more precisely specified, but it seems more internally consistent to say that they both build archives. I've updated the PR to specify zip archives for build_wheel and .tar.gz archives for build_sdist. >> 2) Specify that the wheel generation hook, metadata hook, and sdist >> hook return the name of path that they created as a unicode string. >> Rationale: fixes a point of ambiguity in the current spec. > > And still leaves the door open to supporting multiple wheels in the > future by returning a list instead of string. This makes sense to me, and I've added that to the PR. I have currently specified that they should return only the basename of the file/directory they create, not the full path. I don't think there's any particular reason to prefer one or the other, but it needs to be specified. Does anyone think there's a concrete reason they should be full paths? Finally, I noticed while updating the PEP that there's a section showing how simple a PEP 517 backend can be, with a tiny working example. Following our discussions, that example is incomplete (it's missing at least the sdist hook). Do we want to: 1. Make the example complete again, which would make it much less simple. 2. Remove the whole section about how easy it is to implement a backend (it's not a very convincing example anyway, because it requires the user to manually do most of the work in preparing a wheel). (3. Try to actually make it simple to implement a backend ;-) Thomas From dholth at gmail.com Tue Jun 13 14:36:49 2017 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Jun 2017 18:36:49 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: Took a peek at the warehouse source code to try to understand its sdist requirements, it appears to be less strict than I thought. Probably missed something but it looks like warehouse currently requires a .zip sdist to contain PKG-INFO one level deep but doesn't care about the contents of .tar.gz files. pip of course still expects to be able to execute setup.py egg_info but it doesn't look like it cares about PKG-INFO or anything else. On Tue, Jun 13, 2017 at 5:45 AM Thomas Kluyver wrote: > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: > > Despite being the one to originally propose standardisation on passing > > directory paths around, I'm starting to lean back towards this > > approach. > > > > My rationale for this doesn't really have a lot to do with topics > > we've discussed so far, and instead asks the question: what would work > > best for an installation frontend that wanted to keep the actual build > > tools off the system doing the installation, while still allowing for > > transparent "from sdist" installations? > > I think that we're discovering a variety of reasons why an unpacked > distribution may not be clearly preferable to an archive. In the absence > of a clear benefit, I think it's advantageous to say that the archive is > the canonical interchange format which different tools produce and > consume. This is more compelling for wheels than for sdists, since the > wheel format is more precisely specified, but it seems more internally > consistent to say that they both build archives. > > I've updated the PR to specify zip archives for build_wheel and .tar.gz > archives for build_sdist. > > >> 2) Specify that the wheel generation hook, metadata hook, and sdist > >> hook return the name of path that they created as a unicode string. > >> Rationale: fixes a point of ambiguity in the current spec. > > > > And still leaves the door open to supporting multiple wheels in the > > future by returning a list instead of string. > > This makes sense to me, and I've added that to the PR. > > I have currently specified that they should return only the basename of > the file/directory they create, not the full path. I don't think there's > any particular reason to prefer one or the other, but it needs to be > specified. Does anyone think there's a concrete reason they should be > full paths? > > Finally, I noticed while updating the PEP that there's a section showing > how simple a PEP 517 backend can be, with a tiny working example. > Following our discussions, that example is incomplete (it's missing at > least the sdist hook). Do we want to: > > 1. Make the example complete again, which would make it much less > simple. > 2. Remove the whole section about how easy it is to implement a backend > (it's not a very convincing example anyway, because it requires the user > to manually do most of the work in preparing a wheel). > (3. Try to actually make it simple to implement a backend ;-) > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jun 13 21:53:45 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Jun 2017 11:53:45 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: On 13 June 2017 at 19:44, Thomas Kluyver wrote: > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: >> Despite being the one to originally propose standardisation on passing >> directory paths around, I'm starting to lean back towards this >> approach. >> >> My rationale for this doesn't really have a lot to do with topics >> we've discussed so far, and instead asks the question: what would work >> best for an installation frontend that wanted to keep the actual build >> tools off the system doing the installation, while still allowing for >> transparent "from sdist" installations? > > I think that we're discovering a variety of reasons why an unpacked > distribution may not be clearly preferable to an archive. In the absence > of a clear benefit, I think it's advantageous to say that the archive is > the canonical interchange format which different tools produce and > consume. This is more compelling for wheels than for sdists, since the > wheel format is more precisely specified, but it seems more internally > consistent to say that they both build archives. Agreed. As part of an unrelated discussion [1], I also realised that PEP 517 may actually provide a way for Linux-only projects to add venv-compatible shims to their projects without having to fully decouple themselves from their distro packaging: defining backends based on projects like dirtbike and rewheel that bundle up the project's pure Python pieces into a wheel file. It would definitely be a hack, but may help with cases like RPM's Python bindings (which are currently hard to test on Python versions other than the system Python, since they can't readily be built independently of the system RPM package). [1] https://bugs.python.org/issue30628#msg295970 > I've updated the PR to specify zip archives for build_wheel and .tar.gz > archives for build_sdist. +1 I've added one suggestion, which is to explicitly require PAX_FORMAT for the sdist tarballs produced this way (that's a POSIX format standardised in 2001 and supported by both 2.7 and 3.x that specifically requires that the paths be encoded as UTF-8). While the standard library does still default to GNU_FORMAT in general, the stated rationale for doing so (it being more widely supported than PAX_FORMAT) was last updated more than 10 years ago, and I don't think it applies here. >>> 2) Specify that the wheel generation hook, metadata hook, and sdist >>> hook return the name of path that they created as a unicode string. >>> Rationale: fixes a point of ambiguity in the current spec. >> >> And still leaves the door open to supporting multiple wheels in the >> future by returning a list instead of string. > > This makes sense to me, and I've added that to the PR. > > I have currently specified that they should return only the basename of > the file/directory they create, not the full path. I don't think there's > any particular reason to prefer one or the other, but it needs to be > specified. Does anyone think there's a concrete reason they should be > full paths? +1 from me for relative paths, since the frontend is specifying the base path anyway. It's an affordance that encourages backends to use the supplied storage location, rather than returning arbitrary paths (e.g. to files in temp directories) > Finally, I noticed while updating the PEP that there's a section showing > how simple a PEP 517 backend can be, with a tiny working example. > Following our discussions, that example is incomplete (it's missing at > least the sdist hook). Do we want to: > > 1. Make the example complete again, which would make it much less > simple. > 2. Remove the whole section about how easy it is to implement a backend > (it's not a very convincing example anyway, because it requires the user > to manually do most of the work in preparing a wheel). > (3. Try to actually make it simple to implement a backend ;-) I think it would be worthwhile to make the example complete again, using the "archive everything except dot-prefixed files and directories" sdist construction strategy. Something like: def _exclude_hidden_and_special_files(archive_entry): """Tarfile filter to exclude hidden and special files from the archive""" if entry.isfile() or entry.isdir(): if not os.path.basename(archive_entry.name).startswith("."): return archive_entry return None def build_sdist(sdist_dir, config_settings): sdist_subdir = "mypackage-0.1" sdist_path = pathlib.Path(sdist_dir) / (sdist_subdir + ".tar.gz") sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT) sdist.add(os.getcwd(), arcname=sdist_subdir, filter=_exclude_hidden_and_special_files) (I haven't actually tested that code, but I believe it should be reasonably close to doing the right thing) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jun 13 21:55:52 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Jun 2017 11:55:52 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: On 14 June 2017 at 11:53, Nick Coghlan wrote: > def build_sdist(sdist_dir, config_settings): > sdist_subdir = "mypackage-0.1" > sdist_path = pathlib.Path(sdist_dir) / (sdist_subdir + ".tar.gz") > sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT) > sdist.add(os.getcwd(), arcname=sdist_subdir, > filter=_exclude_hidden_and_special_files) Slight adjustment to return the sdist name: def build_sdist(sdist_dir, config_settings): sdist_subdir = "mypackage-0.1" sdist_name = sdist_subdir + ".tar.gz" sdist_path = pathlib.Path(sdist_dir) / sdist_name sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT) sdist.add(os.getcwd(), arcname=sdist_subdir, filter=_exclude_hidden_and_special_files) return sdist_name Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Wed Jun 14 03:53:59 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Jun 2017 08:53:59 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: On 14 June 2017 at 02:53, Nick Coghlan wrote: > While the > standard library does still default to GNU_FORMAT in general, the > stated rationale for doing so (it being more widely supported than > PAX_FORMAT) was last updated more than 10 years ago, and I don't think > it applies here. Maybe add a note to the PEP explicitly calling out the fact that this isn't the default behaviour for the stdlib? Otherwise I bet people will miss this point. (Some people will probably miss it even with a note, but we improve the odds :-)) Paul From thomas at kluyver.me.uk Wed Jun 14 05:24:49 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Wed, 14 Jun 2017 10:24:49 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: <1497432289.868563.1008943152.42FACEB9@webmail.messagingengine.com> I have updated the PR as requested: 1. Specify the PAX format, and note that it is not yet the default for the tarfile module in Python. 2. Updated the self-contained backend example with Nick's suggestions. Thomas On Wed, Jun 14, 2017, at 08:53 AM, Paul Moore wrote: > On 14 June 2017 at 02:53, Nick Coghlan wrote: > > While the > > standard library does still default to GNU_FORMAT in general, the > > stated rationale for doing so (it being more widely supported than > > PAX_FORMAT) was last updated more than 10 years ago, and I don't think > > it applies here. > > Maybe add a note to the PEP explicitly calling out the fact that this > isn't the default behaviour for the stdlib? Otherwise I bet people > will miss this point. (Some people will probably miss it even with a > note, but we improve the odds :-)) > > Paul From donald at stufft.io Wed Jun 14 16:49:42 2017 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Jun 2017 16:49:42 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: > On Jun 12, 2017, at 9:27 PM, Nick Coghlan wrote: > >> >> 5) Drop the prepare_files hook >> Rationale: it's purpose is somewhat unclear at this point, and it >> gives up the main advantage of going through sdist (the guarantee that >> building from sdist and building from unpacked tree give the same >> results) while still having the main disadvantages (you have to copy >> everything and lose incremental rebuilds). > > The main reason this made it into the PEP is that build_sdist may have > additional dependencies that prepare_build_files doesn't (e.g. Thomas > has indicated that flit needs to call out to VCS tools to build a full > sdist, but can export its own input files directly). > > So while it's technically a "make it faster" hook rather than an > essential "make it right" hook, I think just smoothing out the > interaction between pip & flit specifically provides sufficient value > to make it worth including in the initial version of the > specification. Just as an anecdote, today requests pushed a brown paper bag release because they created a wheel out of a checkout of the requests repository, which happened to have some .pyc files and directories left over from git and some refactoring. This caused the library to be completely broken as released in the wheel. This is the kind of error that creating a sdist ideally hopes to eliminate (and hopefully the make it faster hook doesn?t end up invalidating that point). ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jun 14 16:58:11 2017 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Jun 2017 20:58:11 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: Sounds like an unfortunate side effect of using distutils instead of a better build system. If they were using enscons the wheel generation would be entirely driven by a manifest or a series of glob patterns, and it would not matter if there were any extra files in the build directory. As long as the .pyc file was not listed as a dependency of the final build artifact, it would not have been included in the wheel. On Wed, Jun 14, 2017 at 4:51 PM Donald Stufft wrote: > > On Jun 12, 2017, at 9:27 PM, Nick Coghlan wrote: > > > 5) Drop the prepare_files hook > Rationale: it's purpose is somewhat unclear at this point, and it > gives up the main advantage of going through sdist (the guarantee that > building from sdist and building from unpacked tree give the same > results) while still having the main disadvantages (you have to copy > everything and lose incremental rebuilds). > > > The main reason this made it into the PEP is that build_sdist may have > additional dependencies that prepare_build_files doesn't (e.g. Thomas > has indicated that flit needs to call out to VCS tools to build a full > sdist, but can export its own input files directly). > > So while it's technically a "make it faster" hook rather than an > essential "make it right" hook, I think just smoothing out the > interaction between pip & flit specifically provides sufficient value > to make it worth including in the initial version of the > specification. > > > > Just as an anecdote, today requests pushed a brown paper bag release > because they created a wheel out of a checkout of the requests repository, > which happened to have some .pyc files and directories left over from git > and some refactoring. This caused the library to be completely broken as > released in the wheel. This is the kind of error that creating a sdist > ideally hopes to eliminate (and hopefully the make it faster hook doesn?t > end up invalidating that point). > > ? > > Donald Stufft > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jun 14 22:15:55 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Jun 2017 12:15:55 +1000 Subject: [Distutils] Upcoming change to the packaging.python.org theme (and other PyPA docs) Message-ID: Hi folks, Thanks to Jon Parrott's excellent work, the CPython documentation theme is now pip-installable (directly from git), and there's a PyPA specific derivative of that at https://github.com/pypa/pypa-docs-theme (also intended to be installed directly from git rather than pyPI). The first project to be migrating will be the Python Packaging User Guide, which Jon has posted a preview instance of at http://temp.theadora.io/pypug-pydoctheme/index.html If you'd like to review the related PR, that's at https://github.com/pypa/python-packaging-user-guide/issues/305, while the original discussion leading to the theme change is at https://github.com/pypa/python-packaging-user-guide/issues/304 While the user guide is the first project to switch themes, we expect that other PyPA projects will also switch over at some point (see the issue link above for more details on that). Regards, Nick. P.S. We're aware the new theme has limitations on mobile devices, so if folks would like to address that, the best place to tackle it would be upstream in the CPython docs theme: https://github.com/python/python-docs-theme -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Thu Jun 15 09:12:50 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 15 Jun 2017 14:12:50 +0100 Subject: [Distutils] Finishing up PEP 517 Message-ID: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> Yes, it's PEP 517 again! Here's the current text: https://www.python.org/dev/peps/pep-0517/ We currently say: > All other hooks [except get_build_requires] are executed in an environment which contains both the bootstrap requirements specified in the pyproject.toml hook and those specified by the get_build_requires hook. It's not clear to me whether this should be required for the build_sdist and prepare_build_files hooks, nor whether there are any adverse consequences of specifying it like this anyway. Thoughts? I'm going to start putting together a wrapper to call the PEP517 hooks in a subprocess, which could be used by frontends. Thomas From thomas at kluyver.me.uk Thu Jun 15 10:18:45 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Thu, 15 Jun 2017 15:18:45 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: <1497536325.3916684.1010463344.6990EAE3@webmail.messagingengine.com> On Thu, Jun 15, 2017, at 03:10 PM, C Anthony Risinger wrote: > I'm not trying to open a bikeshedding opportunity here -- and I tried > to ignore it, honest! -- but why are tarballs preferable to zipfiles > for sdists?> > ... > > Just seems a little odd/arbitrary to me that wheel is zip, python > supports zip importing, sdists are often zip, and Windows is zip- > central, but we'd decide to codify tar.gz. I've wondered about that. My rationale for specifying tar.gz in PEP 517 is that we had decided in a previous discussion to move towards standardising on that - specifically, to change distutils/setuptools to default to making tar.gz sdists everywhere, rather than defaulting to zip on Windows. >From what I recall of that discussion, the rationale was: 1. A significant majority of existing sdists are in tar.gz format, so standardising that is less disruptive than standardising zip.2. Zip files are better when you want to access individual files from the archive, but tarballs are marginally better when you want to unpack the whole thing into a directory before using it. We think that sdists are normally used by unpacking them. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Thu Jun 15 10:10:52 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Thu, 15 Jun 2017 09:10:52 -0500 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan wrote: > On 13 June 2017 at 19:44, Thomas Kluyver wrote: > > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: > > > I've updated the PR to specify zip archives for build_wheel and .tar.gz > > archives for build_sdist. > > +1 > > I've added one suggestion, which is to explicitly require PAX_FORMAT > for the sdist tarballs produced this way (that's a POSIX format > standardised in 2001 and supported by both 2.7 and 3.x that > specifically requires that the paths be encoded as UTF-8). While the > standard library does still default to GNU_FORMAT in general, the > stated rationale for doing so (it being more widely supported than > PAX_FORMAT) was last updated more than 10 years ago, and I don't think > it applies here. I'm not trying to open a bikeshedding opportunity here -- and I tried to ignore it, honest! -- but why are tarballs preferable to zipfiles for sdists? I looked around the 517 threads to see if it had been covered already, and all I found was that zipfiles have additional PKG-INFO expectations in existing implementations, and other honorable mentions of their features over tarballs. I've never understood the anti-affinity towards zip because the format itself seems superior in many ways, such as the ability to easily append or replace-via-append (which might actually help perf when being used as an interchange format, with a repack/prune at the end), compress individual files, and the brilliance of placing the central directory/manifest at the end, allowing it to be appended to binaries, etc. and allowing rapid indexing of files. Tarballs are a black box. Just seems a little odd/arbitrary to me that wheel is zip, python supports zip importing, sdists are often zip, and Windows is zip-central, but we'd decide to codify tar.gz. It doesn't affect me personally because I'm Linux all the way down and barely remember how to use Windows, but with all the existing zip usage, and technical superiority(?), if we are going to pick something, why not that? At that point Python is all-zip and no-tar. It's not a strong opinion really, but since the PEP does attempt to limit what's currently possible, can we add some verbiage as to why tar.gz is preferred? Or consider it with more scrutiny? -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 15 11:12:12 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Jun 2017 11:12:12 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: > On Jun 15, 2017, at 10:10 AM, C Anthony Risinger wrote: > > On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan > wrote: > On 13 June 2017 at 19:44, Thomas Kluyver > wrote: > > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: > > > I've updated the PR to specify zip archives for build_wheel and .tar.gz > > archives for build_sdist. > > +1 > > I've added one suggestion, which is to explicitly require PAX_FORMAT > for the sdist tarballs produced this way (that's a POSIX format > standardised in 2001 and supported by both 2.7 and 3.x that > specifically requires that the paths be encoded as UTF-8). While the > standard library does still default to GNU_FORMAT in general, the > stated rationale for doing so (it being more widely supported than > PAX_FORMAT) was last updated more than 10 years ago, and I don't think > it applies here. > > I'm not trying to open a bikeshedding opportunity here -- and I tried to ignore it, honest! -- but why are tarballs preferable to zipfiles for sdists? > > I looked around the 517 threads to see if it had been covered already, and all I found was that zipfiles have additional PKG-INFO expectations in existing implementations, and other honorable mentions of their features over tarballs. > > I've never understood the anti-affinity towards zip because the format itself seems superior in many ways, such as the ability to easily append or replace-via-append (which might actually help perf when being used as an interchange format, with a repack/prune at the end), compress individual files, and the brilliance of placing the central directory/manifest at the end, allowing it to be appended to binaries, etc. and allowing rapid indexing of files. Tarballs are a black box. > > Just seems a little odd/arbitrary to me that wheel is zip, python supports zip importing, sdists are often zip, and Windows is zip-central, but we'd decide to codify tar.gz. It doesn't affect me personally because I'm Linux all the way down and barely remember how to use Windows, but with all the existing zip usage, and technical superiority(?), if we are going to pick something, why not that? At that point Python is all-zip and no-tar. > > It's not a strong opinion really, but since the PEP does attempt to limit what's currently possible, can we add some verbiage as to why tar.gz is preferred? Or consider it with more scrutiny? > Basically it?s the least disruptive option, the vast bulk of sdists are using ``.tar.gz`` already, multiple downstream redistributors need to do extra work to consume a .zip rather than a .tar.gz, and the technical benefits of wheel don?t really matter much in the context of a sdist. Zip isn?t a flat out win technical wise either, for instance .tar.gz can compress smaller than a .zip file because it?s compression will act over the entire set of files and not on a per file basis. But mostly it?s just that most sdists are .tar.gz, and most Pythons except older ones on Windows default to producing .tar.gz. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jun 15 16:29:50 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 15 Jun 2017 13:29:50 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On Wed, Jun 14, 2017 at 1:49 PM, Donald Stufft wrote: > > > On Jun 12, 2017, at 9:27 PM, Nick Coghlan wrote: > > > 5) Drop the prepare_files hook > Rationale: it's purpose is somewhat unclear at this point, and it > gives up the main advantage of going through sdist (the guarantee that > building from sdist and building from unpacked tree give the same > results) while still having the main disadvantages (you have to copy > everything and lose incremental rebuilds). > > > The main reason this made it into the PEP is that build_sdist may have > additional dependencies that prepare_build_files doesn't (e.g. Thomas > has indicated that flit needs to call out to VCS tools to build a full > sdist, but can export its own input files directly). > > So while it's technically a "make it faster" hook rather than an > essential "make it right" hook, I think just smoothing out the > interaction between pip & flit specifically provides sufficient value > to make it worth including in the initial version of the > specification. > > > Just as an anecdote, today requests pushed a brown paper bag release > because they created a wheel out of a checkout of the requests repository, > which happened to have some .pyc files and directories left over from git > and some refactoring. This caused the library to be completely broken as > released in the wheel. This is the kind of error that creating a sdist > ideally hopes to eliminate (and hopefully the make it faster hook doesn?t > end up invalidating that point). > To clarify my position: - My grumpiness last week was at the discussion around PEP 517 moving towards assuming that tree->sdist->wheel is the *only* path that matters. That's not something we can mandate in theory or in practice. - If pip specifically wants to insist on generating sdists every time it builds from a checkout, then I'm dubious but I've made my case and I can live with the pip maintainers' decision. - However, I am more dubious about hacking stuff like prepare_build_files into the standard to support this specific interaction between two specific tools, especially since it's not at all obvious to me that this is the best way to support even those two tools. Our two concrete examples are: 1. distutils/setuptools, which clearly aren't to be trusted (the requests example is great. I mean, it's wince-inducing, but it's a great and convincing example why the gimme-release-artifacts command needs to go through sdist). If you're worried about getting a reliable build here, the solution is to build an sdist; prepare_build_files is irrelevant. 2. flit, which is totally trustworthy (in this regard), and for whom the prepare_build_files *also* accomplishes nothing useful, since it will use exactly the same logic to copy files from the tree->fake unpacked sdist->wheel as it would if going from tree->wheel directly; there's literally no difference. It's a bit cargo-culty isn't it? It makes flit's install path look more like the distutils/setuptools one, but either we trust flit to produce consistent results whether building from a tree or sdist and it's unnecessary, or else we don't trust flit to produce consistent results and then it's harmful and we should insist on a proper sdist. What problem is this solving? I don't really feel like we've fully explored this design space either. Can flit really not make an sdist from an sdist? Or if it really can't, then we should have some way to express this in the spec (maybe build_sdist signals this by raising NotImplementedError?), and then pip needs to have some strategy for dealing with this. If pip finds itself trying to build something where the build backend says it can't make an sdist and can't do prepare_build_files, then it needs to fall back on something, which I guess is either shutil.copytree or doing an in-place build. Either seems about as good as prepare_build_files to me, and they don't require risky standardization work. prepare_build_files is by far the most speculative part of the standard, because it's trying to solve on paper problems we haven't yet solved in code. That makes me nervous. It doesn't make me *as* nervous as if this were a mandatory part of the spec, because if it turns out in a few years that no-one actually uses it, then we can drop it from both frontends and backends and just sort of pretend it never existed. But it makes me somewhat nervous. -n -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jun 15 16:43:03 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 15 Jun 2017 13:43:03 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> Message-ID: On Thu, Jun 15, 2017 at 6:12 AM, Thomas Kluyver wrote: > Yes, it's PEP 517 again! Here's the current text: > > https://www.python.org/dev/peps/pep-0517/ > > We currently say: > >> All other hooks [except get_build_requires] are executed in an environment which contains both the bootstrap requirements specified in the pyproject.toml hook and those specified by the get_build_requires hook. > > It's not clear to me whether this should be required for the build_sdist > and prepare_build_files hooks, nor whether there are any adverse > consequences of specifying it like this anyway. Thoughts? I think we should rename get_build_requires to get_build_wheel_requires, and add a get_build_sdist_requires. And the rule would be: get_build_sdist_requires: can assume build-system.requires are available get_build_wheel_requires: can assume build-system.requires are available build_sdist: can assume build-system.requires and get_build_sdist_requires are available prepare_wheel_metadata, build_wheel: can assume build-system.requires and get_build_wheel_requires are available Rationale: (a) conceptually the sdist and wheel phases are totally separate, so we shouldn't couple them by requiring a single hook to provide the union of requirements for both. (b) there are known cases where building an sdist has different requirements than building a wheel. Examples: packages that run cython at sdist generation time. (This is not *as* necessary if we have robust build-time requirement support, but that's only one of the motivations for projects to do this now; others include avoiding a potentially expensive step for users -- installing cython in slow -- and avoiding the risk of later cython changes breaking an existing release.) In the broader world, it's very common to run automake/autotools etc. to generate ./configure scripts at sdist generation time. Maybe a python web app needs node installed to run its asset pipeline so they do that at sdist generation time rather than forcing end-users to install node. Etc. It's not immediately obvious to me how prepare_build_files would fit into this; if it's only supposed to be used for building from an sdist, then it's like an extra half-phase in between the sdist and wheel phases -- maybe it's more part of the wheel phase and should get the wheel requirements? Is that the only time it's used? I guess I'll wait to worry about it until after I see how people respond to my argument in the other thread that prepare_build_files shouldn't exist at all :-). -n -- Nathaniel J. Smith -- https://vorpus.org From dholth at gmail.com Thu Jun 15 16:49:56 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 Jun 2017 20:49:56 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: Nothing wrong with sdist of course, but one thing I hear from the emphasis on sdist (something that pip can build automatically) is that maybe we are worried that a publisher, who gave us the software for free, will unduly inconvenience the consumer. But that would require an obligation from the producer to the consumer that I don't think exists. The publisher can make their software as useful or useless as possible, including not publishing it at all. In practice many open source publishers through their own intrinsic and mysterious motivations write and publish software that is both useful and easy to consume. If they are not obligated to publish, how can they be obligated to publish an sdist? even though they will likely do the latter in most cases when doing the former. On Thu, Jun 15, 2017 at 4:30 PM Nathaniel Smith wrote: > On Wed, Jun 14, 2017 at 1:49 PM, Donald Stufft wrote: > >> >> >> On Jun 12, 2017, at 9:27 PM, Nick Coghlan wrote: >> >> >> 5) Drop the prepare_files hook >> Rationale: it's purpose is somewhat unclear at this point, and it >> gives up the main advantage of going through sdist (the guarantee that >> building from sdist and building from unpacked tree give the same >> results) while still having the main disadvantages (you have to copy >> everything and lose incremental rebuilds). >> >> >> The main reason this made it into the PEP is that build_sdist may have >> additional dependencies that prepare_build_files doesn't (e.g. Thomas >> has indicated that flit needs to call out to VCS tools to build a full >> sdist, but can export its own input files directly). >> >> So while it's technically a "make it faster" hook rather than an >> essential "make it right" hook, I think just smoothing out the >> interaction between pip & flit specifically provides sufficient value >> to make it worth including in the initial version of the >> specification. >> >> >> Just as an anecdote, today requests pushed a brown paper bag release >> because they created a wheel out of a checkout of the requests repository, >> which happened to have some .pyc files and directories left over from git >> and some refactoring. This caused the library to be completely broken as >> released in the wheel. This is the kind of error that creating a sdist >> ideally hopes to eliminate (and hopefully the make it faster hook doesn?t >> end up invalidating that point). >> > > To clarify my position: > > - My grumpiness last week was at the discussion around PEP 517 moving > towards assuming that tree->sdist->wheel is the *only* path that matters. > That's not something we can mandate in theory or in practice. > > - If pip specifically wants to insist on generating sdists every time it > builds from a checkout, then I'm dubious but I've made my case and I can > live with the pip maintainers' decision. > > - However, I am more dubious about hacking stuff like prepare_build_files > into the standard to support this specific interaction between two specific > tools, especially since it's not at all obvious to me that this is the best > way to support even those two tools. Our two concrete examples are: > > 1. distutils/setuptools, which clearly aren't to be trusted (the requests > example is great. I mean, it's wince-inducing, but it's a great and > convincing example why the gimme-release-artifacts command needs to go > through sdist). If you're worried about getting a reliable build here, the > solution is to build an sdist; prepare_build_files is irrelevant. > > 2. flit, which is totally trustworthy (in this regard), and for whom the > prepare_build_files *also* accomplishes nothing useful, since it will use > exactly the same logic to copy files from the tree->fake unpacked > sdist->wheel as it would if going from tree->wheel directly; there's > literally no difference. It's a bit cargo-culty isn't it? It makes flit's > install path look more like the distutils/setuptools one, but either we > trust flit to produce consistent results whether building from a tree or > sdist and it's unnecessary, or else we don't trust flit to produce > consistent results and then it's harmful and we should insist on a proper > sdist. What problem is this solving? > > I don't really feel like we've fully explored this design space either. > Can flit really not make an sdist from an sdist? Or if it really can't, > then we should have some way to express this in the spec (maybe build_sdist > signals this by raising NotImplementedError?), and then pip needs to have > some strategy for dealing with this. If pip finds itself trying to build > something where the build backend says it can't make an sdist and can't do > prepare_build_files, then it needs to fall back on something, which I guess > is either shutil.copytree or doing an in-place build. Either seems about as > good as prepare_build_files to me, and they don't require risky > standardization work. > > prepare_build_files is by far the most speculative part of the standard, > because it's trying to solve on paper problems we haven't yet solved in > code. That makes me nervous. It doesn't make me *as* nervous as if this > were a mandatory part of the spec, because if it turns out in a few years > that no-one actually uses it, then we can drop it from both frontends and > backends and just sort of pretend it never existed. But it makes me > somewhat nervous. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jun 15 17:05:48 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Jun 2017 22:05:48 +0100 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: On 15 June 2017 at 21:49, Daniel Holth wrote: > Nothing wrong with sdist of course, but one thing I hear from the emphasis > on sdist (something that pip can build automatically) is that maybe we are > worried that a publisher, who gave us the software for free, will unduly > inconvenience the consumer. But that would require an obligation from the > producer to the consumer that I don't think exists. The publisher can make > their software as useful or useless as possible, including not publishing it > at all. In practice many open source publishers through their own intrinsic > and mysterious motivations write and publish software that is both useful > and easy to consume. If they are not obligated to publish, how can they be > obligated to publish an sdist? even though they will likely do the latter in > most cases when doing the former. The problem is not about obligation, so much as making the path that we want to promote the easiest. Personally, I want to promote publication of sdists, as I believe publishing source is a key pillar of open source, and publishing it in a standardised, easy to consume form (not "go to my githib/bitbucket/google code page and work out how to download the source") is important. So, for me, making it as easy as possible for software authors to provide sdists is important - and therefore, I am in favour of requiring backends to provide a route to publish a sdist. Of course, it is perfectly OK for backend authors to *not* offer that facility, but again, I see no harm in making it easier for them to integrate if they do offer a "publish a sdist" route. So my support for the sdist proposals is based on my personal beliefs around how to publish sources. Others may have different beliefs, and those beliefs may mean that they support or oppose my position. But there's no requiring obligations here, just people supporting proposals that make things they care about easy, and (sometimes) make things they disagree with harder. Paul From brett at python.org Thu Jun 15 16:55:28 2017 From: brett at python.org (Brett Cannon) Date: Thu, 15 Jun 2017 20:55:28 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: On Thu, 15 Jun 2017 at 08:25 Donald Stufft wrote: > > On Jun 15, 2017, at 10:10 AM, C Anthony Risinger > wrote: > > On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan wrote: > >> On 13 June 2017 at 19:44, Thomas Kluyver wrote: >> > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: >> >> > I've updated the PR to specify zip archives for build_wheel and .tar.gz >> > archives for build_sdist. >> >> +1 >> >> I've added one suggestion, which is to explicitly require PAX_FORMAT >> for the sdist tarballs produced this way (that's a POSIX format >> standardised in 2001 and supported by both 2.7 and 3.x that >> specifically requires that the paths be encoded as UTF-8). While the >> standard library does still default to GNU_FORMAT in general, the >> stated rationale for doing so (it being more widely supported than >> PAX_FORMAT) was last updated more than 10 years ago, and I don't think >> it applies here. > > > I'm not trying to open a bikeshedding opportunity here -- and I tried to > ignore it, honest! -- but why are tarballs preferable to zipfiles for > sdists? > > I looked around the 517 threads to see if it had been covered already, and > all I found was that zipfiles have additional PKG-INFO expectations in > existing implementations, and other honorable mentions of their features > over tarballs. > > I've never understood the anti-affinity towards zip because the format > itself seems superior in many ways, such as the ability to easily append or > replace-via-append (which might actually help perf when being used as an > interchange format, with a repack/prune at the end), compress individual > files, and the brilliance of placing the central directory/manifest at the > end, allowing it to be appended to binaries, etc. and allowing rapid > indexing of files. Tarballs are a black box. > > Just seems a little odd/arbitrary to me that wheel is zip, python supports > zip importing, sdists are often zip, and Windows is zip-central, but we'd > decide to codify tar.gz. It doesn't affect me personally because I'm Linux > all the way down and barely remember how to use Windows, but with all the > existing zip usage, and technical superiority(?), if we are going to pick > something, why not that? At that point Python is all-zip and no-tar. > > Yeah, the inconsistency bugs me as well and I was about to email about this until this started up. :) > > It's not a strong opinion really, but since the PEP does attempt to limit > what's currently possible, can we add some verbiage as to why tar.gz is > preferred? Or consider it with more scrutiny? > > > > > Basically it?s the least disruptive option, the vast bulk of sdists are > using ``.tar.gz`` already, multiple downstream redistributors need to do > extra work to consume a .zip rather than a .tar.gz, and the technical > benefits of wheel don?t really matter much in the context of a sdist. Zip > isn?t a flat out win technical wise either, for instance .tar.gz can > compress smaller than a .zip file because it?s compression will act over > the entire set of files and not on a per file basis. > Then shouldn't we be pushing for .tar.xz instead? (The Rust community is actually moving to .tar.xz for distributing Rust itself: https://users.rust-lang.org/t/rustup-1-4-0-released/11268 ; I don't know what their plans are for crates) > > But mostly it?s just that most sdists are .tar.gz, and most Pythons except > older ones on Windows default to producing .tar.gz. > Well, I've been actively overriding that default and uploading only .zip files since it's so much easier to work with zip files on Windows and UNIX than tar.gz which are only easy on UNIX. :) And it is rather ironic that historically projects lack a Windows wheel and thus need the sdist and yet it's typically in a format that's the most painful to work with on Windows. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 15 22:24:50 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Jun 2017 22:24:50 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: <145E4F48-671B-46C8-85D0-ABF994BBB288@stufft.io> > On Jun 15, 2017, at 4:29 PM, Nathaniel Smith wrote: > > - However, I am more dubious about hacking stuff like prepare_build_files into the standard to support this specific interaction between two specific tools, especially since it's not at all obvious to me that this is the best way to support even those two tools. Our two concrete examples are: > I don?t have a strong feeling here TBH. I would prefer if we just always sent things through build_sdist (and we said that projects should be able to at a minimum no-op copy/paste files from an already existing unpacked sdist, how they determine that is up to them). However I can see the argument that there could be an optional hook that something like pip *could* call (but probably shouldn?t be mandatory to call it?). I don?t know, the prepare_build_files hook is weird, and I would not personally feel bad if we got rid of it. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 15 22:43:07 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Jun 2017 22:43:07 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: <4C1F3183-819D-42F2-9D32-CAA125B1DA3D@stufft.io> > On Jun 15, 2017, at 4:55 PM, Brett Cannon wrote: > > > > On Thu, 15 Jun 2017 at 08:25 Donald Stufft > wrote: > >> On Jun 15, 2017, at 10:10 AM, C Anthony Risinger > wrote: >> >> On Tue, Jun 13, 2017 at 8:53 PM, Nick Coghlan > wrote: >> On 13 June 2017 at 19:44, Thomas Kluyver > wrote: >> > On Tue, Jun 13, 2017, at 02:27 AM, Nick Coghlan wrote: >> >> > I've updated the PR to specify zip archives for build_wheel and .tar.gz >> > archives for build_sdist. >> >> +1 >> >> I've added one suggestion, which is to explicitly require PAX_FORMAT >> for the sdist tarballs produced this way (that's a POSIX format >> standardised in 2001 and supported by both 2.7 and 3.x that >> specifically requires that the paths be encoded as UTF-8). While the >> standard library does still default to GNU_FORMAT in general, the >> stated rationale for doing so (it being more widely supported than >> PAX_FORMAT) was last updated more than 10 years ago, and I don't think >> it applies here. >> >> I'm not trying to open a bikeshedding opportunity here -- and I tried to ignore it, honest! -- but why are tarballs preferable to zipfiles for sdists? >> >> I looked around the 517 threads to see if it had been covered already, and all I found was that zipfiles have additional PKG-INFO expectations in existing implementations, and other honorable mentions of their features over tarballs. >> >> I've never understood the anti-affinity towards zip because the format itself seems superior in many ways, such as the ability to easily append or replace-via-append (which might actually help perf when being used as an interchange format, with a repack/prune at the end), compress individual files, and the brilliance of placing the central directory/manifest at the end, allowing it to be appended to binaries, etc. and allowing rapid indexing of files. Tarballs are a black box. >> >> Just seems a little odd/arbitrary to me that wheel is zip, python supports zip importing, sdists are often zip, and Windows is zip-central, but we'd decide to codify tar.gz. It doesn't affect me personally because I'm Linux all the way down and barely remember how to use Windows, but with all the existing zip usage, and technical superiority(?), if we are going to pick something, why not that? At that point Python is all-zip and no-tar. > > > Yeah, the inconsistency bugs me as well and I was about to email about this until this started up. :) > >> >> It's not a strong opinion really, but since the PEP does attempt to limit what's currently possible, can we add some verbiage as to why tar.gz is preferred? Or consider it with more scrutiny? >> > > > Basically it?s the least disruptive option, the vast bulk of sdists are using ``.tar.gz`` already, multiple downstream redistributors need to do extra work to consume a .zip rather than a .tar.gz, and the technical benefits of wheel don?t really matter much in the context of a sdist. Zip isn?t a flat out win technical wise either, for instance .tar.gz can compress smaller than a .zip file because it?s compression will act over the entire set of files and not on a per file basis. > > Then shouldn't we be pushing for .tar.xz instead? (The Rust community is actually moving to .tar.xz for distributing Rust itself: https://users.rust-lang.org/t/rustup-1-4-0-released/11268 ; I don't know what their plans are for crates) Absent all other rationale, yes pushing for .tar.xz would be better (as would using the ZIP_LZMA option). However, that would cut out support for Python 2.7 by default and it would require an optional library to exist in Python?s standard library (.tar.gz and .zip does too, but that?s zlib which is near ubiquitous). It?s a balancing act, and a format that something like 80-90% of the downloads from PyPI couldn?t support is way out of balance. > > > But mostly it?s just that most sdists are .tar.gz, and most Pythons except older ones on Windows default to producing .tar.gz. > > Well, I've been actively overriding that default and uploading only .zip files since it's so much easier to work with zip files on Windows and UNIX than tar.gz which are only easy on UNIX. :) And it is rather ironic that historically projects lack a Windows wheel and thus need the sdist and yet it's typically in a format that's the most painful to work with on Windows. Eh, if you have Python it?s not very hard to work with .tar.gz files, ``python3 -m -e foo-1.0.tar.gz`` etc. Honestly though, any argument about the technical merits of one or the other basically pales to the real argument, in that standardizing around .tar.gz means the least amount of change in the ecosystem. We have 591,653 .tar.gz sdists uploaded but only 68,816 .zip using sdists. (Or 86,845 // 13,701 for unique projects) [1]. It?s more disruptive to tell the 86k projects that they need to change instead of the 13k, and absent any really compelling reason that isn?t just relatively minor benefits, reducing churn should be our number one concern. [1] We can play with these numbers in lots of different way, like for instance in the past year 36,687 projects have released a .tar.gz using sdist but only 3,719 have released a .zip using sdist. At the end of the day though, unless you?re really trying all the numbers end up amounting to roughly an order of magnitude more projects/files/whatever are using .tar.gz than .zip. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 15 22:50:31 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Jun 2017 22:50:31 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> Message-ID: > On Jun 15, 2017, at 5:05 PM, Paul Moore wrote: > > On 15 June 2017 at 21:49, Daniel Holth wrote: >> Nothing wrong with sdist of course, but one thing I hear from the emphasis >> on sdist (something that pip can build automatically) is that maybe we are >> worried that a publisher, who gave us the software for free, will unduly >> inconvenience the consumer. But that would require an obligation from the >> producer to the consumer that I don't think exists. The publisher can make >> their software as useful or useless as possible, including not publishing it >> at all. In practice many open source publishers through their own intrinsic >> and mysterious motivations write and publish software that is both useful >> and easy to consume. If they are not obligated to publish, how can they be >> obligated to publish an sdist? even though they will likely do the latter in >> most cases when doing the former. > > The problem is not about obligation, so much as making the path that > we want to promote the easiest. > > Personally, I want to promote publication of sdists, as I believe > publishing source is a key pillar of open source, and publishing it in > a standardised, easy to consume form (not "go to my > githib/bitbucket/google code page and work out how to download the > source") is important. So, for me, making it as easy as possible for > software authors to provide sdists is important - and therefore, I am > in favour of requiring backends to provide a route to publish a sdist. > Of course, it is perfectly OK for backend authors to *not* offer that > facility, but again, I see no harm in making it easier for them to > integrate if they do offer a "publish a sdist" route. > > So my support for the sdist proposals is based on my personal beliefs > around how to publish sources. Others may have different beliefs, and > those beliefs may mean that they support or oppose my position. > > But there's no requiring obligations here, just people supporting > proposals that make things they care about easy, and (sometimes) make > things they disagree with harder. > While I agree publishers here are not being mandated or obligated to do anything (we don?t require they publish sdists *at all*, though we strongly encourage it) I want to push back against this idea that just because the publisher decided to release software for free means we can?t place any constraints on how they release that software. As a platform PyPI can and does impose constraints (for instance, you have to pick a version that fits with PEP 440 to upload to PyPI). If someone finds those constraints onerous they?re free to publish their software elsewhere or to agitate for a relaxation of those constraints. For each specific constraint we may or may not apply, we can argue the merits of that specific constraint and whether the benefits outweigh the burden, however the idea that we can?t impose constraints on an author is entirely absurd. In the end we have just as much responsibility to the consumers of PyPI to ensure that they have a good ecosystem as we do to publishers to ensure we?re not unduly getting in their way. Sometimes that means applying a constraint to ensure some level of uniformity across PyPI other times that means giving authors power to manage their project as best makes sense for them. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Jun 15 22:51:52 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 15 Jun 2017 19:51:52 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: On Thu, Jun 15, 2017 at 1:55 PM, Brett Cannon wrote: > Then shouldn't we be pushing for .tar.xz instead? (The Rust community is > actually moving to .tar.xz for distributing Rust itself: > https://users.rust-lang.org/t/rustup-1-4-0-released/11268 ; I don't know > what their plans are for crates) There's no xz support in the 2.7 stdlib, so I think that's a non-starter. >> But mostly it?s just that most sdists are .tar.gz, and most Pythons except >> older ones on Windows default to producing .tar.gz. > > > Well, I've been actively overriding that default and uploading only .zip > files since it's so much easier to work with zip files on Windows and UNIX > than tar.gz which are only easy on UNIX. :) And it is rather ironic that > historically projects lack a Windows wheel and thus need the sdist and yet > it's typically in a format that's the most painful to work with on Windows. FWIW I've been doing this for years too :-) [1]. But I don't care that much myself either way. I can see some benefit to standardizing on a single format instead of making every backend author learn e.g. the weird quirks required to get unicode filenames correct in two different formats, and I'm not sure why it would be a big deal to change the default for new tools going forward given that all our infrastructure does support .zip already. -n [1] e.g. https://pypi.org/project/patsy/0.1.0/#files -- Nathaniel J. Smith -- https://vorpus.org From donald at stufft.io Thu Jun 15 22:58:54 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Jun 2017 22:58:54 -0400 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> Message-ID: <0CF50E52-2267-4F0E-8E48-12CDD6DA4E86@stufft.io> > On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: > > I can see some benefit to standardizing on a single format instead of > making every backend author learn e.g. the weird quirks required to > get unicode filenames correct in two different formats, and I'm not > sure why it would be a big deal to change the default for new tools > going forward given that all our infrastructure does support .zip > already. I believe it would be fairly disruptive for downstream redistributors like Debian whose tooling is designed around the idea of a .tar.gz file, and whom are forced to repack .zip files into .tar.gz files AIUI. I don?t really personally care that much about the difference between .zip or .tar.gz for a sdist, I just want to minimize people yelling at me unless it?s for a good reason, and I don?t think this is a good reason :) ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jun 15 23:26:55 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Jun 2017 03:26:55 +0000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <0CF50E52-2267-4F0E-8E48-12CDD6DA4E86@stufft.io> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> <0CF50E52-2267-4F0E-8E48-12CDD6DA4E86@stufft.io> Message-ID: The format you want is zstandard or brotli. Maybe fastly be interested in sponsoring bandwidth efficient file formats? :-) It would be cool to have a way to do signatures and hashing before compression. On Thu, Jun 15, 2017, 22:59 Donald Stufft wrote: > > On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: > > I can see some benefit to standardizing on a single format instead of > making every backend author learn e.g. the weird quirks required to > get unicode filenames correct in two different formats, and I'm not > sure why it would be a big deal to change the default for new tools > going forward given that all our infrastructure does support .zip > already. > > > > I believe it would be fairly disruptive for downstream redistributors like > Debian whose tooling is designed around the idea of a .tar.gz file, and > whom are forced to repack .zip files into .tar.gz files AIUI. I don?t > really personally care that much about the difference between .zip or > .tar.gz for a sdist, I just want to minimize people yelling at me unless > it?s for a good reason, and I don?t think this is a good reason :) > > ? > > Donald Stufft > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 16 00:25:43 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 15 Jun 2017 21:25:43 -0700 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: <0CF50E52-2267-4F0E-8E48-12CDD6DA4E86@stufft.io> References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> <0CF50E52-2267-4F0E-8E48-12CDD6DA4E86@stufft.io> Message-ID: On Thu, Jun 15, 2017 at 7:58 PM, Donald Stufft wrote: > > On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: > > I can see some benefit to standardizing on a single format instead of > making every backend author learn e.g. the weird quirks required to > get unicode filenames correct in two different formats, and I'm not > sure why it would be a big deal to change the default for new tools > going forward given that all our infrastructure does support .zip > already. > > > > I believe it would be fairly disruptive for downstream redistributors like > Debian whose tooling is designed around the idea of a .tar.gz file, and > whom are forced to repack .zip files into .tar.gz files AIUI. I don?t > really personally care that much about the difference between .zip or > .tar.gz for a sdist, I just want to minimize people yelling at me unless > it?s for a good reason, and I don?t think this is a good reason :) > You got me curious, and well, this is Debian we're talking about... judging from [1] it looks like at least someone writing their tooling is prepared for: \.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz))) OTOH I think PEP 517 will break everything else about their tooling, since it all assumes the presence of setup.py? Not that any of this makes an argument for .tar.gz or .zip in particular... that's the hard part about this question, it just doesn't matter very much :-). -n [1] https://wiki.debian.org/Python/LibraryStyleGuide -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 16 03:41:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Jun 2017 17:41:20 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> Message-ID: Thanks for continuing to push us forward on this, Thomas :) A small PEP readability request: given how the number of hooks has grown, could we get a section that just lists the required hooks and the optional hooks? Alternatively, give each hook its own subsection under "Build backend interface", so the table of contents at the top of the PEP serves as a quick summary. On 16 June 2017 at 06:43, Nathaniel Smith wrote: > On Thu, Jun 15, 2017 at 6:12 AM, Thomas Kluyver wrote: >> Yes, it's PEP 517 again! Here's the current text: >> >> https://www.python.org/dev/peps/pep-0517/ >> >> We currently say: >> >>> All other hooks [except get_build_requires] are executed in an environment which contains both the bootstrap requirements specified in the pyproject.toml hook and those specified by the get_build_requires hook. >> >> It's not clear to me whether this should be required for the build_sdist >> and prepare_build_files hooks, nor whether there are any adverse >> consequences of specifying it like this anyway. Thoughts? > > I think we should rename get_build_requires to > get_build_wheel_requires, and add a get_build_sdist_requires. And the > rule would be: > > get_build_sdist_requires: can assume build-system.requires are available > get_build_wheel_requires: can assume build-system.requires are available > > build_sdist: can assume build-system.requires and > get_build_sdist_requires are available > prepare_wheel_metadata, build_wheel: can assume build-system.requires > and get_build_wheel_requires are available +1 from me > Rationale: (a) conceptually the sdist and wheel phases are totally > separate, so we shouldn't couple them by requiring a single hook to > provide the union of requirements for both. (b) there are known cases > where building an sdist has different requirements than building a > wheel. Examples: packages that run cython at sdist generation time. Another example: Thomas expects flit to require VCS interaction support for sdist generation, but not for wheel building (or wheel build file preparation). > It's not immediately obvious to me how prepare_build_files would fit > into this; if it's only supposed to be used for building from an > sdist, then it's like an extra half-phase in between the sdist and > wheel phases -- maybe it's more part of the wheel phase and should get > the wheel requirements? Is that the only time it's used? I guess I'll > wait to worry about it until after I see how people respond to my > argument in the other thread that prepare_build_files shouldn't exist > at all :-). As with get_build_requires, this was less ambiguous when "build_wheel" was the only build hook. Now that we have "build_sdist" as well, then it may make more sense to rename it to "prepare_wheel_input_files". Something else that I believe the PEP currently leaves implicit is the assumptions that backend hook implementations are allowed to make regarding the current working directory when they're invoked. As I understand it, there are three defined possibilities: - original source tree (potentially VCS metadata, no PKG-INFO file) - unpacked sdist (no VCS metadata, PKG-INFO file) - prepared wheel input files (no VCS metadata, maybe PKG-INFO file) Given those options, the hooks that can be called given a particular kind of input directory are: * original source tree: * all hooks * unpacked sdist: * all hooks * prepared wheel input files (if `prepare_wheel_input_files` is defined): * prepare_wheel_metadata * build_wheel * NOT get_build_wheel_requires (see below) My rationale for requiring get_build_wheel_requires to be called in the source directory is that it means that `prepare_wheel_input_files` can rely on those dynamic dependencies, appropriately reflecting it's status as an optional substep of the `build wheel` process. The available dependencies for each hook are then: * build-system.requires only: * get_build_sdist_requires * get_build_wheel_requires * build-system.requires + get_build_sdist_requires: * build_sdist * build-system.requires + get_build_wheel_requires: * prepare_wheel_input_files * prepare_wheel_metadata * build_wheel Make sense? If folks agree with that, we could make the above explicit in the PEP using a pair of dedicated paragraphs in each hook description: * One starting "Current working directory: ..." * One starting "Available dependencies: ..." Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jun 16 04:13:49 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Jun 2017 18:13:49 +1000 Subject: [Distutils] PEP 517: Open questions around artifact export directories In-Reply-To: References: <1497085937.2484993.1004923848.578343ED@webmail.messagingengine.com> <934E30EB-F5FA-4025-9270-94F8B97E0763@stufft.io> <1497297681.2012358.1007068408.757E43CA@webmail.messagingengine.com> <502D6026-5EEB-4B90-B7EB-8E42D6623E7A@stufft.io> <1497301043.2029314.1007123120.50B8AF89@webmail.messagingengine.com> <1497347098.2975282.1007679432.57141A28@webmail.messagingengine.com> <0CF50E52-2267-4F0E-8E48-12CDD6DA4E86@stufft.io> Message-ID: On 16 June 2017 at 14:25, Nathaniel Smith wrote: > On Thu, Jun 15, 2017 at 7:58 PM, Donald Stufft wrote: >> >> >> On Jun 15, 2017, at 10:51 PM, Nathaniel Smith wrote: >> >> I can see some benefit to standardizing on a single format instead of >> making every backend author learn e.g. the weird quirks required to >> get unicode filenames correct in two different formats, and I'm not >> sure why it would be a big deal to change the default for new tools >> going forward given that all our infrastructure does support .zip >> already. >> >> >> >> I believe it would be fairly disruptive for downstream redistributors like >> Debian whose tooling is designed around the idea of a .tar.gz file, and whom >> are forced to repack .zip files into .tar.gz files AIUI. I don?t really >> personally care that much about the difference between .zip or .tar.gz for a >> sdist, I just want to minimize people yelling at me unless it?s for a good >> reason, and I don?t think this is a good reason :) > > > You got me curious, and well, this is Debian we're talking about... judging > from [1] it looks like at least someone writing their tooling is prepared > for: > > \.(?:zip|tgz|tbz|txz|(?:tar\.(?:gz|bz2|xz))) > > OTOH I think PEP 517 will break everything else about their tooling, since > it all assumes the presence of setup.py? Not that any of this makes an > argument for .tar.gz or .zip in particular... that's the hard part about > this question, it just doesn't matter very much :-). Which is why the "current popularity" argument carries the day. It doesn't really matter all that much (since Python can handle both regardless of platform, and we don't care all that about the ease-of-use of working with sdist's without unpacking them first), and tar.gz is an order of magnitude more popular on PyPI (with no likelihood of that ratio ever shifting significantly in zip's favour), so tar.gz wins. If current popularity doesn't feel persuasive, then we can also go with "BDFL-Delegate fiat", since that also applies here :) The previous discussion on this is covered in https://www.python.org/dev/peps/pep-0527/, which changed PyPI to only allow uploads of sdists as .tar.gz *or* .zip files (and only one or the other for any given release). In that case, we didn't want to disrupt the projects currently using zip sdists by deliberate choice (rather than as the WIndows default), so we left them alone (but also changed the tar.gz default to be platform independent). For PEP 517 though, we're defining a *new* build interface, so anyone adopting it is necessarily changing their publishing processes anyway. I'm OK with having some folks resent me for making "use tar.gz sdists" a hard requirement for PEP 517 adoption if the pay-off is to make it easier for tool developers to adapt to PEP 517 (since they only need to handle that tar.gz sdist format, leaving zip sdists as purely a setuptools implementation detail) and (given time) for the overall ecosystem to see increased consistency in the format of published sdists. For wheel files, by contrast, we *do* care about supporting random access, so tar was never even considered as an option. That means that if folks are looking for a post hoc rationalisation of a decision that was really made for hysterical raisins [1], then "wheels are designed to support efficient random access, but sdists arent, so wheels use zip, while sdists use tar.gz" works. Cheers, Nick. [1] courtesy of a former teammate, "hysterical raisins" = historical reasons for design decisions that don't necessarily make all that much sense when considered outside their particular historical context :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Fri Jun 16 05:08:44 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 16 Jun 2017 10:08:44 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> Message-ID: <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote: > > I think we should rename get_build_requires to > > get_build_wheel_requires, and add a get_build_sdist_requires. And the > > rule would be: > > > > get_build_sdist_requires: can assume build-system.requires are available > > get_build_wheel_requires: can assume build-system.requires are available > > > > build_sdist: can assume build-system.requires and > > get_build_sdist_requires are available > > prepare_wheel_metadata, build_wheel: can assume build-system.requires > > and get_build_wheel_requires are available > > +1 from me *Sigh*, another hook. It makes sense in context, but I can't shake the feeling that what was a relatively simple spec is steadily turning into a complex monster. I still resent that we're trying to standardise an interface to build sdists at the same time as one to build wheels. > Another example: Thomas expects flit to require VCS interaction > support for sdist generation, but not for wheel building (or wheel > build file preparation). It's not much help for this, though, because I can't specify git as a dependency. > My rationale for requiring get_build_wheel_requires to be called in > the source directory is that it means that `prepare_wheel_input_files` > can rely on those dynamic dependencies, appropriately reflecting it's > status as an optional substep of the `build wheel` process. That makes sense to me. Thomas From dholth at gmail.com Fri Jun 16 14:58:33 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Jun 2017 18:58:33 +0000 Subject: [Distutils] PEP 517: editable installs Message-ID: I noticed PEP 517 uses a prefix for editable installs. What enscons needs for development install, which is cribbed from setuptools, is to perform a build-in-place and write to a directory that accepts Python .pth files, essentially adding the build-in-place directory to PYTHONPATH. Enscons uses the 'purelib' directory for editable installs. It adds the source directory of the development package to the path and writes an .egg-link to the 'purelib' directory. pip knows how to uninstall these. The code is in enscons.setup. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Jun 16 15:13:29 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Jun 2017 19:13:29 +0000 Subject: [Distutils] PEP 517: Python 3 finder? Message-ID: Build systems should be able to run under a different version of Python than the one that is running 'pip install'. Does PEP 517 have anything to say about that? Then a flit back end could have a small amount of Python 2.7 compatible interface code and create the wheel with Python 3 anyway. -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Fri Jun 16 15:31:49 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Fri, 16 Jun 2017 21:31:49 +0200 Subject: [Distutils] PEP 517: Python 3 finder? In-Reply-To: References: Message-ID: I'm wondering about this one as well. Even when it is not a different interpreter version it could also be a different environment. Who "owns" environment variables like PYTHONHOME and PYTHONPATH? On Fri, Jun 16, 2017 at 9:13 PM, Daniel Holth wrote: > Build systems should be able to run under a different version of Python > than the one that is running 'pip install'. Does PEP 517 have anything to > say about that? Then a flit back end could have a small amount of Python > 2.7 compatible interface code and create the wheel with Python 3 anyway. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 16 16:42:01 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 13:42:01 -0700 Subject: [Distutils] PEP 517: editable installs In-Reply-To: References: Message-ID: On Jun 16, 2017 11:59 AM, "Daniel Holth" wrote: I noticed PEP 517 uses a prefix for editable installs. Which part of the pep are you looking at? All I see about editable installs is the note saying that they're deferred to a future pep. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Jun 16 16:47:55 2017 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Jun 2017 20:47:55 +0000 Subject: [Distutils] PEP 517: editable installs In-Reply-To: References: Message-ID: Probably this older version from http://legacy.python.org/dev/peps/pep-0517/ On Fri, Jun 16, 2017 at 4:42 PM Nathaniel Smith wrote: > On Jun 16, 2017 11:59 AM, "Daniel Holth" wrote: > > I noticed PEP 517 uses a prefix for editable installs. > > > Which part of the pep are you looking at? All I see about editable > installs is the note saying that they're deferred to a future pep. > > -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 16 17:01:20 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 14:01:20 -0700 Subject: [Distutils] PEP 517: Python 3 finder? In-Reply-To: References: Message-ID: On Jun 16, 2017 12:13 PM, "Daniel Holth" wrote: Build systems should be able to run under a different version of Python than the one that is running 'pip install'. Does PEP 517 have anything to say about that? No, and I don't think it should. At the start of the bootstrap process, the *only* thing we know about the system doing the build is that the python being used to run pip is installed. Even if the build system could declare a requirement for some other version of python, then what could we even do with that information? pip3 install python==2.7? I guess I can imagine building some system that tries to find other interpreters, or where you can configure a database of installed interpreters that pip reads, or even making it so that you actually can do the equivalent of "pip3 install python==2.7". But that's all *way* out of scope for this initial PEP. Then a flit back end could have a small amount of Python 2.7 compatible interface code and create the wheel with Python 3 anyway. If someone wants to experiment with this, then it's possible within the PEP 517 framework to write a 2.7-compatible backend that searches the system for a python 3 install and then uses it. I'm not sure it's a good idea, but you can do it :-). For flit in particular I suspect this is unnecessary though. I see three cases: 1. Developers building wheels to release: they can use python 3, no big deal. The resulting wheels will be tagged as py2 or py2.py3 as appropriate. 2. End users installing from pypi: there are wheels on pypi so they never need to run flit. (Key observation here: flit wheels are always pure python, so they work everywhere the project is supported all.) 3. End users who have for whatever reason decided to manually get a source tree (via git checkout or unpacking an sdist), and want to install it: if they can manually get a source tree, they can also manually read and follow directions to use python 3 to build it :-) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 16 17:03:11 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 14:03:11 -0700 Subject: [Distutils] PEP 517: Python 3 finder? In-Reply-To: References: Message-ID: There's a canonical way to say "I want to run another python process in the same environment as the one I'm already running in". You look at sys.executable and run the binary named there. On Jun 16, 2017 12:32 PM, "Freddy Rietdijk" wrote: > I'm wondering about this one as well. Even when it is not a different > interpreter version it could also be a different environment. Who "owns" > environment variables like PYTHONHOME and PYTHONPATH? > > On Fri, Jun 16, 2017 at 9:13 PM, Daniel Holth wrote: > >> Build systems should be able to run under a different version of Python >> than the one that is running 'pip install'. Does PEP 517 have anything to >> say about that? Then a flit back end could have a small amount of Python >> 2.7 compatible interface code and create the wheel with Python 3 anyway. >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 16 17:06:04 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 14:06:04 -0700 Subject: [Distutils] PEP 517: editable installs In-Reply-To: References: Message-ID: Oh, yeah, that's totally an ancient version of the PEP. That sucks. I thought the legacy.python.org URLs were redirecting to python.org now. Probably we should file a bug with the python.org infra team, but I'm not sure where their tracker is. On Jun 16, 2017 1:48 PM, "Daniel Holth" wrote: > Probably this older version from http://legacy.python.org/ > dev/peps/pep-0517/ > > On Fri, Jun 16, 2017 at 4:42 PM Nathaniel Smith wrote: > >> On Jun 16, 2017 11:59 AM, "Daniel Holth" wrote: >> >> I noticed PEP 517 uses a prefix for editable installs. >> >> >> Which part of the pep are you looking at? All I see about editable >> installs is the note saying that they're deferred to a future pep. >> >> -n >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jun 16 17:48:22 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 14:48:22 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> Message-ID: On Fri, Jun 16, 2017 at 2:08 AM, Thomas Kluyver wrote: > > On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote: > > > I think we should rename get_build_requires to > > > get_build_wheel_requires, and add a get_build_sdist_requires. And the > > > rule would be: > > > > > > get_build_sdist_requires: can assume build-system.requires are available > > > get_build_wheel_requires: can assume build-system.requires are available > > > > > > build_sdist: can assume build-system.requires and > > > get_build_sdist_requires are available > > > prepare_wheel_metadata, build_wheel: can assume build-system.requires > > > and get_build_wheel_requires are available > > > > +1 from me > > *Sigh*, another hook. It makes sense in context, but I can't shake the > feeling that what was a relatively simple spec is steadily turning into > a complex monster. I still resent that we're trying to standardise an > interface to build sdists at the same time as one to build wheels. Yeah. Well, except I'm not *too* bothered by the sdist part in particular, since the spec already had to define what an sdist was. And if you zoom out then we have get_build_{wheel,sdist}_requires which have identical interfaces, and build_{wheel,sdist} which also have identical interfaces, so the total complexity for this part isn't too high at all. The messy complications come from prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't surprising, since those are the two hooks where we're squinting into our crystal ball to try and guess what will be useful for software that doesn't exist yet, but will later, maybe, we hope. Hmm, here's another plea for simplicity, but from a slightly different direction that I just thought of: what if we said that any hooks starting with "ext_pip_..." are reserved for pip's use, and pip can make up whatever semantics it likes for them. And then as the parts of pip that actually want to use prepare_wheel_metadata and/or get_prepared_wheel_input_files come online, we use the ext_pip_* versions of those hooks to prototype them and work out any issues. And then once there's an actual implementation and proven value, we write a new PEP to drop the ext_pip_ prefix and make them standard. What do you think? -n -- Nathaniel J. Smith -- https://vorpus.org From thomas at kluyver.me.uk Fri Jun 16 17:54:01 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 16 Jun 2017 22:54:01 +0100 Subject: [Distutils] PEP 517: Python 3 finder? In-Reply-To: References: Message-ID: <1497650041.3193900.1012080808.5339109B@webmail.messagingengine.com> On Fri, Jun 16, 2017, at 10:01 PM, Nathaniel Smith wrote: > On Jun 16, 2017 12:13 PM, "Daniel Holth" wrote: >> Build systems should be able to run under a different version of >> Python than the one that is running 'pip install'. Does PEP 517 have >> anything to say about that?> > No, and I don't think it should. I agree. This would add considerable complexity to the spec, and then create a headache for frontend implementors - how would pip organise running a backend on another version of Python? -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Fri Jun 16 18:05:11 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 16 Jun 2017 23:05:11 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> Message-ID: <1497650711.3196641.1012092480.05BD541D@webmail.messagingengine.com> On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote: > The messy complications come from > prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't > surprising, since those are the two hooks where we're squinting into > our crystal ball to try and guess what will be useful for software > that doesn't exist yet, but will later, maybe, we hope. I'm not exactly clear on what use cases the prepare_wheel_metadata hook satisfies - that was in the spec before I was particularly involved in it. I do think we've hashed out a concrete need for prepare_build_files (or whatever it ends up called): to copy the files to a build directory without either copying large amounts of unnecessary data or figuring out what belongs in an sdist. The only alternative would be to require backends to avoid leaving clutter in the working directory, so they would take care of any necessary copy step rather than exposing a separate hook. People insist that trusting backends is a non-starter, though. From donald at stufft.io Fri Jun 16 20:25:50 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 Jun 2017 20:25:50 -0400 Subject: [Distutils] PEP 517: Python 3 finder? In-Reply-To: References: Message-ID: <2DAAA4B5-453F-4E89-8B72-E66A81B68201@stufft.io> > On Jun 16, 2017, at 5:01 PM, Nathaniel Smith wrote: > > If someone wants to experiment with this, then it's possible within the PEP 517 framework to write a 2.7-compatible backend that searches the system for a python 3 install and then uses it. I'm not sure it's a good idea, but you can do it :-). I just want to hammer this specific point home a bit more. The only real requirement on Python version PEP 517 puts into place is that you write the API that the frontend will call in the version of Python the frontend is running under. What you do beyond that is entirely up to you, including running a different version of Python or something that?s not written in Python at all. The only ?downside" to this really is that your options if that build system isn?t available is to have some sort of fallback (in which case why have the other thing at all?) or just error out. However I don?t think this is a solvable downside, because in either case if you?re running on py3 and the build backend needs py2, there?s nothing really for a frontend to do here except bail out. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 16 20:45:23 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 Jun 2017 20:45:23 -0400 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1497650711.3196641.1012092480.05BD541D@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <1497650711.3196641.1012092480.05BD541D@webmail.messagingengine.com> Message-ID: <85BAB5D6-7B08-4FF7-8697-F43C081A0E5D@stufft.io> > On Jun 16, 2017, at 6:05 PM, Thomas Kluyver wrote: > > On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote: >> The messy complications come from >> prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't >> surprising, since those are the two hooks where we're squinting into >> our crystal ball to try and guess what will be useful for software >> that doesn't exist yet, but will later, maybe, we hope. > > I'm not exactly clear on what use cases the prepare_wheel_metadata hook > satisfies - that was in the spec before I was particularly involved in > it. Basically it exists because when we?re resolving dependencies to decide what set of projects/versions to install, we need a way to take a sdist (because a wheel might not exist!) and decide what dependencies it needs. We ultimately might end up throwing this away because maybe it ends up conflicting and we can?t actually utilize this particular version at all. So we might end up cycling through 10 different versions of the same project looking for a version whose dependencies don?t conflict with the rest of what we?re trying to install. Without prepare_wheel_metadata our only real option is to just build the wheel and inspect that. For a tool like flit that is probably fine, because since it?s pure Python only building a wheel is going to be exceedingly quick. However something like Scikit or something where building the wheel might take tens of minutes or longer that can degrade things very quickly. If you imagine something that takes even 5 minutes to go from sdist to wheel, if we need to do that 10 times while looking for a version that we can use, that means resolution ends up taking over an hour just because of that package alone. This cost is tempered somewhat by the fact that in the ideal case we can cache those wheels so in the future resolution can be extremely quick, however not everyone can or will run with caching enabled and even then, ``pip install ?`` taking an hour even just the first time is still a pretty horrible user experience. The goal of prepare_wheel_metadata then is basically allowing us to ask a sdist ?what dependencies would you have, if we were to build you as a wheel? without having to go through the entire build process. This will ideally be much much faster. It?s not *just* dependencies either, in many cases we don?t know the name or version of something because we?re just given a fairly generic tarball (for instance `master.zip` from Github). We need a way to get name/version from that tarball, which would exist in the wheel but we end up falling into the same ?but what if things take forever to build? problem. Another possible solution is to go down the path of trying to make a sdist 2.0 that has this metadata in a static format so we can promise that we can get it without needing to build anything. However not only is that a significant chunk of extra work, but some folks (Nathaniel I think?) has indicated that some projects simply can?t determine their dependencies statically at sdist build time, because it?s going to change at build time (I think the example was something can build against Numpy >=X, but once built against X+5, it has to have Numpy >=X.5 at runtime). > > I do think we've hashed out a concrete need for prepare_build_files (or > whatever it ends up called): to copy the files to a build directory > without either copying large amounts of unnecessary data or figuring out > what belongs in an sdist. The only alternative would be to require > backends to avoid leaving clutter in the working directory, so they > would take care of any necessary copy step rather than exposing a > separate hook. People insist that trusting backends is a non-starter, > though. The other solution of course is to just say that all backends needs to be able to no-op copy a sdist from an existing sdist. So we have three options really: 1) Require backends to be able to no-op copy a sdist from an existing unpacked sdist. However Thomas is against this, as it would make flit?s job harder. 2) Require frontends to trust that backends are going to DTRT with regards to in-place builds and isolation. However myself and the other pip devs were against this, as we feel it is important for pip to do it?s job. 3) Add a hook that let?s the backends copy the files it needs into a staging area, without the need to prepare a full sdist. This is basically (1) except with some compromises to make the use cases that Thomas says makes flit?s job harder easier to deal with. Nathaniel is against (3) for simplicity sake, and personally I would prefer (1) because I think it is simpler and because I think that it shouldn?t be *too* much additional effort for a backend to make a no-op build_sdist in the case they?re being made out of something that is already a sdist. That being said, I am OK with (3) since Thomas believes that is better for flit than (1) and I don?t have any evidence to refute or back up that claim personally. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 16 20:52:55 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 16 Jun 2017 20:52:55 -0400 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> Message-ID: <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> > On Jun 16, 2017, at 5:48 PM, Nathaniel Smith wrote: > > On Fri, Jun 16, 2017 at 2:08 AM, Thomas Kluyver > wrote: >> >> On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote: >>>> I think we should rename get_build_requires to >>>> get_build_wheel_requires, and add a get_build_sdist_requires. And the >>>> rule would be: >>>> >>>> get_build_sdist_requires: can assume build-system.requires are available >>>> get_build_wheel_requires: can assume build-system.requires are available >>>> >>>> build_sdist: can assume build-system.requires and >>>> get_build_sdist_requires are available >>>> prepare_wheel_metadata, build_wheel: can assume build-system.requires >>>> and get_build_wheel_requires are available >>> >>> +1 from me >> >> *Sigh*, another hook. It makes sense in context, but I can't shake the >> feeling that what was a relatively simple spec is steadily turning into >> a complex monster. I still resent that we're trying to standardise an >> interface to build sdists at the same time as one to build wheels. > > Hmm, here's another plea for simplicity, but from a slightly different > direction that I just thought of: what if we said that any hooks > starting with "ext_pip_..." are reserved for pip's use, and pip can > make up whatever semantics it likes for them. And then as the parts of > pip that actually want to use prepare_wheel_metadata and/or > get_prepared_wheel_input_files come online, we use the ext_pip_* > versions of those hooks to prototype them and work out any issues. And > then once there's an actual implementation and proven value, we write > a new PEP to drop the ext_pip_ prefix and make them standard. > I?d probably want to spec this out as ext_{name-on-pypi}_* to remove a special case on pip in the PEP, to let others do experimentation as well. However this solution is fine with me on both of the Non build_{sdist,wheel} hooks. Although I feel like it kind of defeats the purpose of the prepare_build_files hooks, because I think Thomas? goal is to not have to support build_sdist from within another sdist, and if he depends on a pip extension (rather than it being an optional speed up like prepare_wheel_metadata) then we either break flit users if we decide it wasn?t worth it and we remove it, or we force other frontends to understand ext_pip_prepare_build_files and we get into the same kind of ?everyone is just trying to emulate setuptools? bog that we?re trying to escape. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jun 16 22:27:27 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Jun 2017 12:27:27 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1497650711.3196641.1012092480.05BD541D@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <1497650711.3196641.1012092480.05BD541D@webmail.messagingengine.com> Message-ID: On 17 June 2017 at 08:05, Thomas Kluyver wrote: > On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote: >> The messy complications come from >> prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't >> surprising, since those are the two hooks where we're squinting into >> our crystal ball to try and guess what will be useful for software >> that doesn't exist yet, but will later, maybe, we hope. > > I'm not exactly clear on what use cases the prepare_wheel_metadata hook > satisfies - that was in the spec before I was particularly involved in > it. As Donald notes, this is primarily a workaround for the underspecification of the sdist format - wheel metadata is the most trustworthy information source we have for post-installation project metadata (name, version, runtime dependencies, etc), since it's exactly what will end up in the post-install dist-info directory (minus the files generated by an installer at installation time). Pure-Python backends like flit can probably skip providing it, since building a wheel is a cheap operation, but something like enscons is probably going to want to implement it (since only generating the Python wheel metadata should be much cheaper than running the full build). And unlike defining a new iteration of the sdist format, encouraging backend developers to make wheel metadata generation a cheap operation doesn't require any new speculative format definition work on our part. > I do think we've hashed out a concrete need for prepare_build_files (or > whatever it ends up called): to copy the files to a build directory > without either copying large amounts of unnecessary data or figuring out > what belongs in an sdist. The only alternative would be to require > backends to avoid leaving clutter in the working directory, so they > would take care of any necessary copy step rather than exposing a > separate hook. People insist that trusting backends is a non-starter, > though. It isn't that trusting them is a non-starter in all situations, it's that *requiring* front-ends to trust them is a non-starter. Instead we want front-ends to have the following expectation: 1. either a backend's build_sdist is always fast, and doesn't require any dependencies not already needed to build a wheel; 2. or else, if build_sdist isn't always fast, or requires extra dependencies, then prepare_wheel_input_files will be provided "Dependencies" there refers to both actual Python-level dependencies expressible in terms of PEP 508, as well as external dependencies that currently can't be represented (e.g. requiring particular command line tools to be available). Cheers, Nick. P.S. This also seems like an opportune time to remind folks that Tennessee Leeuwenburg and Robert Collins started a draft PEP for expressing external dependencies a couple of years ago: https://github.com/pypa/interoperability-peps/pull/30/files While any further work along those lines would need significant updates to account for PEP 508, PEP 518, PEP 517, and other developments, the core concept of using "" so that external dependencies can potentially be passed anywhere that PEP 508 dependencies are currently supported remains sound, and would likely provide a solid foundation for a plugin based model where (for example), there might be common "bin", "clib", and "cheader" types at the dependency declaration level (but platform dependent implementations of those plugins), as well as cross-platform namespaces for 3rd party dependency management ecosystems (e.g. npm, cargo, maven) (which would then either have appropriate build system requirements like "bin!!npm", "bin!!cargo" and "bin!!maven", dependencies on Python wrappers for those tools, or else dependencies on suitable extras in the underlying build backend). The one place where we likely *wouldn't* mix internal and external requirements is in interoperability specifications: there, we'd keep the external requirements separate, so that older tools don't get confused trying to parse them, and instead just treat satisfying them as somebody else's problem (i.e. preserving the status quo). -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jun 16 22:48:35 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Jun 2017 12:48:35 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> Message-ID: On 17 June 2017 at 07:48, Nathaniel Smith wrote: > Hmm, here's another plea for simplicity, but from a slightly different > direction that I just thought of: what if we said that any hooks > starting with "ext_pip_..." are reserved for pip's use, and pip can > make up whatever semantics it likes for them. And then as the parts of > pip that actually want to use prepare_wheel_metadata and/or > get_prepared_wheel_input_files come online, we use the ext_pip_* > versions of those hooks to prototype them and work out any issues. And > then once there's an actual implementation and proven value, we write > a new PEP to drop the ext_pip_ prefix and make them standard. > > What do you think? pip's needs here aren't unique to pip - they're just "system integrator" needs, rather than "software developer" needs. It's fairly common for system integrator needs to seem like pointless complexity to folks more focused on addressing the "single component publisher (aka software developer)" case, but that's part of why we have the PEP process: to encourage us to figure out the best place for complexity to live in order to advance the overall Python ecosystem, rather than trying to convince ourselves that the underlying complexity doesn't exist. The funding dynamics of open source *do* play into those discussions as a design consideration, since any work done by volunteers needs to offer some kind of inherent benefit to those volunteers (or they won't have any incentive to do the work), while work that we expect to be done by professionals on work time requires some form of justification as being in their customer's interests (e.g. "more software available more reliably in the formats that you prefer"). Cheers, Nick. P.S. This "Software distribution, even over the internet, is a more complex problem than it may first appear" is also why PEP 426 opens with an attempted overview of the full complexity of the "original software component publisher to downstream application, service, or appliance user", and why that's also one of the points I dwelled on in [1]. Large scale systems engineering is a distinct engineering discipline for a reason, and it mostly boils down to making smart decisions about how you divide responsibilities between components. [1] http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Fri Jun 16 23:08:31 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 20:08:31 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> Message-ID: On Fri, Jun 16, 2017 at 5:52 PM, Donald Stufft wrote: > I?d probably want to spec this out as ext_{name-on-pypi}_* to remove a > special case on pip in the PEP, to let others do experimentation as well. Sure, like the [tool.*] escape hatch in pyproject.toml. > However this solution is fine with me on both of the Non build_{sdist,wheel} > hooks. Although I feel like it kind of defeats the purpose of the > prepare_build_files hooks, because I think Thomas? goal is to not have to > support build_sdist from within another sdist, and if he depends on a pip > extension (rather than it being an optional speed up like > prepare_wheel_metadata) then we either break flit users if we decide it > wasn?t worth it and we remove it, or we force other frontends to understand > ext_pip_prepare_build_files and we get into the same kind of ?everyone is > just trying to emulate setuptools? bog that we?re trying to escape. How I'm imagining it would work is: Right now: pip just unconditionally uses shutil.copytree, and doesn't even try to generate an sdist. Eventually: someone writes the patch to pip to attempt to do the copy step via generating an sdist. This is going to require some fallback strategy for when building an sdist fails (because the legacy setup.py command errors out, because the build_sdist hook is missing, because the build_sdist hook errors out...); probably falling back on shutil.copytree. For flit's purposes, this doesn't seem like a disaster: flit projects don't have tens of megabytes of random build artifacts lying around, and flit sdists in particular don't have tens of megabytes of irrelevant VCS history. In fact, an unpacked flit sdist is generally just as cheap to copy via shutil.copytree as any other project would be to copy via build_sdist. Eventually^2 (possibly part of the previous step): someone comes up with a prototype ext_pip_prepare_build_files spec to optimize this further, and it gets implemented in pip and flit. Experience shows that the speedup is worthwhile, enough so that other build frontends want to get in on the action. Eventually^3: we write up a PEP that's just a copy/paste of the mature pip behavior and tell other frontends to go for it. But of course there are also other ways this could go. Like, I'm not saying it's likely, but for all we know, between now and when pip gets around to implementing the copy-via-sdist approach, Thomas might have some brainstorm and realize that flit needs to implement build-sdist-from-sdist for some other reason. Making predictions is hard, especially about the future. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Fri Jun 16 23:15:31 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Jun 2017 20:15:31 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> Message-ID: On Fri, Jun 16, 2017 at 7:48 PM, Nick Coghlan wrote: > On 17 June 2017 at 07:48, Nathaniel Smith wrote: >> Hmm, here's another plea for simplicity, but from a slightly different >> direction that I just thought of: what if we said that any hooks >> starting with "ext_pip_..." are reserved for pip's use, and pip can >> make up whatever semantics it likes for them. And then as the parts of >> pip that actually want to use prepare_wheel_metadata and/or >> get_prepared_wheel_input_files come online, we use the ext_pip_* >> versions of those hooks to prototype them and work out any issues. And >> then once there's an actual implementation and proven value, we write >> a new PEP to drop the ext_pip_ prefix and make them standard. >> >> What do you think? > > pip's needs here aren't unique to pip - they're just "system > integrator" needs, rather than "software developer" needs. > > It's fairly common for system integrator needs to seem like pointless > complexity to folks more focused on addressing the "single component > publisher (aka software developer)" case, but that's part of why we > have the PEP process: to encourage us to figure out the best place for > complexity to live in order to advance the overall Python ecosystem, > rather than trying to convince ourselves that the underlying > complexity doesn't exist. I'm not making an argument that the complexity doesn't exist or that pip is the only project that matters; I'm arguing that it's risky to standardize things before we've really hit the problem, because we might not fully understand it yet. Which is a truism :-). But sometimes you don't have much choice. The real suggestion is hey, maybe in this case there's a way to get these important-but-not-fully-understood issues off the critical path for PEP 517 without losing our chance to handle them later. -n -- Nathaniel J. Smith -- https://vorpus.org From donald at stufft.io Sat Jun 17 02:28:51 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 17 Jun 2017 02:28:51 -0400 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> Message-ID: <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> > > On Jun 16, 2017, at 11:08 PM, Nathaniel Smith wrote: > > (because the legacy setup.py > command errors out, because the build_sdist hook is missing, because > the build_sdist hook errors out...); probably falling back on > shutil.copytree. Ah okay. This is where the disconnect was. I assumed if a project was using a PEP 517 build tool the result of a build_sdist failure (or a build_wheel failure) would be surfacing an error to the end user, not some sort of fallback. From njs at pobox.com Sat Jun 17 03:07:23 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 17 Jun 2017 00:07:23 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Fri, Jun 16, 2017 at 11:28 PM, Donald Stufft wrote: >> >> On Jun 16, 2017, at 11:08 PM, Nathaniel Smith wrote: >> >> (because the legacy setup.py >> command errors out, because the build_sdist hook is missing, because >> the build_sdist hook errors out...); probably falling back on >> shutil.copytree. > > > Ah okay. This is where the disconnect was. I assumed if a project was using a PEP 517 build tool the result of a build_sdist failure (or a build_wheel failure) would be surfacing an error to the end user, not some sort of fallback. What we might be able to get away with in the PEP 517 transition is to provide an explicit way of signalling "build_sdist failed because that's just not a supported operation", so that pip could distinguish that from "something unexpected blew up" and only fall back in the first case. Obviously we want build_sdist to be supported in as many places as possible, but given that we anticipate that there will be times when it won't be (flit, or ad hoc build systems that just never implement it) then I think having a standard way to communicate that is a good idea regardless. It's surprisingly tricky to come up with a good signal for this though. We don't want to use a user-defined exception, because we don't have like a "standard PEP 517 library" to put it in. We could use a built-in exception like NotImplementedError, but then there's the risk that that gets raised for some other reason internally, it leaks out, and then we misinterpret it as being an intentional signal. Proposal: "NotImplemented" is a legal return value from build_sdist (similar to dunder methods), and should trigger whatever fallback behavior the frontend would do if the hook was simply undefined. -n -- Nathaniel J. Smith -- https://vorpus.org From p.f.moore at gmail.com Sat Jun 17 03:36:47 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 17 Jun 2017 08:36:47 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 17 June 2017 at 08:07, Nathaniel Smith wrote: > Proposal: "NotImplemented" is a legal return value from build_sdist > (similar to dunder methods), and should trigger whatever fallback > behavior the frontend would do if the hook was simply undefined. The PEP currently allows build_sdist to fail, but frontends don't have much option do do anything other than present that failure direct to the user. Not because there isn't a special "fall back to something else" return value, but because the PEP provides no guarantee that any fallback is possible. We've been very careful to *not* require anything of source trees than what the hooks offer, as I understand it, so it seems like we're doing a bit of a U-turn now if we say "if the hook fails, front ends will fall back" and don't provide any information as to what front ends are allowed to assume as part of that fallback. You mention falling back to copying, but the PEP doesn't even specify that source trees must be copyable. I'd probably have to work quite hard to come up with a scenario where they aren't, but if we want to make that guarantee, let's just be explicit about it. A single sentence "backend operations on source trees must behave the same if the source tree is copied to a different filesystem location, and front ends are free to copy source trees if needed" is easy to add, if that's what we want to guarantee. Personally, I prefer not adding constraints on source trees, and leaving the PEP as it is, with no fallback required or expected if a hook fails. Paul From ncoghlan at gmail.com Sat Jun 17 10:28:48 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Jun 2017 00:28:48 +1000 Subject: [Distutils] PEP 517: Python 3 finder? In-Reply-To: <2DAAA4B5-453F-4E89-8B72-E66A81B68201@stufft.io> References: <2DAAA4B5-453F-4E89-8B72-E66A81B68201@stufft.io> Message-ID: On 17 June 2017 at 10:25, Donald Stufft wrote: > > On Jun 16, 2017, at 5:01 PM, Nathaniel Smith wrote: > > If someone wants to experiment with this, then it's possible within the PEP > 517 framework to write a 2.7-compatible backend that searches the system for > a python 3 install and then uses it. I'm not sure it's a good idea, but you > can do it :-). > > > > I just want to hammer this specific point home a bit more. The only real > requirement on Python version PEP 517 puts into place is that you write the > API that the frontend will call in the version of Python the frontend is > running under. What you do beyond that is entirely up to you, including > running a different version of Python or something that?s not written in > Python at all. > > The only ?downside" to this really is that your options if that build system > isn?t available is to have some sort of fallback (in which case why have the > other thing at all?) or just error out. However I don?t think this is a > solvable downside, because in either case if you?re running on py3 and the > build backend needs py2, there?s nothing really for a frontend to do here > except bail out. Right, from PEP 517's point of view, running an external build backend like scons/waf/meson under Python 3 to produce Python 2.7 wheels (or even vice-versa) is the same as running any other external binary as part of the build process. The *downside* of doing it that way is that it forces wrappers around those tools (like enscons) to choose between the following two options for 2.7 support: - retain 2.7 compatibility in the backend tool, declare it as a dependency in build-system.requires - run the backend as an external application, losing the ability to automatically install it However, even though I acknowledge that as an unfortunate limitation, I think it would still be better to tackle doing something about it as a separate proposal that improves support for external dependencies. It just highlights that if/when we move forward with that idea, we should have a "pycli" category of external dependencies that specifically handles scripts that can be installed via a tool like `pipsi` based on a PyPI dependency specification (as opposed to arbitrary applications handled by the system package manager). The semantics of a "pycli" external dependency would then be "Install into a dedicated virtual environment using the latest installed Python", with a separate "py2cli" for tools that are currently still Python-2 only. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 17 10:41:35 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Jun 2017 00:41:35 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 17 June 2017 at 17:07, Nathaniel Smith wrote: > On Fri, Jun 16, 2017 at 11:28 PM, Donald Stufft wrote: >>> >>> On Jun 16, 2017, at 11:08 PM, Nathaniel Smith wrote: >>> >>> (because the legacy setup.py >>> command errors out, because the build_sdist hook is missing, because >>> the build_sdist hook errors out...); probably falling back on >>> shutil.copytree. >> >> >> Ah okay. This is where the disconnect was. I assumed if a project was using a PEP 517 build tool the result of a build_sdist failure (or a build_wheel failure) would be surfacing an error to the end user, not some sort of fallback. > > What we might be able to get away with in the PEP 517 transition is to > provide an explicit way of signalling "build_sdist failed because > that's just not a supported operation", so that pip could distinguish > that from "something unexpected blew up" and only fall back in the > first case. That's what `prepare_wheel_input_files` gives us: if the backend doesn't implement it, then `build_sdist` is expected to always succeed, and any errors should be reported to the end user. OTOH, if `build_sdist` can fail (e.g. due to missing dependencies that aren't needed just to build a wheel), then the backend should offer a `prepare_wheel_input_files` that offers the expected guarantee (i.e. it will only fail for reasons that should be reported to the user attempting to do the build). The frontend then *won't* have a fallback copy strategy for PEP 517 backends - if `prepare_wheel_input_files` is present and fails, or if it's missing and `build_sdist` fails, then you have a failed wheel build on your hands. The legacy copying strategy would then only be used when falling all the way back to the `setup.py` interface, not as an attempt to continue the build in the face of a backend bug. This all gets murkier if you try to make the build_sdist hook more complex instead of just having two different hooks for frontends to call that tell the backend what the frontend actually wants (i.e. either an actual sdist, or just a clean set of exported files to use as input to a wheel build). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Sat Jun 17 11:41:17 2017 From: brett at python.org (Brett Cannon) Date: Sat, 17 Jun 2017 15:41:17 +0000 Subject: [Distutils] PEP 517: editable installs In-Reply-To: References: Message-ID: https://github.com/python/pythondotorg for website stuff. (and the most recent version should be at https://www.python.org/dev/peps/pep-0517/). On Fri, 16 Jun 2017 at 14:06 Nathaniel Smith wrote: > Oh, yeah, that's totally an ancient version of the PEP. That sucks. I > thought the legacy.python.org URLs were redirecting to python.org now. > Probably we should file a bug with the python.org infra team, but I'm not > sure where their tracker is. > > On Jun 16, 2017 1:48 PM, "Daniel Holth" wrote: > >> Probably this older version from >> http://legacy.python.org/dev/peps/pep-0517/ >> >> On Fri, Jun 16, 2017 at 4:42 PM Nathaniel Smith wrote: >> >>> On Jun 16, 2017 11:59 AM, "Daniel Holth" wrote: >>> >>> I noticed PEP 517 uses a prefix for editable installs. >>> >>> >>> Which part of the pep are you looking at? All I see about editable >>> installs is the note saying that they're deferred to a future pep. >>> >>> -n >>> >> _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Jun 17 16:15:36 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 17 Jun 2017 13:15:36 -0700 Subject: [Distutils] PEP 517: editable installs In-Reply-To: References: Message-ID: https://github.com/python/pythondotorg/issues/1090 On Sat, Jun 17, 2017 at 8:41 AM, Brett Cannon wrote: > https://github.com/python/pythondotorg for website stuff. (and the most > recent version should be at https://www.python.org/dev/peps/pep-0517/). > > On Fri, 16 Jun 2017 at 14:06 Nathaniel Smith wrote: >> >> Oh, yeah, that's totally an ancient version of the PEP. That sucks. I >> thought the legacy.python.org URLs were redirecting to python.org now. >> Probably we should file a bug with the python.org infra team, but I'm not >> sure where their tracker is. >> >> On Jun 16, 2017 1:48 PM, "Daniel Holth" wrote: >>> >>> Probably this older version from >>> http://legacy.python.org/dev/peps/pep-0517/ >>> >>> On Fri, Jun 16, 2017 at 4:42 PM Nathaniel Smith wrote: >>>> >>>> On Jun 16, 2017 11:59 AM, "Daniel Holth" wrote: >>>> >>>> I noticed PEP 517 uses a prefix for editable installs. >>>> >>>> >>>> Which part of the pep are you looking at? All I see about editable >>>> installs is the note saying that they're deferred to a future pep. >>>> >>>> -n >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig -- Nathaniel J. Smith -- https://vorpus.org From donald at stufft.io Thu Jun 22 12:32:52 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 22 Jun 2017 12:32:52 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI Message-ID: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> Hello All! As most people are aware, there has been an effort under way to rewrite PyPI in order to solve a lot of long standing problems. For those who aren't aware, that is currently available at https://pypi.org/ and it uses the same database that "Legacy" PyPI does, so the two are essentially just different views over the same data. For a awhile now, Python, setuptools, and twine have all defaulted to using this new code base for uploading artifacts to PyPI. Now that we've gotten some testing of that code base, the infrastructure team and myself feel comfortable directing everyone to using the new endpoint and we're planning on shutting down uploads to Legacy PyPI. If you're using the latest versions of Python, setuptools, or twine and you have not put an explicit URL in your ~/.pypirc file, then there's nothing you should need to do. If you are not, then you should ideally upgrade to the latest version of whatever tool you're using to upload (the preferred tool is twine) and edit your ~/.pypirc so that it removes any explicit mention of an URL. Thus it should look something like: [distutils] index-servers = pypi [pypi] username:yourusername password:yourpassword If for some reason you're not able to update to the newest version of your upload tool, then you can configure it to upload to the new code base by switching the URL to use https://upload.pypi.org/legacy/ instead of https://pypi.python.org/pypi. Thus your ~/.pypirc would then become: [distutils] index-servers = pypi [pypi] repository:https://upload.pypi.org/legacy/ username:yourusername password:yourpassword For those of you who are using TestPyPI, that will also be affected, and the required URL for the new upload endpoint for TestPyPI is https://test.pypi.org/legacy/. We plan to disable uploads to legacy PyPI on July 3rd, 2017 so any configuration change will need to be made before that date. In addition, we plan to have a "brownout" on June 29th where we will shut the legacy endpoint down for that day. For TestPyPI the change to disable uploads to legacy will be made in the next couple of days, likely this weekend. As part of the error message that users will get when attempting to upload to legacy PyPI, we will include a link to a page that details how to ensure that they are using the new code base and not the legacy code base. Thanks everyone! ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From lele at metapensiero.it Thu Jun 22 13:00:36 2017 From: lele at metapensiero.it (Lele Gaifax) Date: Thu, 22 Jun 2017 19:00:36 +0200 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> Message-ID: <87zid0hs97.fsf@metapensiero.it> Donald Stufft writes: > As most people are aware, there has been an effort under way to rewrite PyPI in > order to solve a lot of long standing problems. For those who aren't aware, that > is currently available at https://pypi.org/ and it uses the same database that > "Legacy" PyPI does, so the two are essentially just different views over the > same data. Great, thank you all for the effort! As I'm used to keep an eye from time to time on https://pypi.python.org/pypi, just to spot relevant upgrades, or see something new that catches my attention, I wonder if the new interface offers a similar view, or some kind of RSS feed, with more than the five "New Releases" box. Should I change my habits? Thanks again, ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From donald at stufft.io Thu Jun 22 13:37:43 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 22 Jun 2017 13:37:43 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <87zid0hs97.fsf@metapensiero.it> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <87zid0hs97.fsf@metapensiero.it> Message-ID: > On Jun 22, 2017, at 1:00 PM, Lele Gaifax wrote: > > Donald Stufft writes: > >> As most people are aware, there has been an effort under way to rewrite PyPI in >> order to solve a lot of long standing problems. For those who aren't aware, that >> is currently available at https://pypi.org/ and it uses the same database that >> "Legacy" PyPI does, so the two are essentially just different views over the >> same data. > > Great, thank you all for the effort! > > As I'm used to keep an eye from time to time on https://pypi.python.org/pypi, > just to spot relevant upgrades, or see something new that catches my > attention, I wonder if the new interface offers a similar view, or some kind > of RSS feed, with more than the five "New Releases" box. > Can you open an issue on https://github.com/pypa/warehouse? I think we could probably add something along this lines. At a minimum an RSS feed should be easy (which I think we already have, but it may not be a great one) but I?m thinking maybe the search interface could allow ordering by last update too, then you can search for nothing and get the most recently updated things across the entire site, but it would also let you search for say ?Django? and get the most recently updated things that have to do with Django. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 22 13:38:25 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 22 Jun 2017 13:38:25 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <87zid0hs97.fsf@metapensiero.it> Message-ID: <41197D43-E504-40F8-8FDA-40E150C23B5B@stufft.io> > On Jun 22, 2017, at 1:37 PM, Donald Stufft wrote: > >> >> On Jun 22, 2017, at 1:00 PM, Lele Gaifax > wrote: >> >> Donald Stufft > writes: >> >>> As most people are aware, there has been an effort under way to rewrite PyPI in >>> order to solve a lot of long standing problems. For those who aren't aware, that >>> is currently available at https://pypi.org/ and it uses the same database that >>> "Legacy" PyPI does, so the two are essentially just different views over the >>> same data. >> >> Great, thank you all for the effort! >> >> As I'm used to keep an eye from time to time on https://pypi.python.org/pypi , >> just to spot relevant upgrades, or see something new that catches my >> attention, I wonder if the new interface offers a similar view, or some kind >> of RSS feed, with more than the five "New Releases" box. >> > > > Can you open an issue on https://github.com/pypa/warehouse? I think we could probably add something along this lines. At a minimum an RSS feed should be easy (which I think we already have, but it may not be a great one) but I?m thinking maybe the search interface could allow ordering by last update too, then you can search for nothing and get the most recently updated things across the entire site, but it would also let you search for say ?Django? and get the most recently updated things that have to do with Django. > Oh never mind, this already works: https://pypi.org/search/?q=&o=-created ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From lele at metapensiero.it Thu Jun 22 14:10:37 2017 From: lele at metapensiero.it (Lele Gaifax) Date: Thu, 22 Jun 2017 20:10:37 +0200 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <87zid0hs97.fsf@metapensiero.it> <41197D43-E504-40F8-8FDA-40E150C23B5B@stufft.io> Message-ID: <87vannj3ky.fsf@metapensiero.it> Donald Stufft writes: > Oh never mind, this already works: https://pypi.org/search/?q=&o=-created > Thank you! Even if a bit more verbose than the old interface, being able to look back several days is a bonus, I will relax my compulsive visits :-) ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From donald at stufft.io Thu Jun 22 14:14:08 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 22 Jun 2017 14:14:08 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <87vannj3ky.fsf@metapensiero.it> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <87zid0hs97.fsf@metapensiero.it> <41197D43-E504-40F8-8FDA-40E150C23B5B@stufft.io> <87vannj3ky.fsf@metapensiero.it> Message-ID: <08C7130E-CD0C-44F7-8BC7-8CB5D1BFF07E@stufft.io> > On Jun 22, 2017, at 2:10 PM, Lele Gaifax wrote: > > Donald Stufft writes: > >> Oh never mind, this already works: https://pypi.org/search/?q=&o=-created >> > > Thank you! Even if a bit more verbose than the old interface, being able > to look back several days is a bonus, I will relax my compulsive visits :-) > Yea! The only caveat is right now the search index isn?t updated on demand (it?s on my TODO list) so there is some lag time. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Jun 23 06:44:21 2017 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 23 Jun 2017 20:44:21 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> Message-ID: <85mv8zt24a.fsf@benfinney.id.au> Donald Stufft writes: > [?] rewrite PyPI in order to solve a lot of long standing problems. > For those who aren't aware, that is currently available at > https://pypi.org/ Thank you to everyone involved in bringing this to fruition! > For a awhile now, Python, setuptools, and twine have all defaulted to > using this new code base for uploading artifacts to PyPI. Where can we find the exact versions of each that default to ?pypi.org? for uploads? > [?pypi.org?] uses the same database that "Legacy" PyPI does, so the > two are essentially just different views over the same data. At what point does the big red warning banner (? This is a pre-production deployment of Warehouse.?) come down? What is the cross-over period: ?pypi.org? as the primary recommendation, while ?pypi.python.org? continues to work? -- \ ?You can never entirely stop being what you once were. That's | `\ why it's important to be the right person today, and not put it | _o__) off until tomorrow.? ?Larry Wall | Ben Finney From ncoghlan at gmail.com Fri Jun 23 07:36:00 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Jun 2017 21:36:00 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <85mv8zt24a.fsf@benfinney.id.au> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> Message-ID: On 23 June 2017 at 20:44, Ben Finney wrote: > Donald Stufft writes: >> [?pypi.org?] uses the same database that "Legacy" PyPI does, so the >> two are essentially just different views over the same data. > > At what point does the big red warning banner (? This is a > pre-production deployment of Warehouse.?) come down? For non-logged-in use, pretty much anytime, but a lot of the logged-in publisher pages aren't complete enough to allow a full migration yet. > What is the cross-over period: ?pypi.org? as the primary recommendation, > while ?pypi.python.org? continues to work? Full migration = pypi.python.org becomes a CNAME for pypi.org. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben+python at benfinney.id.au Fri Jun 23 07:39:09 2017 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 23 Jun 2017 21:39:09 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> Message-ID: <85fuerszky.fsf@benfinney.id.au> Nick Coghlan writes: > On 23 June 2017 at 20:44, Ben Finney wrote: > > What is the cross-over period: ?pypi.org? as the primary > > recommendation, while ?pypi.python.org? continues to work? > > Full migration = pypi.python.org becomes a CNAME for pypi.org. Thank you, that is the state at the *end* of the cross-over period. But that's not what I asked. What is the cross-over period, during which we have ?pypi.org? as the primary recommendation, while ?pypi.python.org? continues to work as it currently does? -- \ ?If you always want the latest and greatest, then you have to | `\ buy a new iPod at least once a year.? ?Steve Jobs, MSNBC | _o__) interview 2006-05-25 | Ben Finney From donald at stufft.io Fri Jun 23 10:21:08 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 23 Jun 2017 10:21:08 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <85mv8zt24a.fsf@benfinney.id.au> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> Message-ID: <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> > On Jun 23, 2017, at 6:44 AM, Ben Finney wrote: > > Donald Stufft writes: > >> [?] rewrite PyPI in order to solve a lot of long standing problems. >> For those who aren't aware, that is currently available at >> https://pypi.org/ > > Thank you to everyone involved in bringing this to fruition! > >> For a awhile now, Python, setuptools, and twine have all defaulted to >> using this new code base for uploading artifacts to PyPI. > > Where can we find the exact versions of each that default to ?pypi.org? > for uploads? Umm. Twine 1.8.0, Python 3.4.6, 3.5.3, 3.6.0, Not sure on 2.7, the Github interface seems to be broken and not listing the tags for that one. Essentially it was committed 11 months ago and whatever releases came after that. > >> [?pypi.org?] uses the same database that "Legacy" PyPI does, so the >> two are essentially just different views over the same data. > > At what point does the big red warning banner (? This is a > pre-production deployment of Warehouse.?) come down? Roughly whenever we have Warehouse hosted in a place that can handle that 30-40tb/month that Rackspace says our origin servers receive that would come Warehouse?s way at that point. PyPI is massively read heavy to an absurd degree, so the traffic from uploads is relatively small compared to the traffic from downloads, and currently Warehouse is not hosted in a way to handle our read traffic. The other thing the banner accomplishes is to provide warning that you may see weirdness in the UI (for example, hardcoded demo data etc) and to indicate that this is normal and please don?t file bugs for it. > > What is the cross-over period: ?pypi.org? as the primary recommendation, > while ?pypi.python.org? continues to work? I?m not sure what specifically you?re asking here? Are you looking for how long they?re both going to remain active or the process we?re going to migrate things over with? Or something else? ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Jun 23 10:43:35 2017 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 24 Jun 2017 00:43:35 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> Message-ID: <858tkiu5m0.fsf@benfinney.id.au> Donald Stufft writes: > Umm. Twine 1.8.0, Python 3.4.6, 3.5.3, 3.6.0, Not sure on 2.7, the > Github interface seems to be broken and not listing the tags for that > one. Thanks. What version of Setuptools is needed? I'm hoping that the above list will be prominently published with the documentation that we point people toward for announcing we should be using those tools. (?Use the latest version of the tool? doesn't cut it; we'll need to check what versions our operating system provides, and know whether it's in or out of compliance with the above list.) > > What is the cross-over period: ?pypi.org? as the primary > > recommendation, while ?pypi.python.org? continues to work? > > I?m not sure what specifically you?re asking here? Are you looking for > how long they?re both going to remain active or the process we?re > going to migrate things over with? Or something else? I'm working on the assumption the sequence will include: * Now: Uploading to ?pypi.python.org? is officially recommended; it goes to the same data store as uploading to ?pypi.org?. Both work fine. * Crossover: Uploading to ?pypi.org? is officially recommended; it goes to the same data store as uploading to ?pypi.python.org?. Both work fine. * Warehouse Rules Alone: Uploading to ?pypi.org? is recommended, because uploading to ?pypi.python.org? doesn't work. So, what is the period ? when does it start and when does it end ? that the Crossover state will active? -- \ ?? correct code is great, code that crashes could use | `\ improvement, but incorrect code that doesn?t crash is a | _o__) horrible nightmare.? ?Chris Smith, 2008-08-22 | Ben Finney From mmericke at gmail.com Fri Jun 23 11:11:21 2017 From: mmericke at gmail.com (Michael Merickel) Date: Fri, 23 Jun 2017 10:11:21 -0500 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <858tkiu5m0.fsf@benfinney.id.au> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> Message-ID: On Fri, Jun 23, 2017 at 9:43 AM, Ben Finney wrote: > So, what is the period ? when does it start and when does it end ? that > the Crossover state will active? > The time of officially recommending "pypi.org" was Sept 2017 when setuptools v27 was released which switched the default. https://setuptools.readthedocs.io/en/latest/history.html#v27-0-0 I think Donald is saying that period of them both working will be over on July 3rd at which point only pypi.org will work. As far as which UI is recommended to manage your packages and for end-users to view things - I'm not sure about that timeline. For viewing packages I've been using pypi.org for a while now with minimal issues but it still has no management features in the UI. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 23 12:30:12 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 23 Jun 2017 12:30:12 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> Message-ID: <1DA43C4C-6642-452A-9692-F93CF424275D@stufft.io> > On Jun 23, 2017, at 11:11 AM, Michael Merickel wrote: > > On Fri, Jun 23, 2017 at 9:43 AM, Ben Finney > wrote: > So, what is the period ? when does it start and when does it end ? that > the Crossover state will active? > > The time of officially recommending "pypi.org " was Sept 2017 when setuptools v27 was released which switched the default. > > https://setuptools.readthedocs.io/en/latest/history.html#v27-0-0 > > I think Donald is saying that period of them both working will be over on July 3rd at which point only pypi.org will work. > > As far as which UI is recommended to manage your packages and for end-users to view things - I'm not sure about that timeline. For viewing packages I've been using pypi.org for a while now with minimal issues but it still has no management features in the UI. > Basically this yea. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 24 01:24:44 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jun 2017 15:24:44 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> Message-ID: On 24 June 2017 at 01:11, Michael Merickel wrote: > As far as which UI is recommended to manage your packages and for end-users > to view things - I'm not sure about that timeline. For viewing packages I've > been using pypi.org for a while now with minimal issues but it still has no > management features in the UI. Thanks for that. I better understand Ben's question now, and it made me realise that https://packaging.python.org/guides/tool-recommendations/ is missing a section on "Publishing platform" that explains the migration that is currently in progress and covers what pypi.org currently handles better than pypi.python.org (publishing), which should still use pypi.python.org (downloading due to server bandwidth availability, package management UI), and those where it doesn't matter much which you use (browsing). I've created a PR to add that here: https://github.com/pypa/python-packaging-user-guide/pull/334 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 24 03:45:27 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jun 2017 17:45:27 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> Message-ID: On 24 June 2017 at 15:24, Nick Coghlan wrote: > https://packaging.python.org/guides/tool-recommendations/ is missing a > section on "Publishing platform" that explains the migration that is > currently in progress and covers what pypi.org currently handles > better than pypi.python.org (publishing), which should still use > pypi.python.org (downloading due to server bandwidth availability, > package management UI), and those where it doesn't matter much which > you use (browsing). The new "Publishing Platform Migration" section is now live: https://packaging.python.org/guides/tool-recommendations/#publishing-platform-migration Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Sat Jun 24 04:56:21 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 24 Jun 2017 09:56:21 +0100 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> Message-ID: <1498294581.2774227.1019825472.64435BE5@webmail.messagingengine.com> For reference, I switched the default upload server for flit in version 0.11. On Sat, Jun 24, 2017, at 08:45 AM, Nick Coghlan wrote: > On 24 June 2017 at 15:24, Nick Coghlan wrote: > > https://packaging.python.org/guides/tool-recommendations/ is missing a > > section on "Publishing platform" that explains the migration that is > > currently in progress and covers what pypi.org currently handles > > better than pypi.python.org (publishing), which should still use > > pypi.python.org (downloading due to server bandwidth availability, > > package management UI), and those where it doesn't matter much which > > you use (browsing). > > The new "Publishing Platform Migration" section is now live: > https://packaging.python.org/guides/tool-recommendations/#publishing-platform-migration > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Sat Jun 24 09:15:26 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Jun 2017 23:15:26 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <1498294581.2774227.1019825472.64435BE5@webmail.messagingengine.com> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> <1498294581.2774227.1019825472.64435BE5@webmail.messagingengine.com> Message-ID: On 24 June 2017 at 18:56, Thomas Kluyver wrote: > For reference, I switched the default upload server for flit in version > 0.11. Oh, I'd also missed that 0.11 published sdists now. Given that, I think we should start recommending flit for new pure Python projects (the lack of sdists was the only thing previously making me hesitant about that), and then add it to the PyPI upload migration list as well. It's also missing an entry in the Key Projects list: https://packaging.python.org/key_projects/ I've filed an issue to discuss that here: https://github.com/pypa/python-packaging-user-guide/issues/336 We should probably add enscons to the Key Projects list as well, since that allows folks the option of using a full pip-installable C/C++ build system as an alternative to setuptools/distutils, so I also filed an issue to discuss that: https://github.com/pypa/python-packaging-user-guide/issues/337 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Sat Jun 24 09:44:03 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 24 Jun 2017 14:44:03 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: <1498311843.2844465.1019960624.421AFA26@webmail.messagingengine.com> I have prepared a PR against the PEP adding get_build_sdist_requires , and renaming a couple of the other hooks for clarity (get_build_wheel_requires, prepare_build_wheel_files): https://github.com/python/peps/pull/297 From priya.venky2001 at gmail.com Sat Jun 24 00:02:23 2017 From: priya.venky2001 at gmail.com (Priya Lakshmi) Date: Sat, 24 Jun 2017 09:32:23 +0530 Subject: [Distutils] Can u provide metadata about the PyPI Message-ID: Respected Python expert I just want a help. Can u provide us the metadata available about the modules in PyPI giant database. For example, we want the number of packages available, the number of downloads for each package, etc.... So we can think of doing some good maintenance work. thank you. G Priyalakshmi Assistant Professor (Selection Grade) Department of Applied Mathematics and Computational Sciences PSG College of Technology Coimbatore -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Sat Jun 24 10:57:02 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 24 Jun 2017 15:57:02 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1498311843.2844465.1019960624.421AFA26@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498311843.2844465.1019960624.421AFA26@webmail.messagingengine.com> Message-ID: <1498316222.3887873.1019999072.063E60AA@webmail.messagingengine.com> Nick has merged that PR, and the updated PEP is visible here: https://www.python.org/dev/peps/pep-0517/ Hopefully we're nearing a consensus on this now. If you're interested, please do have a read through the latest version. Thomas On Sat, Jun 24, 2017, at 02:44 PM, Thomas Kluyver wrote: > I have prepared a PR against the PEP adding get_build_sdist_requires , > and renaming a couple of the other hooks for clarity > (get_build_wheel_requires, prepare_build_wheel_files): > > https://github.com/python/peps/pull/297 > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From randy at thesyrings.us Sat Jun 24 11:18:06 2017 From: randy at thesyrings.us (Randy Syring) Date: Sat, 24 Jun 2017 11:18:06 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> <1498294581.2774227.1019825472.64435BE5@webmail.messagingengine.com> Message-ID: <154f5a8f-bd1c-52b4-c18c-1e39847d6a9d@thesyrings.us> For what it's worth, for the projects that are almost ready to recommend, but not quite there (like Flit per the discussion on issue 336), would it hurt to have an "provisional" area to the key projects list that makes people aware of the projects without "blessing" them as Key Projects. The whole point of that section of the guide is to raise awareness, yes? Well if something like Flit is almost ready, but pending for a small reason or two, why not publish it anyway with some kind of "provision" label or something and an explanation of why the project is important, but not quite yet ready? "Provisional" may not be the best term, couldn't think of a better term at the moment. Just my $0.02. *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 06/24/2017 09:15 AM, Nick Coghlan wrote: > On 24 June 2017 at 18:56, Thomas Kluyver wrote: >> For reference, I switched the default upload server for flit in version >> 0.11. > Oh, I'd also missed that 0.11 published sdists now. > > Given that, I think we should start recommending flit for new pure > Python projects (the lack of sdists was the only thing previously > making me hesitant about that), and then add it to the PyPI upload > migration list as well. > > It's also missing an entry in the Key Projects list: > https://packaging.python.org/key_projects/ > > I've filed an issue to discuss that here: > https://github.com/pypa/python-packaging-user-guide/issues/336 > > We should probably add enscons to the Key Projects list as well, since > that allows folks the option of using a full pip-installable C/C++ > build system as an alternative to setuptools/distutils, so I also > filed an issue to discuss that: > https://github.com/pypa/python-packaging-user-guide/issues/337 > > Cheers, > Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at nycresistor.com Sat Jun 24 12:01:12 2017 From: matt at nycresistor.com (Matt Joyce) Date: Sat, 24 Jun 2017 17:01:12 +0100 Subject: [Distutils] Can u provide metadata about the PyPI In-Reply-To: References: Message-ID: https://kirankoduru.github.io/python/pypi-stats.html this may be relevant to your interests. some of the pypi info is available on google big table. On Sat, Jun 24, 2017 at 5:02 AM, Priya Lakshmi wrote: > Respected Python expert > > I just want a help. Can u provide us the metadata available about the > modules in PyPI giant database. For example, we want the number of packages > available, the number of downloads for each package, etc.... > So we can think of doing some good maintenance work. > > thank you. > G Priyalakshmi > Assistant Professor (Selection Grade) > Department of Applied Mathematics and Computational Sciences > PSG College of Technology > Coimbatore > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jun 24 13:34:35 2017 From: brett at python.org (Brett Cannon) Date: Sat, 24 Jun 2017 17:34:35 +0000 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? Message-ID: When you go to PyPI.org for a project you will find a link to the "homepage". Now for some projects that's their development site, e.g. GitHub URL. For others it's their documentation site, e.g. Read the Docs. And not all projects link to both from their PyPI page (e.g. yesterday I noticed flit didn't directly link to its doc site, although Thomas fixed this when I pointed it out). So my question/idea is if it would make sense to have separate, explicit development and documentation URLs in the PyPI metadata? For me that would make a project's PyPI page a true homepage as I would know that no matter what I could find where it's developed or where the docs are by going to PyPI. This compares to now where either I gamble and go to PyPI in hopes the developer provided the link there or hope I craft the right search on Google (which based on my search yesterday for [Sphinx makefile] shows I don't always succeed at). Anyway, just an idea I had based on my flit experience yesterday plus a tweet sent to me. (And if PyPI already supports this somehow then Thomas should brace for the feature request from me ?.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Sat Jun 24 13:48:31 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sat, 24 Jun 2017 18:48:31 +0100 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: References: Message-ID: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> On Sat, Jun 24, 2017, at 06:34 PM, Brett Cannon wrote: > Anyway, just an idea I had based on my flit experience yesterday plus > a tweet sent to me. (And if PyPI already supports this somehow then > Thomas should brace for the feature request from me ?.) This prompted me to go and look at the metadata PEPs. I thought the URL fields were only 'Home-Page' and 'Download-URL', but in fact PEP 345 added a multi-use 'Project-URL' field: https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use I quite like this idea - rather than prescribing a couple of specific kinds of URL, it lets the project author list as many as make sense. Perhaps we should recommend a set of common labels for URLs in this field, e.g. 'Documentation', 'Bug tracker', 'Source repository'. This does leave open the question of which one to put in the mandatory 'Home-Page' field (I usually use the address of the repo), and whether to duplicate it in a 'Project-URL'. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Jun 24 13:49:10 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 24 Jun 2017 13:49:10 -0400 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: References: Message-ID: <3C8584C4-9B86-40BC-A8FA-7D0134C7B4D1@stufft.io> PyPI supports arbitrary key => URL mappings called project urls. The best way to implement this is probably to expose that feature with maybe some recommended keys to use (to avoid docs vs documentation fracture etc). As far as I know disutils2 was the only thing to ever support them. Sent from my iPhone > On Jun 24, 2017, at 1:34 PM, Brett Cannon wrote: > > When you go to PyPI.org for a project you will find a link to the "homepage". Now for some projects that's their development site, e.g. GitHub URL. For others it's their documentation site, e.g. Read the Docs. And not all projects link to both from their PyPI page (e.g. yesterday I noticed flit didn't directly link to its doc site, although Thomas fixed this when I pointed it out). > > So my question/idea is if it would make sense to have separate, explicit development and documentation URLs in the PyPI metadata? For me that would make a project's PyPI page a true homepage as I would know that no matter what I could find where it's developed or where the docs are by going to PyPI. This compares to now where either I gamble and go to PyPI in hopes the developer provided the link there or hope I craft the right search on Google (which based on my search yesterday for [Sphinx makefile] shows I don't always succeed at). > > Anyway, just an idea I had based on my flit experience yesterday plus a tweet sent to me. (And if PyPI already supports this somehow then Thomas should brace for the feature request from me ?.) > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Sat Jun 24 13:51:17 2017 From: donald at stufft.io (Donald Stufft) Date: Sat, 24 Jun 2017 13:51:17 -0400 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> Message-ID: It's only kind of mandatory. The spec says it is but nothing fails IIRC if you omit it. Perhaps we should just deprecate it and move everything to project urls. Sent from my iPhone > On Jun 24, 2017, at 1:48 PM, Thomas Kluyver wrote: > > This does leave open the question of which one to put in the mandatory 'Home-Page' field (I usually use the address of the repo), and whether to duplicate it in a 'Project-URL'. From brett at python.org Sat Jun 24 15:59:42 2017 From: brett at python.org (Brett Cannon) Date: Sat, 24 Jun 2017 19:59:42 +0000 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> Message-ID: On Sat, Jun 24, 2017, 10:49 Thomas Kluyver, wrote: > On Sat, Jun 24, 2017, at 06:34 PM, Brett Cannon wrote: > > Anyway, just an idea I had based on my flit experience yesterday plus a > tweet sent to me. (And if PyPI already supports this somehow then Thomas > should brace for the feature request from me ?.) > > > This prompted me to go and look at the metadata PEPs. I thought the URL > fields were only 'Home-Page' and 'Download-URL', but in fact PEP 345 added > a multi-use 'Project-URL' field: > > https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use > > I quite like this idea - rather than prescribing a couple of specific > kinds of URL, it lets the project author list as many as make sense. > Perhaps we should recommend a set of common labels for URLs in this field, > e.g. 'Documentation', 'Bug tracker', 'Source repository'. > In flit's case it could have some reasonable set of defaults supported in the metadata section and then have a more general "URLs" section that's more free-form. -brett > This does leave open the question of which one to put in the mandatory > 'Home-Page' field (I usually use the address of the repo), and whether to > duplicate it in a 'Project-URL'. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jun 24 16:00:33 2017 From: brett at python.org (Brett Cannon) Date: Sat, 24 Jun 2017 20:00:33 +0000 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> Message-ID: On Sat, Jun 24, 2017, 10:51 Donald Stufft, wrote: > It's only kind of mandatory. The spec says it is but nothing fails IIRC if > you omit it. Perhaps we should just deprecate it and move everything to > project urls. > That sounds reasonable toe if the flexible, general case is going to be supported. -Brett > Sent from my iPhone > > > On Jun 24, 2017, at 1:48 PM, Thomas Kluyver > wrote: > > > > This does leave open the question of which one to put in the mandatory > 'Home-Page' field (I usually use the address of the repo), and whether to > duplicate it in a 'Project-URL'. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jwilk at jwilk.net Sat Jun 24 17:32:00 2017 From: jwilk at jwilk.net (Jakub Wilk) Date: Sat, 24 Jun 2017 23:32:00 +0200 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> Message-ID: <20170624213200.kjgpzlbtxbarrsx4@jwilk.net> * Donald Stufft , 2017-06-23, 10:21: >>Where can we find the exact versions of each that default to ?pypi.org? for >>uploads? > >Umm. Twine 1.8.0, Python 3.4.6, 3.5.3, 3.6.0, Not sure on 2.7, $ git tag --contains=7127b62702bd50bf44138e6f57334887cbcb5ca8 v2.7.13rc1 v2.7.13 -- Jakub Wilk From ben+python at benfinney.id.au Sat Jun 24 20:42:09 2017 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 25 Jun 2017 10:42:09 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> <1498294581.2774227.1019825472.64435BE5@webmail.messagingengine.com> <154f5a8f-bd1c-52b4-c18c-1e39847d6a9d@thesyrings.us> Message-ID: <85zicwsxsu.fsf@benfinney.id.au> Randy Syring writes: > [?] would it hurt to have an "provisional" area to the key projects > list that makes people aware of the projects without "blessing" them > as Key Projects. It might hurt, yes. I am already quite confused by what is meant to be done in the Brave New World described in this thread. Pointing to a document that gives a set of tools is helpful ? now I have only the confusion of figuring out whether all my development and deployment environments can use those ? but that document becomes *less* useful as you add more decisions I need to make. > The whole point of that section of the guide is to raise awareness, > yes? That describes the purpose of an announcement like this. I'm now aware of the change that's looming. On the conitrary, the point of the section of the guide, for a user like me, is to *simplify* the actions I need to take to survive the transition. Adding more options to decide between, however well-intentioned, will work against that purpose. -- \ ?You could augment an earwig to the point where it understood | `\ nuclear physics, but it would still be a very stupid thing to | _o__) do!? ?The Doctor, _The Two Doctors_ | Ben Finney From ncoghlan at gmail.com Sat Jun 24 21:33:17 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 11:33:17 +1000 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> Message-ID: On 25 June 2017 at 03:51, Donald Stufft wrote: > It's only kind of mandatory. The spec says it is but nothing fails IIRC if you omit it. Perhaps we should just deprecate it and move everything to project urls. That's the direction I was going in PEP 426/459: https://www.python.org/dev/peps/pep-0459/#project-urls The two missing pieces I now see would be recommended tags for "Participate" and "Funding" URLs. I'd favour "Participate" over any variant of "Contribute", as without context, "Contribute" makes me think of financial support in the crowdfunding/tip jar sense. "Funding" is general enough to be suitable for crowdfunding/tip jar links, freelancing/contracting links, and links to employer/sponsor open source info pages. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Jun 24 21:38:53 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 11:38:53 +1000 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <154f5a8f-bd1c-52b4-c18c-1e39847d6a9d@thesyrings.us> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <85mv8zt24a.fsf@benfinney.id.au> <49B223D5-2C34-4FF8-AFEE-B90F36F4188D@stufft.io> <858tkiu5m0.fsf@benfinney.id.au> <1498294581.2774227.1019825472.64435BE5@webmail.messagingengine.com> <154f5a8f-bd1c-52b4-c18c-1e39847d6a9d@thesyrings.us> Message-ID: On 25 June 2017 at 01:18, Randy Syring wrote: > For what it's worth, for the projects that are almost ready to recommend, > but not quite there (like Flit per the discussion on issue 336), would it > hurt to have an "provisional" area to the key projects list that makes > people aware of the projects without "blessing" them as Key Projects. The > whole point of that section of the guide is to raise awareness, yes? Well > if something like Flit is almost ready, but pending for a small reason or > two, why not publish it anyway with some kind of "provision" label or > something and an explanation of why the project is important, but not quite > yet ready? The key projects list is already aimed more at folks wanting to dive into the details of how PyPA and the Python packaging ecosystem actually work in practice (e.g. packaging and distlib are both listed, and those are really only relevant to folks writing their own packaging related tools). The "definitely relevant to end users" list is the one at https://packaging.python.org/guides/tool-recommendations/ (and after discussing the idea with Thomas, we're not going to add flit there until after its 1.0 release) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Sun Jun 25 02:58:52 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 24 Jun 2017 23:58:52 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Sat, Jun 17, 2017 at 7:41 AM, Nick Coghlan wrote: > On 17 June 2017 at 17:07, Nathaniel Smith wrote: >> On Fri, Jun 16, 2017 at 11:28 PM, Donald Stufft wrote: >>>> >>>> On Jun 16, 2017, at 11:08 PM, Nathaniel Smith wrote: >>>> >>>> (because the legacy setup.py >>>> command errors out, because the build_sdist hook is missing, because >>>> the build_sdist hook errors out...); probably falling back on >>>> shutil.copytree. >>> >>> >>> Ah okay. This is where the disconnect was. I assumed if a project was using a PEP 517 build tool the result of a build_sdist failure (or a build_wheel failure) would be surfacing an error to the end user, not some sort of fallback. >> >> What we might be able to get away with in the PEP 517 transition is to >> provide an explicit way of signalling "build_sdist failed because >> that's just not a supported operation", so that pip could distinguish >> that from "something unexpected blew up" and only fall back in the >> first case. > > That's what `prepare_wheel_input_files` gives us: if the backend > doesn't implement it, then `build_sdist` is expected to always > succeed, and any errors should be reported to the end user. Well, yeah, this is a discussion about a potentially simpler/more flexible alternative to prepare_wheel_input_files, so I'd hope that it gives us something similar :-). > OTOH, if `build_sdist` can fail (e.g. due to missing dependencies that > aren't needed just to build a wheel), then the backend should offer a > `prepare_wheel_input_files` that offers the expected guarantee (i.e. > it will only fail for reasons that should be reported to the user > attempting to do the build). > > The frontend then *won't* have a fallback copy strategy for PEP 517 > backends - if `prepare_wheel_input_files` is present and fails, or if > it's missing and `build_sdist` fails, then you have a failed wheel > build on your hands. > > The legacy copying strategy would then only be used when falling all > the way back to the `setup.py` interface, not as an attempt to > continue the build in the face of a backend bug. > > This all gets murkier if you try to make the build_sdist hook more > complex instead of just having two different hooks for frontends to > call that tell the backend what the frontend actually wants (i.e. > either an actual sdist, or just a clean set of exported files to use > as input to a wheel build). Hmm, I don't think it does get murkier. Here's how I'm thinking about it: Everyone agrees: - build_sdist's semantics are that it builds an sdist, it's up to the caller to figure out why they want that - the build_sdist hook can be missing (unless we're planning to enforce its presence from the start, but that seems unlikely given that pip currently has no sdist handling at all), which is a pretty obvious signal to the caller that it's not supported - there's some problem with what to do for flit, where "can I build an sdist" is determined at runtime. I guess it could make the build_sdist attribute be defined via a __getattr__ method that sniffs the source tree before deciding whether the attribute should be present, but that seems unpleasant. The current spec's solution: - Let's define an exhaustive list of all the reasons you might want to make an sdist: 1) you're a hypothetical future version of pip that has decided to implement some specific build-from-unpacked-source-tree semantics 2) the user has specifically requested an sdist, so if one can't be made then the only thing to do is to fail and pass on some unstructured error log 3) that's it, we've made a list of all possible front-end semantics. - Then define a hook that does (1) and a hook that does (2). My suggestion: - Let's keep the spec around build_sdist very simple, all it does is worry about building an sdist and doesn't care at all what it's for, that's the frontend's job - Within that scope, one very well-defined thing that a backend might want to say to a frontend is "this backend doesn't support building an sdist from this source tree". And we even support this already, just in a kinda awkward way requiring __getattr__. - so instead let's provide a standard way to do that - and then backends and the standard don't (yet) need to know anything about pip's particular use case for sdists of wanting to potentially in the future use them as an intermediate step in building wheels from source directories; all the necessary logic is a concern for pip (and possibly some future PEP), not for PEP 517. (Hopefully this also answers Paul's objection ? I'm explicitly trying to take all the fallback-y logic out of the PEP, and replace it with a nicely semantic "that operation isn't supported" signal. It just so happens that one of the things pip could do with that signal is to decide to use an alternative method of accomplishing its goal that doesn't involve creating an sdist.) -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Sun Jun 25 03:26:32 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Jun 2017 00:26:32 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1498316222.3887873.1019999072.063E60AA@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498311843.2844465.1019960624.421AFA26@webmail.messagingengine.com> <1498316222.3887873.1019999072.063E60AA@webmail.messagingengine.com> Message-ID: On Sat, Jun 24, 2017 at 7:57 AM, Thomas Kluyver wrote: > Nick has merged that PR, and the updated PEP is visible here: > > https://www.python.org/dev/peps/pep-0517/ > > Hopefully we're nearing a consensus on this now. If you're interested, > please do have a read through the latest version. > Reading through it just now, there were some bits that gave me the impression that prepare_build_wheel_files must *always* be called before calling build_wheel (e.g. "Because the wheel will be built from a temporary build directory, build_wheel may create intermediate files in the working directory, and does not need to take care to clean them up."). If we end up keeping prepare_build_wheel_files, then I think we should add some text making clear exactly which paths are legal. ...and in the current spec this is complex enough that something like a state machine might be the clearest way? Here's an attempt at an NFA representation. Notation: [foo] is a state, --bar--> is a labeled transition. Required transitions: [VCS checkout] --build_wheel--> [wheel] Required transitions if you want to have a useful sdist at all: [VCS checkout] --build_sdist+unpack--> [non-VCS tree] [non-VCS tree] --build_wheel--> [wheel] Required if you have a prepare_wheel_metadata hook: [VCS checkout] --prepare_wheel_metadata--> [checkout+metadata] [checkout+metadata] --build_wheel(..., metadata_directory=...)--> [wheel] [non-VCS tree] --prepare_wheel_metadata--> [non-VCS tree+metadata] [non-VCS tree+metadata] --build_wheel(..., metadata_directory=...)--> [wheel] Plus you should support EITHER: [non-VCS tree] --build_sdist+unpack--> [non-VCS tree] OR: [VCS checkout] --prepare_wheel_build_files--> [non-VCS tree] [non-VCS tree] --prepare_wheel_build_files--> [non-VCS tree] ...I'm not sure this is really making things clearer. (Though a graphviz version would be less terrible than the text version, maybe?) -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sun Jun 25 03:26:36 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 17:26:36 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 25 June 2017 at 16:58, Nathaniel Smith wrote: > The current spec's solution: > > - Let's define an exhaustive list of all the reasons you might want to > make an sdist: > 1) you're a hypothetical future version of pip that has decided to > implement some specific build-from-unpacked-source-tree semantics > 2) the user has specifically requested an sdist, so if one can't be > made then the only thing to do is to fail and pass on some > unstructured error log > 3) that's it, we've made a list of all possible front-end semantics. > - Then define a hook that does (1) and a hook that does (2). pip doesn't care about making sdists, it only cares about doing out of tree wheel builds. It's other tools (e.g. tox) that specifically care about making sdists. So we only have two front-end scanarios that we need to handle: 1. preparing for an out-of-tree call to build_wheel 2. actually building an sdist (e.g. for testing or publication) The possibility of only defining one hook arises from the fact that providing just the build_sdist hook would be sufficient for both tasks *if* we were willing to impose the constraint that everything a backend depends on to build an sdist will always also be a pre-requisite for building a wheel file. Thomas has indicated that that *isn't* the case for flit - he expects the `build_sdist` hook to need runtime dependencies (either Python level or external) that `build_wheel` doesn't. Thus `prepare_build_wheel_files` to cover the case where the frontend doesn't really want an sdist at all, it just wants to do an out-of-tree wheel build, but the backend doesn't want to ensure that the preconditions for `build_sdist` and `build_wheel` are identical. However, we've chosen to make the second hook optional, so backends that have a very simple `build_sdist` implementation don't have to worry about the optional wheel preparation hook. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Sun Jun 25 03:41:00 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Jun 2017 00:41:00 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Sun, Jun 25, 2017 at 12:26 AM, Nick Coghlan wrote: > On 25 June 2017 at 16:58, Nathaniel Smith wrote: >> The current spec's solution: >> >> - Let's define an exhaustive list of all the reasons you might want to >> make an sdist: >> 1) you're a hypothetical future version of pip that has decided to >> implement some specific build-from-unpacked-source-tree semantics >> 2) the user has specifically requested an sdist, so if one can't be >> made then the only thing to do is to fail and pass on some >> unstructured error log >> 3) that's it, we've made a list of all possible front-end semantics. >> - Then define a hook that does (1) and a hook that does (2). > > pip doesn't care about making sdists, it only cares about doing out of > tree wheel builds. It's other tools (e.g. tox) that specifically care > about making sdists. > > So we only have two front-end scanarios that we need to handle: > > 1. preparing for an out-of-tree call to build_wheel > 2. actually building an sdist (e.g. for testing or publication) Right, the core point of disagreement is: - I'm not convinced that this list is exhaustive. - I'm not convinced that we understand the out-of-tree build case, given that the sdist approach has never been implemented or deployed. - Even based on our current imperfect understanding of the out-of-tree build case, I'm not convinced that prepare_build_wheel_files solution is actually the best solution. (For all the reasons I've said before: the stated motivation for doing out-of-tree builds is that we don't trust the build system, but prepare_build_wheel_files forces us to trust the build system; flit *could* support sdists, the question is whether it's worth the effort, and I don't believe we really understand the trade-offs well enough to know the answer to that; in the flit case copytree() would work just as well as prepare_build_wheel_files anyway so there's no demonstrated need for this complexity.) Maybe you're right and there are exactly 2 front-end use cases and it will turn out that the current PEP addresses them perfectly. I don't have a crystal ball; I'm making an argument from ignorance. But I feel like allowing NotImplemented returns + ext_{toolname}_* hooks seems like it covers all the known cases about as well as the current design, while being substantially simpler and more future-proof. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sun Jun 25 03:41:19 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 17:41:19 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498311843.2844465.1019960624.421AFA26@webmail.messagingengine.com> <1498316222.3887873.1019999072.063E60AA@webmail.messagingengine.com> Message-ID: On 25 June 2017 at 17:26, Nathaniel Smith wrote: > On Sat, Jun 24, 2017 at 7:57 AM, Thomas Kluyver wrote: >> Nick has merged that PR, and the updated PEP is visible here: >> >> https://www.python.org/dev/peps/pep-0517/ >> >> Hopefully we're nearing a consensus on this now. If you're interested, >> please do have a read through the latest version. >> > > Reading through it just now, there were some bits that gave me the > impression that prepare_build_wheel_files must *always* be called > before calling build_wheel (e.g. "Because the wheel will be built from > a temporary build directory, build_wheel may create intermediate files > in the working directory, and does not need to take care to clean > them up."). If we end up keeping prepare_build_wheel_files, then I > think we should add some text making clear exactly which paths are > legal. Doing out-of-tree builds is completely optional for frontends, so the possible build paths are: In-place build: * just call build_wheel on the source tree Out-of-tree build (via sdist): * call build_sdist * unpack it into the build directory * call build_wheel on the build directory Out-of-tree build (prepare hook defined): * call prepare_wheel_build_files * call build_wheel on the prepared build directory The critical constraint on front-ends is that they need to ensure the appropriate dependencies are present before calling the affected hooks - beyond that, they're entirely free to do things however they like (e.g. always build in-place, always go via the sdist hook, or some combination there-of). Aside from not producing an archive, the bit that makes `prepare_wheel_build_files` notably different from `build_sdist` is that it relies on `get_build_wheel_requires` *not* on `get_build_sdist_requires`: https://www.python.org/dev/peps/pep-0517/#build-environment Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From thomas at kluyver.me.uk Sun Jun 25 03:46:16 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Sun, 25 Jun 2017 08:46:16 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498311843.2844465.1019960624.421AFA26@webmail.messagingengine.com> <1498316222.3887873.1019999072.063E60AA@webmail.messagingengine.com> Message-ID: <1498376776.4134899.1020443512.1A66903A@webmail.messagingengine.com> On Sun, Jun 25, 2017, at 08:41 AM, Nick Coghlan wrote: > Aside from not producing an archive, the bit that makes > `prepare_wheel_build_files` notably different from `build_sdist` is > that it relies on `get_build_wheel_requires` *not* on > `get_build_sdist_requires`: > https://www.python.org/dev/peps/pep-0517/#build-environment This reminds me: while implementing a wrapper to call PEP 517 hooks, a problem occurred to me. I have implemented prepare_wheel_build_files with a fallback to making an sdist if the hook is not defined: https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7a8500180f7/pep517/_in_process.py#L136 But in this case, the build_sdist hook will be invoked in an environment with the build-wheel-deps installed, not the build-sdist-deps. Working around this would require moving the fallback a couple of levels up the stack to a component that knows about installing packages for the build process. That's not impossible, but it's inelegant and less efficient. Thomas From ncoghlan at gmail.com Sun Jun 25 03:46:34 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Jun 2017 17:46:34 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 25 June 2017 at 17:41, Nathaniel Smith wrote: > Maybe you're right and there are exactly 2 front-end use cases and it > will turn out that the current PEP addresses them perfectly. I don't > have a crystal ball; I'm making an argument from ignorance. I'm not - we have two concrete potential consumers of the interface (pip and tox, aka "build to use" and "build to test"), and I'm designing the interface to cover their needs (i.e. out-of-tree wheel builds and actual sdists). If we discover other use cases later, we'll worry about them then (and the easy of doing so is the nicest benefit of defining this as a Python API), but the temptation to design in hyper-flexibility now falls under YAGNI (You Ain' Gonna Need It). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jwodder at gmail.com Sat Jun 24 13:43:12 2017 From: jwodder at gmail.com (John Thorvald Wodder II) Date: Sat, 24 Jun 2017 13:43:12 -0400 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: References: Message-ID: On 2017 Jun 24, at 13:34, Brett Cannon wrote: > When you go to PyPI.org for a project you will find a link to the "homepage". Now for some projects that's their development site, e.g. GitHub URL. For others it's their documentation site, e.g. Read the Docs. And not all projects link to both from their PyPI page (e.g. yesterday I noticed flit didn't directly link to its doc site, although Thomas fixed this when I pointed it out). > > So my question/idea is if it would make sense to have separate, explicit development and documentation URLs in the PyPI metadata? For me that would make a project's PyPI page a true homepage as I would know that no matter what I could find where it's developed or where the docs are by going to PyPI. This compares to now where either I gamble and go to PyPI in hopes the developer provided the link there or hope I craft the right search on Google (which based on my search yesterday for [Sphinx makefile] shows I don't always succeed at). The package data exposed by PyPI's JSON and XML-RPC APIs already includes a "docs_url" field; however, this seems to only ever be set for projects whose documentation is on http://pythonhosted.org with no way to point it to other domains. PEP 345 defines a Project-URL field[1] that could be used to tell PyPI where documentation is hosted, but I'm not aware of a single tool that does anything with or even lets you set that field. [1]: https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use From Ken.Lewis at trinityhealth.org Fri Jun 23 17:14:52 2017 From: Ken.Lewis at trinityhealth.org (Ken R. Lewis) Date: Fri, 23 Jun 2017 21:14:52 +0000 Subject: [Distutils] Python 3.6 Message-ID: Hello! I am new to Python as I just downloaded it today. I have a script I need to run that has "requests" in it. How do I go about getting this set up? Also, I am on a Windows 7 machine. Here is the code: #Need to install requests package for python #easy_install requests import requests thanks, Ken Ken Lewis Trinity Health I.T. Programmer Lead (701) 858-6423 Cell (701)833-8234 Fax (701) 858-6407 Ken.Lewis at trinityhealth.org ________________________________ Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is privileged, confidential, or otherwise exempt from disclosure under applicable law. Any unauthorized review, copy, use, disclosure or distribution is prohibited. If you have received this communication in error, please delete it from your system without copying or forwarding it and notify the sender of the error by reply e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jun 25 17:12:04 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 25 Jun 2017 22:12:04 +0100 Subject: [Distutils] Python 3.6 In-Reply-To: References: Message-ID: On 23 June 2017 at 22:14, Ken R. Lewis wrote: > I have a script I need to run that has ?requests? in it. How do I go about > getting this set up? At a command prompt (*not* the Python interpreter prompt) enter "py -m pip install requests". That will install requests for you and make it available for import. Paul From rfox.beeches at btinternet.com Mon Jun 26 05:03:28 2017 From: rfox.beeches at btinternet.com (Rick) Date: Mon, 26 Jun 2017 10:03:28 +0100 Subject: [Distutils] Fatal error "Python.h" Message-ID: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> Hi Can anyone help? I?m pretty new to Python and Raspberry Pi. When running PIP, I seem to get fatal error: Python.h: No such file or directory include ?Python.h? compilation terminated I get this when attempting to download both Cython and scikit-image. So I?m assuming it is some absent file or path reference on the Raspberry Pi installation I have, rather than in the distribution package? What is this file ?Python.h? that it is trying to include? -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at nycresistor.com Mon Jun 26 06:42:53 2017 From: matt at nycresistor.com (Matt Joyce) Date: Mon, 26 Jun 2017 06:42:53 -0400 Subject: [Distutils] Fatal error "Python.h" In-Reply-To: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> References: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> Message-ID: Look for a python-dev or python-devel package. On Jun 26, 2017 6:36 AM, "Rick" wrote: > Hi > Can anyone help? > I?m pretty new to Python and Raspberry Pi. > When running PIP, I seem to get fatal error: > > Python.h: No such file or directory > > include ?Python.h? > > compilation terminated > > I get this when attempting to download both *Cython* and *scikit-image*. > So I?m assuming it is some absent file or path reference on the Raspberry > Pi installation I have, rather than in the distribution package? > > What is this file ?Python.h? that it is trying to include? > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubernostrum at gmail.com Mon Jun 26 06:43:38 2017 From: ubernostrum at gmail.com (James Bennett) Date: Mon, 26 Jun 2017 03:43:38 -0700 Subject: [Distutils] Fatal error "Python.h" In-Reply-To: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> References: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> Message-ID: Most Linux distributions package the Python development header files (which are needed to compile Python modules written in C) separately from the Python language itself. Ubuntu calls their package 'python-dev', and you could 'apt-get install python-dev' to install it. Other Linux distributions will call it something similar (such as 'python-devel'). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Mon Jun 26 07:00:45 2017 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 26 Jun 2017 12:00:45 +0100 Subject: [Distutils] Fatal error "Python.h" In-Reply-To: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> References: <07A22CD8-341B-4E55-98AD-3C3569783B8C@btinternet.com> Message-ID: <33fdcac0-7f44-759a-b6ce-9111293a47df@timgolden.me.uk> On 26/06/2017 10:03, Rick wrote: > Hi > Can anyone help? > I?m pretty new to Python and Raspberry Pi. > When running PIP, I seem to get fatal error: > > Python.h: No such file or directory > > include ?Python.h? > > compilation terminated > > I get this when attempting to download both _Cython_ and _scikit-image_. > So I?m assuming it is some absent file or path reference on the > Raspberry Pi installation I have, rather than in the distribution package? > > What is this file ?Python.h? that it is trying to include? Others have answered the immediate question. But, in general, you'll probably want to install tools like Cython and scikit from the Raspberry Pi repositories directly (where they come prebuilt). I don't have an RPi here, but try something like "sudo apt install cython scikit" TJG From njs at pobox.com Mon Jun 26 08:36:07 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Jun 2017 05:36:07 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Jun 25, 2017 12:46 AM, "Nick Coghlan" wrote: On 25 June 2017 at 17:41, Nathaniel Smith wrote: > Maybe you're right and there are exactly 2 front-end use cases and it > will turn out that the current PEP addresses them perfectly. I don't > have a crystal ball; I'm making an argument from ignorance. I'm not - we have two concrete potential consumers of the interface (pip and tox, aka "build to use" and "build to test"), and I'm designing the interface to cover their needs (i.e. out-of-tree wheel builds and actual sdists). If we discover other use cases later, we'll worry about them then (and the easy of doing so is the nicest benefit of defining this as a Python API), but the temptation to design in hyper-flexibility now falls under YAGNI (You Ain' Gonna Need It). My proposal also covers their needs AFAICT? At least I thought Donald said he thought the would work for pip. And you can't use YAGNI to argue for a more complicated proposal, that's cheating :-). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Jun 26 11:33:16 2017 From: dholth at gmail.com (Daniel Holth) Date: Mon, 26 Jun 2017 15:33:16 +0000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: My impressions on reading what is hopefully the current version of the PEP Are there different paths to get to sdist/wheel for example tree -> prepare sdist/wheel files -> build sdist and tree -> build sdist, tree -> build wheel depending on what [pip] decides to do? Should the frontend do both and compare the result to make sure it is the same? I'm not entirely clear on what the prepare hooks should do. The rationalizations interleaved between the descriptions of each hook distract from what the hooks should actually do. Suggested replacement rationalization: "Some people like to build [format]. This hook builds [format]". I will make an effort to implement this for enscons, probably with the most direct tree -> wheel, tree -> sdist hooks and without any optional hooks. Thanks, Daniel On Mon, Jun 26, 2017 at 8:36 AM Nathaniel Smith wrote: > On Jun 25, 2017 12:46 AM, "Nick Coghlan" wrote: > > On 25 June 2017 at 17:41, Nathaniel Smith wrote: > > Maybe you're right and there are exactly 2 front-end use cases and it > > will turn out that the current PEP addresses them perfectly. I don't > > have a crystal ball; I'm making an argument from ignorance. > > I'm not - we have two concrete potential consumers of the interface > (pip and tox, aka "build to use" and "build to test"), and I'm > designing the interface to cover their needs (i.e. out-of-tree wheel > builds and actual sdists). > > If we discover other use cases later, we'll worry about them then (and > the easy of doing so is the nicest benefit of defining this as a > Python API), but the temptation to design in hyper-flexibility now > falls under YAGNI (You Ain' Gonna Need It). > > > My proposal also covers their needs AFAICT? At least I thought Donald said > he thought the would work for pip. And you can't use YAGNI to argue for a > more complicated proposal, that's cheating :-). > > -n > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jun 27 03:27:30 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Jun 2017 17:27:30 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 26 June 2017 at 22:36, Nathaniel Smith wrote: > On Jun 25, 2017 12:46 AM, "Nick Coghlan" wrote: > > On 25 June 2017 at 17:41, Nathaniel Smith wrote: >> Maybe you're right and there are exactly 2 front-end use cases and it >> will turn out that the current PEP addresses them perfectly. I don't >> have a crystal ball; I'm making an argument from ignorance. > > I'm not - we have two concrete potential consumers of the interface > (pip and tox, aka "build to use" and "build to test"), and I'm > designing the interface to cover their needs (i.e. out-of-tree wheel > builds and actual sdists). > > If we discover other use cases later, we'll worry about them then (and > the easy of doing so is the nicest benefit of defining this as a > Python API), but the temptation to design in hyper-flexibility now > falls under YAGNI (You Ain' Gonna Need It). > > > My proposal also covers their needs AFAICT? No, as you don't know in your proposal whether or not build_sdist can fail until after you've already called it, and you always need to call both `get_build_sdist_requires()` and `build_sdist()`, even if you only care about doing an out-of-tree wheel build. That categorically rules out two-pass introspection based implementations that first interrogate the backend to find out which hooks it supports, and then use that to decide their execution strategy. Tangent: it turns out the word order difference between `get_build_wheel_requires` and `prepare_wheel_build_files` hurts my brain, so I've changed the latter to `prepare_wheel_input_files` below. That keeps the common prefix with `prepare_wheel_metadata`, while avoiding the confusing word order reversal relative to `build_wheel`. That is, the current PEP allows a frontend to do the following when given an arbitrary source tree rather than an sdist archive: 1. Look for `prepare_wheel_input_files()` on the backend. If you find it: - call `get_build_wheel_requires()` - install the additional dependencies - call `prepare_wheel_input_files()` - call `build_wheel()` 2. If you don't find the input preparation hook: - call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - unpack the sdist into the build directory - call `build_wheel()` Failure of any hook in any way (whether that's raising an exception or returning something that isn't in line with the requirements of PEP 517) indicates a failed build. To put it another way, the key reason we can be confident that PEP 517 is comprehensive as currently drafted is that we only have two target artifact types (sdist & wheel), one source archiving strategy (in-tree), and two binary build strategies (in-tree or out-of-tree). That means source archiving is straightforward: - call `get_build_sdist_requires()` - install the additional dependencies - call `build_sdist()` In-tree binary builds are also straightforward: - call `get_build_wheel_requires()` - install the additional dependencies - call `build_wheel()` Out-of-tree binary builds via sdist combine the two: - call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - unpack the sdist into the build directory - call `build_wheel()` (Frontends would also be free to use two distinct environments, one to create the sdist, and then a separate one to build the wheel) However, that last strategy has an undesirable property: it means that installing the sdist archiving dependencies becomes a constraint on out-of-tree builds, which is annoying for both frontends and backends. Adding direct support for out-of-tree builds resolves that problem. Out-of-tree binary builds via a dedicated hook: - call `get_build_wheel_requires()` - install the additional dependencies - call `prepare_wheel_input_files()` - call `build_wheel()` By contrast, if we only define a "try it and see if it works" approach for build_sdist, and provide no other way to request an out-of-tree build, then frontends *have* to do the following: - call `get_build_sdist_requires()` - call `get_build_wheel_requires()` - install both sets of additional dependencies - call `build_sdist()` - if it returns `NotImplemented`... umm, give up, same as if it failed? Fall back to the bad "copy everything" default that pip is trying to get away from? - unpack the sdist into the build directory - call `build_wheel()` Without a dedicated out-of-tree build preparation hook, there's no way for frontends to know in advance that build_sdist might fail, there's no way for them to avoid installing the sdist archiving dependencies when doing an out-of-tree build, and there's no way for backends to provide a fallback build directory preparation strategy that relies solely on the wheel building dependencies. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Jun 27 03:36:58 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Jun 2017 17:36:58 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 27 June 2017 at 01:33, Daniel Holth wrote: > My impressions on reading what is hopefully the current version of the PEP > > Are there different paths to get to sdist/wheel for example tree -> prepare > sdist/wheel files -> build sdist and tree -> build sdist, tree -> build > wheel depending on what [pip] decides to do? Should the frontend do both and > compare the result to make sure it is the same? > > I'm not entirely clear on what the prepare hooks should do. > > The rationalizations interleaved between the descriptions of each hook > distract from what the hooks should actually do. Suggested replacement > rationalization: "Some people like to build [format]. This hook builds > [format]". I think it would be worth restructuring that part of the PEP so the required hooks (build_sdist, build_wheel) are listed in their own section, followed by a separate section for the optional ones (get_build_sdist_requires, get_build_wheel_requires, prepare_wheel_metadata, prepare_wheel_input_files) The latter four all already define what frontends are expected to do if they're missing: get_build_sdist_requires: assume no extra dependencies for build_sdist get_build_wheel_requires: assume no extra dependencies for build_wheel prepare_wheel_metadata: call build_wheel and extract the dist-info directory prepare_wheel_input_files: call build_sdist and unpack the resulting archive Backends will only need to implement those if the default behaviours are either wrong (e.g. the backend may need extra platform dependent dependencies), or grossly inefficient. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Tue Jun 27 04:54:26 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jun 2017 01:54:26 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan wrote: > On 26 June 2017 at 22:36, Nathaniel Smith wrote: >> On Jun 25, 2017 12:46 AM, "Nick Coghlan" wrote: >> >> On 25 June 2017 at 17:41, Nathaniel Smith wrote: >>> Maybe you're right and there are exactly 2 front-end use cases and it >>> will turn out that the current PEP addresses them perfectly. I don't >>> have a crystal ball; I'm making an argument from ignorance. >> >> I'm not - we have two concrete potential consumers of the interface >> (pip and tox, aka "build to use" and "build to test"), and I'm >> designing the interface to cover their needs (i.e. out-of-tree wheel >> builds and actual sdists). >> >> If we discover other use cases later, we'll worry about them then (and >> the easy of doing so is the nicest benefit of defining this as a >> Python API), but the temptation to design in hyper-flexibility now >> falls under YAGNI (You Ain' Gonna Need It). >> >> >> My proposal also covers their needs AFAICT? > > No, as you don't know in your proposal whether or not build_sdist can > fail until after you've already called it, and you always need to call > both `get_build_sdist_requires()` and `build_sdist()`, even if you > only care about doing an out-of-tree wheel build. Ah, this is an interesting point, thank you! I hadn't thought of that. But... it doesn't seem like a big deal in practice? If a build backend knows that it can't build an sdist from a given tree, then its get_build_sdist_requires hook (which is arbitrary code after all) can surely figure this out and return []. Or we could allow get_build_sdist_requires to also return NotImplemented (defining that to mean "building sdists is not implemented, don't even try calling build_sdist"), though maybe that's unnecessary fiddliness. > 2. If you don't find the input preparation hook: > > - call `get_build_sdist_requires()` > - call `get_build_wheel_requires()` > - install both sets of additional dependencies > - call `build_sdist()` > - unpack the sdist into the build directory > - call `build_wheel()` Side note: I think this is wrong -- I think you can't call get_build_wheel_requires until after you've unpacked the sdist, because that's a whole new tree that might be arbitrarily different. (In practice I guess this should never happen, but we should be clear conceptually I think.) -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Tue Jun 27 07:20:27 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Jun 2017 21:20:27 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 27 June 2017 at 18:54, Nathaniel Smith wrote: > On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan wrote: >> 2. If you don't find the input preparation hook: >> >> - call `get_build_sdist_requires()` >> - call `get_build_wheel_requires()` >> - install both sets of additional dependencies >> - call `build_sdist()` >> - unpack the sdist into the build directory >> - call `build_wheel()` > > Side note: I think this is wrong -- I think you can't call > get_build_wheel_requires until after you've unpacked the sdist, > because that's a whole new tree that might be arbitrarily different. > (In practice I guess this should never happen, but we should be clear > conceptually I think.) No, both `get_build_wheel_requires()` and `build_wheel()` also have to work in-place - backends aren't allowed to assume that *all* builds will be out-of-tree, even though pip specifically is expected to work that way. Neither of these hooks is allowed to give different results depending on whether you're doing an in-place or out-of-tree build. To put it another way, there are two build paths that backends are required to support: - in-place, direct from the original source tree - out-of-tree, from an unpacked sdist There's also a third, optional, build path for backends that define `prepare_wheel_input_files`: - out-of-tree, from a prepared wheel input directory The result wheel should be "the same" regardless of how the build is executed. In practice, we expect there will be some variation, just because reproducible builds are hard. However, the output of the different paths should be at least as similar as any two builds of the same source tree run at different times with the same system configuration. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Tue Jun 27 11:56:08 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 27 Jun 2017 11:56:08 -0400 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? References: Message-ID: <20170627115608.5d97cf7b@subdivisions.wooz.org> On Jun 24, 2017, at 05:34 PM, Brett Cannon wrote: >So my question/idea is if it would make sense to have separate, explicit >development and documentation URLs in the PyPI metadata? Wholehearted +1 I already use PyPI as the first stop for finding out about a library or Python project. I have a nice Chrome search shortcut to do a search, and once I've found the PyPI page I have a much greater chance of finding a link to upstream. Regardless of whether that points to the development site or the documentation site, I can usually find a way back to the other one. Still, I'd love for it to be much more explicit. I also think pythonhosted should be deprecated if it's not already. It's kind of more of a pain than useful these days. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Jun 27 11:58:43 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 27 Jun 2017 11:58:43 -0400 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> Message-ID: <20170627115843.27a95390@subdivisions.wooz.org> On Jun 25, 2017, at 11:33 AM, Nick Coghlan wrote: >I'd favour "Participate" over any variant of "Contribute", as without >context, "Contribute" makes me think of financial support in the >crowdfunding/tip jar sense. "Participate" may mean two different things. * Here's the development home, with a repo and issue tracker, contributions welcome! * Here's the mailing list or other forum where we discuss the future of Guido's Magical Mystery Time Machine. It would be nice to be able to capture both meanings. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From thomas at kluyver.me.uk Tue Jun 27 12:57:30 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 27 Jun 2017 17:57:30 +0100 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: <20170627115843.27a95390@subdivisions.wooz.org> References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> <20170627115843.27a95390@subdivisions.wooz.org> Message-ID: <1498582650.2206361.1023063584.60D9190F@webmail.messagingengine.com> On Tue, Jun 27, 2017, at 04:58 PM, Barry Warsaw wrote: > On Jun 25, 2017, at 11:33 AM, Nick Coghlan wrote: > >I'd favour "Participate" over any variant of "Contribute", as without > >context, "Contribute" makes me think of financial support in the > >crowdfunding/tip jar sense. > > "Participate" may mean two different things. > > * Here's the development home, with a repo and issue tracker, > contributions > welcome! > > * Here's the mailing list or other forum where we discuss the future of > Guido's Magical Mystery Time Machine. Perhaps this points to labelling URLs with nouns rather than verbs: things like 'mailing list', 'source code' or 'issue tracker' seem less ambiguous than 'participate' or 'contribute'. Thomas From ncoghlan at gmail.com Tue Jun 27 22:09:51 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jun 2017 12:09:51 +1000 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: <20170627115843.27a95390@subdivisions.wooz.org> References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> <20170627115843.27a95390@subdivisions.wooz.org> Message-ID: On 28 June 2017 at 01:58, Barry Warsaw wrote: > On Jun 25, 2017, at 11:33 AM, Nick Coghlan wrote: > >>I'd favour "Participate" over any variant of "Contribute", as without >>context, "Contribute" makes me think of financial support in the >>crowdfunding/tip jar sense. > > "Participate" may mean two different things. > > * Here's the development home, with a repo and issue tracker, contributions > welcome! > > * Here's the mailing list or other forum where we discuss the future of > Guido's Magical Mystery Time Machine. > > It would be nice to be able to capture both meanings. This isn't feasible to capture generically, since projects have so many different approaches to communication. 1. A lot of modern projects *only* have their repo & issue tracker, with no other dedicated discussion or help forum (with sites like Stack Overflow handling requests for help) 2. Bigger projects will have *multiple* venues for communication, usually including at least one near-real-time channel (e.g. IRC, Slack, Gitter), and one more asynchronous one (e.g. a mailing list or web forum) So the only generic reader-centric (rather than publisher-centric) distinctions we can reliably make are: * "I want to use this project, tell me how" (aka "Documentation") * "I want to participate in the design and development of this project, tell me how" (aka "Participate") * "I want to help fund the design and development of this project, tell me how" (aka "Funding") That doesn't preclude having other more specific links (e.g. the PEP 459 draft also proposes "Home", "Repository" and "Tracker": https://www.python.org/dev/peps/pep-0459/#project-urls), it's just about reserving the "Documentation" tag primarily for user documentation rather than contributor documentation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Tue Jun 27 23:43:41 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jun 2017 20:43:41 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan wrote: > On 27 June 2017 at 18:54, Nathaniel Smith wrote: >> On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan wrote: >>> 2. If you don't find the input preparation hook: >>> >>> - call `get_build_sdist_requires()` >>> - call `get_build_wheel_requires()` >>> - install both sets of additional dependencies >>> - call `build_sdist()` >>> - unpack the sdist into the build directory >>> - call `build_wheel()` >> >> Side note: I think this is wrong -- I think you can't call >> get_build_wheel_requires until after you've unpacked the sdist, >> because that's a whole new tree that might be arbitrarily different. >> (In practice I guess this should never happen, but we should be clear >> conceptually I think.) > > No, both `get_build_wheel_requires()` and `build_wheel()` also have to > work in-place - backends aren't allowed to assume that *all* builds > will be out-of-tree, even though pip specifically is expected to work > that way. Neither of these hooks is allowed to give different results > depending on whether you're doing an in-place or out-of-tree build. Right. But if you're building and sdist and unpacking it and then calling build_wheel there, then that's not an out-of-tree build, it's an in-tree build in a new tree. The wheel hooks are certainly allowed to give different results in a VCS-tree and an unpacked-sdist-tree, they have different pyproject.toml files! > To put it another way, there are two build paths that backends are > required to support: > > - in-place, direct from the original source tree > - out-of-tree, from an unpacked sdist > > There's also a third, optional, build path for backends that define > `prepare_wheel_input_files`: > > - out-of-tree, from a prepared wheel input directory I would say that there's only one required path, which is in-place. (But the tree that they do the in-place build in might have a more or less complicated history). It worries me that you seem to think that build_sdist and build_wheel should be coupled like this... this proliferation of cases and the reification of "out of tree builds" as a special thing that's different from an in-tree build in a temporary tree is already what worried me about prepare_wheel_input_files, and now it seems to be spreading :-). -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Wed Jun 28 00:20:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jun 2017 14:20:25 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 28 June 2017 at 13:43, Nathaniel Smith wrote: > On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan wrote: >> On 27 June 2017 at 18:54, Nathaniel Smith wrote: >>> On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan wrote: >>>> 2. If you don't find the input preparation hook: >>>> >>>> - call `get_build_sdist_requires()` >>>> - call `get_build_wheel_requires()` >>>> - install both sets of additional dependencies >>>> - call `build_sdist()` >>>> - unpack the sdist into the build directory >>>> - call `build_wheel()` >>> >>> Side note: I think this is wrong -- I think you can't call >>> get_build_wheel_requires until after you've unpacked the sdist, >>> because that's a whole new tree that might be arbitrarily different. >>> (In practice I guess this should never happen, but we should be clear >>> conceptually I think.) >> >> No, both `get_build_wheel_requires()` and `build_wheel()` also have to >> work in-place - backends aren't allowed to assume that *all* builds >> will be out-of-tree, even though pip specifically is expected to work >> that way. Neither of these hooks is allowed to give different results >> depending on whether you're doing an in-place or out-of-tree build. > > Right. But if you're building and sdist and unpacking it and then > calling build_wheel there, then that's not an out-of-tree build, it's > an in-tree build in a new tree. The wheel hooks are certainly allowed > to give different results in a VCS-tree and an unpacked-sdist-tree, > they have different pyproject.toml files! Umm, no. The pyproject.toml file MUST NOT change when building the sdist. It's a human-managed input file - if any step of the build process mutates it, that step is *broken*. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Wed Jun 28 00:25:06 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jun 2017 21:25:06 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Tue, Jun 27, 2017 at 9:20 PM, Nick Coghlan wrote: > On 28 June 2017 at 13:43, Nathaniel Smith wrote: >> On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan wrote: >>> On 27 June 2017 at 18:54, Nathaniel Smith wrote: >>>> On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan wrote: >>>>> 2. If you don't find the input preparation hook: >>>>> >>>>> - call `get_build_sdist_requires()` >>>>> - call `get_build_wheel_requires()` >>>>> - install both sets of additional dependencies >>>>> - call `build_sdist()` >>>>> - unpack the sdist into the build directory >>>>> - call `build_wheel()` >>>> >>>> Side note: I think this is wrong -- I think you can't call >>>> get_build_wheel_requires until after you've unpacked the sdist, >>>> because that's a whole new tree that might be arbitrarily different. >>>> (In practice I guess this should never happen, but we should be clear >>>> conceptually I think.) >>> >>> No, both `get_build_wheel_requires()` and `build_wheel()` also have to >>> work in-place - backends aren't allowed to assume that *all* builds >>> will be out-of-tree, even though pip specifically is expected to work >>> that way. Neither of these hooks is allowed to give different results >>> depending on whether you're doing an in-place or out-of-tree build. >> >> Right. But if you're building and sdist and unpacking it and then >> calling build_wheel there, then that's not an out-of-tree build, it's >> an in-tree build in a new tree. The wheel hooks are certainly allowed >> to give different results in a VCS-tree and an unpacked-sdist-tree, >> they have different pyproject.toml files! > > Umm, no. The pyproject.toml file MUST NOT change when building the > sdist. It's a human-managed input file - if any step of the build > process mutates it, that step is *broken*. I agree that changing the pyproject.toml file is a terrible idea in most cases I still think we should do the "right thing" when it happens, where "right" means "doesn't involve complex underspecified coupling between conceptually distinct phases", because this thing needs to fit in people's brains. It's not so much that pyproject.toml changing is an important case, as that if we can even *tell* whether it changed then there's something wrong with our architecture. pyproject.toml is just an example anyway; get_build_wheel_requires runs arbitrary code, and that code could potentially return different results in different trees. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Wed Jun 28 00:29:05 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jun 2017 14:29:05 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: Sorry, I also meant to reply to this part before hitting send. On 28 June 2017 at 13:43, Nathaniel Smith wrote: > I would say that there's only one required path, which is in-place. > (But the tree that they do the in-place build in might have a more or > less complicated history). It worries me that you seem to think that > build_sdist and build_wheel should be coupled like this... this > proliferation of cases and the reification of "out of tree builds" as > a special thing that's different from an in-tree build in a temporary > tree is already what worried me about prepare_wheel_input_files, and > now it seems to be spreading :-). I don't know what additional coupling you're seeing: the only coupling is that building a wheel directly from a VCS source tree and first exporting an sdist and then building a wheel from that *must give the same result* (modulo any problems related to reproducible builds, or the lack thereof). If a project can't meet that expectation, then their sdist generation is broken, since it clearly isn't exporting the necessary content to actually build the project properly. If you weren't aware of that inherent coupling, and are surprised by the fact that it's a constraint on publishing useful sdists, then I'd consider it a good thing that PEP 517 is now making it more explicit. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Wed Jun 28 00:36:36 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Jun 2017 21:36:36 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On Tue, Jun 27, 2017 at 9:29 PM, Nick Coghlan wrote: > Sorry, I also meant to reply to this part before hitting send. > > On 28 June 2017 at 13:43, Nathaniel Smith wrote: >> I would say that there's only one required path, which is in-place. >> (But the tree that they do the in-place build in might have a more or >> less complicated history). It worries me that you seem to think that >> build_sdist and build_wheel should be coupled like this... this >> proliferation of cases and the reification of "out of tree builds" as >> a special thing that's different from an in-tree build in a temporary >> tree is already what worried me about prepare_wheel_input_files, and >> now it seems to be spreading :-). > > I don't know what additional coupling you're seeing: the only coupling > is that building a wheel directly from a VCS source tree and first > exporting an sdist and then building a wheel from that *must give the > same result* (modulo any problems related to reproducible builds, or > the lack thereof). If a project can't meet that expectation, then > their sdist generation is broken, since it clearly isn't exporting the > necessary content to actually build the project properly. > > If you weren't aware of that inherent coupling, and are surprised by > the fact that it's a constraint on publishing useful sdists, then I'd > consider it a good thing that PEP 517 is now making it more explicit. What I'm concerned about is that in your formulation, you're running different code to build a wheel from an sdist that we just generated, versus an sdist that we downloaded from PyPI. By making it possible to distinguish these cases, you're facilitating projects that break that inherent coupling (either accidentally or on purpose). I think that in these two cases, all the hooks used in the wheel-building phase should be run in exactly the same way, i.e., inside the unpacked sdist, with no reference to the original working tree that that sdist was generated from. Since the PyPI case can't access the original working tree, the local out-of-place-build case generate the sdist and then after that pretend it can't access the original working tree either. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Wed Jun 28 00:37:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jun 2017 14:37:20 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 28 June 2017 at 14:25, Nathaniel Smith wrote: > pyproject.toml is just an example anyway; get_build_wheel_requires > runs arbitrary code, and that code could potentially return different > results in different trees. No, I don't want to give backends permission to work that way. If a frontend runs get_build_wheel_requires() against the VCS tree, then that should also be sufficient to set up an environment to build an unpacked sdist - the backend isn't permitted to require that both hooks be run against the exact same tree. (It's fine if it returns some extra dependencies that may not technically be needed for a from-sdist build, though) To put it another way, if the "VCS-tree-for-build-requirements, unpacked-sdist-tree for build" combination doesn't work, that's a bug in the backend, not a bug in the frontend. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jun 28 00:40:51 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Jun 2017 14:40:51 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 28 June 2017 at 14:36, Nathaniel Smith wrote: > I think that in these two cases, all the hooks used in the > wheel-building phase should be run in exactly the same way, i.e., > inside the unpacked sdist, with no reference to the original working > tree that that sdist was generated from. Since the PyPI case can't > access the original working tree, the local out-of-place-build case > generate the sdist and then after that pretend it can't access the > original working tree either. That's up to the frontend - they're certainly free to do things that way, for the reasons you give. However, backends aren't allowed to assume that *all* frontends will work that way - they need to ensure that the "query-VCS-tree, build-from-sdist" combination works sensibly (and most of them will, through the simple expedient of not reporting any dynamic build dependencies in the first place). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Wed Jun 28 05:24:43 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Jun 2017 10:24:43 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: On 28 June 2017 at 05:29, Nick Coghlan wrote: > On 28 June 2017 at 13:43, Nathaniel Smith wrote: >> I would say that there's only one required path, which is in-place. >> (But the tree that they do the in-place build in might have a more or >> less complicated history). It worries me that you seem to think that >> build_sdist and build_wheel should be coupled like this... this >> proliferation of cases and the reification of "out of tree builds" as >> a special thing that's different from an in-tree build in a temporary >> tree is already what worried me about prepare_wheel_input_files, and >> now it seems to be spreading :-). > > I don't know what additional coupling you're seeing: the only coupling > is that building a wheel directly from a VCS source tree and first > exporting an sdist and then building a wheel from that *must give the > same result* (modulo any problems related to reproducible builds, or > the lack thereof). If a project can't meet that expectation, then > their sdist generation is broken, since it clearly isn't exporting the > necessary content to actually build the project properly. > > If you weren't aware of that inherent coupling, and are surprised by > the fact that it's a constraint on publishing useful sdists, then I'd > consider it a good thing that PEP 517 is now making it more explicit. Note that the whole concept of "out of tree builds" is not something theoretical - it comes directly from pip's requirement to be able to do clean builds. So saying that tree->sdist->unpacked sdist is "just another in-place build of a different tree" is missing the point, which is that we need a mechanism to produce an effectively identical, in the sense that it must result in the same wheel, copy of a tree - and because we chose the sdist as the route for that, there are constraints on the contents of a sdist. I view those constraints as pretty much self evident, as I would be extremely surprised if I had a source tree of *any* form for a project, and I build a sdist from it and gave it to a friend, and when we both built wheels and installed them, we had different installed copies of the project. The alternative would be to have a separate "produce a copy of this tree that builds the same wheel" hook. And I thought we already went round that loop, which is how we ended up deciding that the sdist was our approach... Paul From robertc at robertcollins.net Wed Jun 28 06:57:49 2017 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 28 Jun 2017 22:57:49 +1200 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: Re: returning magic strings to indicate exceptions. Please no. Just no. Have any pep build host add a small library to Python path with any symbols we want to define. Hardly an implementation hurdle. Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Jun 28 10:03:12 2017 From: donald at stufft.io (Donald Stufft) Date: Wed, 28 Jun 2017 10:03:12 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> Message-ID: <032FF309-1003-4ECF-BE75-F18841508DC4@stufft.io> > On Jun 22, 2017, at 12:32 PM, Donald Stufft wrote: > > For TestPyPI the change to disable uploads to legacy will be made in the next > couple of days, likely this weekend. This is done now. Tomorrow we?ll deploy it to real PyPI for the day, but it will then be reverted until July 3rd. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jun 28 13:07:14 2017 From: dholth at gmail.com (Daniel Holth) Date: Wed, 28 Jun 2017 17:07:14 +0000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: Is there a prototype implementation of pep 517 yet? On Wed, Jun 28, 2017, 06:58 Robert Collins wrote: > > > Re: returning magic strings to indicate exceptions. Please no. Just no. > Have any pep build host add a small library to Python path with any symbols > we want to define. Hardly an implementation hurdle. > > Rob > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Jun 28 13:30:19 2017 From: brett at python.org (Brett Cannon) Date: Wed, 28 Jun 2017 17:30:19 +0000 Subject: [Distutils] Provide separate development and documentation URLs in PyPI metadata? In-Reply-To: <1498582650.2206361.1023063584.60D9190F@webmail.messagingengine.com> References: <1498326511.3929161.1020088472.2D2AB7FF@webmail.messagingengine.com> <20170627115843.27a95390@subdivisions.wooz.org> <1498582650.2206361.1023063584.60D9190F@webmail.messagingengine.com> Message-ID: On Tue, 27 Jun 2017 at 10:06 Thomas Kluyver wrote: > On Tue, Jun 27, 2017, at 04:58 PM, Barry Warsaw wrote: > > On Jun 25, 2017, at 11:33 AM, Nick Coghlan wrote: > > >I'd favour "Participate" over any variant of "Contribute", as without > > >context, "Contribute" makes me think of financial support in the > > >crowdfunding/tip jar sense. > > > > "Participate" may mean two different things. > > > > * Here's the development home, with a repo and issue tracker, > > contributions > > welcome! > > > > * Here's the mailing list or other forum where we discuss the future of > > Guido's Magical Mystery Time Machine. > > Perhaps this points to labelling URLs with nouns rather than verbs: > things like 'mailing list', 'source code' or 'issue tracker' seem less > ambiguous than 'participate' or 'contribute'. > I agree with Thomas on this one. Seeing a link that says "Participate" just feels like it's missing an exclamation point and subtext saying I could earn $2,000/week from it. ;) Since this has turned into bikeshedding over names when we have a general metadata solution, I've filed https://github.com/takluyver/flit/issues/116 and consider my question answered. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jun 28 15:15:16 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 28 Jun 2017 12:15:16 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: I don't think anyone has proposed anything involving magic strings? On Jun 28, 2017 3:58 AM, "Robert Collins" wrote: > > > Re: returning magic strings to indicate exceptions. Please no. Just no. > Have any pep build host add a small library to Python path with any symbols > we want to define. Hardly an implementation hurdle. > > Rob > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Wed Jun 28 15:52:38 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Wed, 28 Jun 2017 20:52:38 +0100 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> Message-ID: <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote: > Is there a prototype implementation of pep 517 yet? - Flit has a PR with a prototype backend implementation, though it's not up to date with all the changes the PEP has undergone. I'll update it when we've agreed on a spec - it's still a fast moving target right now.- I have a prototype module frontends could use to call hooks here: https://github.com/takluyver/pep517 . It's mostly up to date, except for the issue with using sdist as a fallback for copying files for a wheel. Re: magic strings - like Nathaniel said, I haven't noticed them as part of any proposal so far. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jun 28 21:42:52 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Jun 2017 01:42:52 +0000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> Message-ID: I was able to implement PEP 517 build_wheel and build_sdist for enscons (on bitbucket), and a click cli calling any backend mentioned in pyproject.toml. Pretty simple. Not sure what to do with the config dictionary. SCons is not designed to be called twice in the same process with different arguments. It would be easier for me to know that pip would only invoke one PEP 517 function per subprocess, or to know all target directories in advance. Otherwise the enscons.api subprocess has to invoke SCons in another subprocess. SCons also builds all targets (wheel and sdist in same invocation) by default. "Alakazam!" On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver wrote: > On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote: > > Is there a prototype implementation of pep 517 yet? > > > - Flit has a PR with a prototype backend implementation, though it's not > up to date with all the changes the PEP has undergone. I'll update it when > we've agreed on a spec - it's still a fast moving target right now. > - I have a prototype module frontends could use to call hooks here: > https://github.com/takluyver/pep517 . It's mostly up to date, except for > the issue with using sdist as a fallback for copying files for a wheel. > > Re: magic strings - like Nathaniel said, I haven't noticed them as part of > any proposal so far. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jun 28 22:25:45 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Jun 2017 12:25:45 +1000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> Message-ID: On 29 June 2017 at 11:42, Daniel Holth wrote: > I was able to implement PEP 517 build_wheel and build_sdist for enscons (on > bitbucket), and a click cli calling any backend mentioned in pyproject.toml. > Pretty simple. Not sure what to do with the config dictionary. That's designed to allow frontends to tunnel custom settings from the user through to the backend, so if you don't support any config settings yet, it probably makes sense to just error out with "Unknown config setting" if the dictionary is non-empty. Alternatively, you could silently ignore it. > SCons is not designed to be called twice in the same process with different > arguments. It would be easier for me to know that pip would only invoke one > PEP 517 function per subprocess, or to know all target directories in > advance. Otherwise the enscons.api subprocess has to invoke SCons in another > subprocess. SCons also builds all targets (wheel and sdist in same > invocation) by default. That's an interesting point of tension between supporting imperative frontends like pip (which have a strong opinion about the order in which steps should be executed) and declarative build systems like Scons (which are more "produce the defined artifact set"). However, I think the way to go for now would be to say: - each hook call should be made in a fresh subprocess (so backends don't need to use a second subprocess "just to be on the safe side") - in a *future* API extension, we may add an optional "build_all" hook (whereby the backend produced both an sdist and all wheels it knew how to create for the current platform), but anything like that will be out of scope for the initial PEP 517 API Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dholth at gmail.com Wed Jun 28 22:28:37 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Jun 2017 02:28:37 +0000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> Message-ID: That would fit with setup.py, which also likes to sys.exit() after one command. On Wed, Jun 28, 2017, 22:25 Nick Coghlan wrote: > On 29 June 2017 at 11:42, Daniel Holth wrote: > > I was able to implement PEP 517 build_wheel and build_sdist for enscons > (on > > bitbucket), and a click cli calling any backend mentioned in > pyproject.toml. > > Pretty simple. Not sure what to do with the config dictionary. > > That's designed to allow frontends to tunnel custom settings from the > user through to the backend, so if you don't support any config > settings yet, it probably makes sense to just error out with "Unknown > config setting" if the dictionary is non-empty. Alternatively, you > could silently ignore it. > > > SCons is not designed to be called twice in the same process with > different > > arguments. It would be easier for me to know that pip would only invoke > one > > PEP 517 function per subprocess, or to know all target directories in > > advance. Otherwise the enscons.api subprocess has to invoke SCons in > another > > subprocess. SCons also builds all targets (wheel and sdist in same > > invocation) by default. > > That's an interesting point of tension between supporting imperative > frontends like pip (which have a strong opinion about the order in > which steps should be executed) and declarative build systems like > Scons (which are more "produce the defined artifact set"). > > However, I think the way to go for now would be to say: > > - each hook call should be made in a fresh subprocess (so backends > don't need to use a second subprocess "just to be on the safe side") > - in a *future* API extension, we may add an optional "build_all" hook > (whereby the backend produced both an sdist and all wheels it knew how > to create for the current platform), but anything like that will be > out of scope for the initial PEP 517 API > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jun 28 23:07:21 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Jun 2017 03:07:21 +0000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> Message-ID: The other thing I noticed is that build-system requires has no hyphen but build-backend does. On Wed, Jun 28, 2017, 22:28 Daniel Holth wrote: > That would fit with setup.py, which also likes to sys.exit() after one > command. > > On Wed, Jun 28, 2017, 22:25 Nick Coghlan wrote: > >> On 29 June 2017 at 11:42, Daniel Holth wrote: >> > I was able to implement PEP 517 build_wheel and build_sdist for enscons >> (on >> > bitbucket), and a click cli calling any backend mentioned in >> pyproject.toml. >> > Pretty simple. Not sure what to do with the config dictionary. >> >> That's designed to allow frontends to tunnel custom settings from the >> user through to the backend, so if you don't support any config >> settings yet, it probably makes sense to just error out with "Unknown >> config setting" if the dictionary is non-empty. Alternatively, you >> could silently ignore it. >> >> > SCons is not designed to be called twice in the same process with >> different >> > arguments. It would be easier for me to know that pip would only invoke >> one >> > PEP 517 function per subprocess, or to know all target directories in >> > advance. Otherwise the enscons.api subprocess has to invoke SCons in >> another >> > subprocess. SCons also builds all targets (wheel and sdist in same >> > invocation) by default. >> >> That's an interesting point of tension between supporting imperative >> frontends like pip (which have a strong opinion about the order in >> which steps should be executed) and declarative build systems like >> Scons (which are more "produce the defined artifact set"). >> >> However, I think the way to go for now would be to say: >> >> - each hook call should be made in a fresh subprocess (so backends >> don't need to use a second subprocess "just to be on the safe side") >> - in a *future* API extension, we may add an optional "build_all" hook >> (whereby the backend produced both an sdist and all wheels it knew how >> to create for the current platform), but anything like that will be >> out of scope for the initial PEP 517 API >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jun 29 12:09:24 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Jun 2017 12:09:24 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <032FF309-1003-4ECF-BE75-F18841508DC4@stufft.io> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <032FF309-1003-4ECF-BE75-F18841508DC4@stufft.io> Message-ID: <929D259D-957E-4163-A08E-E0756009A1D1@stufft.io> > On Jun 28, 2017, at 10:03 AM, Donald Stufft wrote: > > >> On Jun 22, 2017, at 12:32 PM, Donald Stufft > wrote: >> >> For TestPyPI the change to disable uploads to legacy will be made in the next >> couple of days, likely this weekend. > > > This is done now. Tomorrow we?ll deploy it to real PyPI for the day, but it will then be reverted until July 3rd. > This is done on PyPI now, tomorrow I?ll revert it until July 3rd. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Jun 29 15:26:16 2017 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 29 Jun 2017 21:26:16 +0200 Subject: [Distutils] PEP 518 build dependencies reinstalled unconditionally? Message-ID: <86dc39df-9bf9-e318-7b13-cdb52d3779ef@iki.fi> Hi, PEP 518 introduces build requirements that pip installs before calling setup.py. As it currently appears to be, pip installs them to a temporary environment used during the build, with "pip install --ignore-installed ...", so that packages given in pyproject.toml are reinstalled temporarily even if they are already available in the system. Better behavior could be to reinstall build-requirements only if the requirement is not already satisfied by system packages. Pip issue: https://github.com/pypa/pip/issues/4582 Numpy is a common build requirement for several packages providing C extensions that deal with numpy arrays. The current preinstallation behavior causes the following problems: 1) Numpy C extension ABI is backward compatible, but not forward compatible. As a consequence, C extensions built using a newer version of Numpy than installed on the system may simply segfault. (If there's a way to mark the version requirement in the built wheel, the end result is incompatible with the system numpy version.) This can easily happen if unsuspecting people put 'requires=numpy' to pyproject.toml. 2) It wastes time and resources unnecessarily: numpy rebuild can be 10 min per pop, plus downloading stuff from internet. -- Pauli Virtanen From bill at baddogconsulting.com Thu Jun 29 17:54:16 2017 From: bill at baddogconsulting.com (Bill Deegan) Date: Thu, 29 Jun 2017 14:54:16 -0700 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> Message-ID: Daniel, Perhaps the better way to go about it than to call scons N times is to have N variant dirs? where there's a variant dir for each build type you want, and then youd call scons build_wheel which is aliased Alias('build_wheel','variant_dir for building wheel path') Maybe bring this up on scons users mailing list and we'll see if it can be resolved? On Wed, Jun 28, 2017 at 6:42 PM, Daniel Holth wrote: > I was able to implement PEP 517 build_wheel and build_sdist for enscons > (on bitbucket), and a click cli calling any backend mentioned in > pyproject.toml. Pretty simple. Not sure what to do with the config > dictionary. > > SCons is not designed to be called twice in the same process with > different arguments. It would be easier for me to know that pip would only > invoke one PEP 517 function per subprocess, or to know all target > directories in advance. Otherwise the enscons.api subprocess has to invoke > SCons in another subprocess. SCons also builds all targets (wheel and sdist > in same invocation) by default. > > "Alakazam!" > > On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver > wrote: > >> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote: >> >> Is there a prototype implementation of pep 517 yet? >> >> >> - Flit has a PR with a prototype backend implementation, though it's not >> up to date with all the changes the PEP has undergone. I'll update it when >> we've agreed on a spec - it's still a fast moving target right now. >> - I have a prototype module frontends could use to call hooks here: >> https://github.com/takluyver/pep517 . It's mostly up to date, except for >> the issue with using sdist as a fallback for copying files for a wheel. >> >> Re: magic strings - like Nathaniel said, I haven't noticed them as part >> of any proposal so far. >> >> Thomas >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Jun 29 17:59:33 2017 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Jun 2017 21:59:33 +0000 Subject: [Distutils] Finishing up PEP 517 In-Reply-To: References: <1497532370.3900461.1010360176.0BA029B7@webmail.messagingengine.com> <1497604124.3745126.1011381184.16F82982@webmail.messagingengine.com> <08EBE708-32FF-4E43-B5D1-4ED69ABD9421@stufft.io> <9B20707C-3B19-4AB8-93AE-55CBD6C38036@stufft.io> <1498679558.2900173.1024492416.51921D32@webmail.messagingengine.com> Message-ID: Sounds good, I'll bring it up there. On Thu, Jun 29, 2017, 17:54 Bill Deegan wrote: > Daniel, > > Perhaps the better way to go about it than to call scons N times is to > have N variant dirs? where there's a variant dir for each build type you > want, and then youd call > > scons build_wheel > which is aliased Alias('build_wheel','variant_dir for building wheel path') > > Maybe bring this up on scons users mailing list and we'll see if it can be > resolved? > > On Wed, Jun 28, 2017 at 6:42 PM, Daniel Holth wrote: > >> I was able to implement PEP 517 build_wheel and build_sdist for enscons >> (on bitbucket), and a click cli calling any backend mentioned in >> pyproject.toml. Pretty simple. Not sure what to do with the config >> dictionary. >> >> SCons is not designed to be called twice in the same process with >> different arguments. It would be easier for me to know that pip would only >> invoke one PEP 517 function per subprocess, or to know all target >> directories in advance. Otherwise the enscons.api subprocess has to invoke >> SCons in another subprocess. SCons also builds all targets (wheel and sdist >> in same invocation) by default. >> >> "Alakazam!" >> >> On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver >> wrote: >> >>> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote: >>> >>> Is there a prototype implementation of pep 517 yet? >>> >>> >>> - Flit has a PR with a prototype backend implementation, though it's not >>> up to date with all the changes the PEP has undergone. I'll update it when >>> we've agreed on a spec - it's still a fast moving target right now. >>> - I have a prototype module frontends could use to call hooks here: >>> https://github.com/takluyver/pep517 . It's mostly up to date, except >>> for the issue with using sdist as a fallback for copying files for a wheel. >>> >>> Re: magic strings - like Nathaniel said, I haven't noticed them as part >>> of any proposal so far. >>> >>> Thomas >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jun 30 10:48:19 2017 From: donald at stufft.io (Donald Stufft) Date: Fri, 30 Jun 2017 10:48:19 -0400 Subject: [Distutils] ANNOUNCE: Sunsetting of uploading to the legacy PyPI/TestPyPI In-Reply-To: <929D259D-957E-4163-A08E-E0756009A1D1@stufft.io> References: <46E9C5AD-A689-443B-9CE5-19922780D5D1@stufft.io> <032FF309-1003-4ECF-BE75-F18841508DC4@stufft.io> <929D259D-957E-4163-A08E-E0756009A1D1@stufft.io> Message-ID: <90B7ABC1-D383-4483-9090-52CF53DD892B@stufft.io> > On Jun 29, 2017, at 12:09 PM, Donald Stufft wrote: > > This is done on PyPI now, tomorrow I?ll revert it until July 3rd. Upload API is re-enabled until July 3rd. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: