From p at 2015.forums.dobrogost.net Thu Oct 1 16:41:22 2015 From: p at 2015.forums.dobrogost.net (Piotr Dobrogost) Date: Thu, 1 Oct 2015 16:41:22 +0200 Subject: [Distutils] =?utf-8?q?Unicode_in_the_=22long=5Fdescription=22_?= =?utf-8?q?=E2=80=93_is_this_possible_now=3F?= Message-ID: Hi! I'd like to know if anything changed since 2011 with regard to using unicode as a value of "long_description" argument to setuptools/distutils setup function? The problem was described by Carl Meyer in 2011 here https://github.com/pypa/virtualenv/issues/201#issuecomment-3145690 ? "The initial error is on reading from the file containing non-ASCII (docs/news.txt in our case). This part is easily fixable by switching from open(...) to codecs.open(..., encoding="UTF-8"). But then you get an encoding error on the output side. The usual way to fix this would be to explicitly encode the string to bytes in the desired encoding before output, e.g. long_description = long_description.encode("UTF-8"). We can't do that for long-description in setup.py, though, because distutils rfc822-escapes it internally, and that requires string-only methods (split). So long-description must be a string (unicode), it can't be encoded bytes. Attempts to change the default encoding of sys.stdout are pointless, as in the installation case long-description isn't written to stdout, it's written to a file." Btw, in what situation does long_description get written to sys.stdout? Regards, Piotr Dobrogost From guettliml at thomas-guettler.de Thu Oct 1 19:52:04 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 1 Oct 2015 19:52:04 +0200 Subject: [Distutils] README.rst vs DESCRIPTION.rst In-Reply-To: <5607C303.1040201@thomas-guettler.de> References: <5607C303.1040201@thomas-guettler.de> Message-ID: <560D72C4.5060004@thomas-guettler.de> one week has passed. I check the feedback. Most people want one README.rst. I see the file was already updated: https://github.com/pypa/sampleproject/commit/e7de7861963274595c5b3b0d6942e35ce5a733bc Issue solved. Thank you Carl Meyer! Regards, Thomas G?ttler Am 27.09.2015 um 12:20 schrieb Thomas G?ttler: > Zen of Python: > > There should be one-- and preferably only one --obvious way to do it. > > I would like to find a default for the description file of a python package. > > The sampleproject uses DESCRIPTION.rst > https://github.com/pypa/sampleproject/blob/master/setup.py > > But I guess it is more common to use README.rst. > For example django uses this file name. > > Any good reason to **not** use README.rst but a different > name like DESCRIPTION.rst? > > Of course anyone can use the name he wants. I just want an agreement > for the name to make life easier for newcomers. > > I will check this mail thread in a week or two and write > a pull request to https://github.com/pypa/sampleproject/blob/master/setup.py > if there is an agreement. > > If an agreement was found, which other documents should be updated? > > Regards, > Thomas G?ttler > > > > > -- http://www.thomas-guettler.de/ From njs at pobox.com Fri Oct 2 06:53:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 1 Oct 2015 21:53:39 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils Message-ID: Hi all, We realized that actually as far as we could tell, it wouldn't be that hard at this point to clean up how sdists work so that it would be possible to migrate away from distutils. So we wrote up a little draft proposal. The main question is, does this approach seem sound? -n --- PEP: ?? Title: Standard interface for interacting with source trees and source distributions Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith Thomas Kluyver Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: Discussions-To: Abstract ======== Distutils delenda est. Extended abstract ================= While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like autoconfiguration and usable build-time dependency declaration, (b) extending them is quirky, complicated, and fragile, (c) you are forced to use them anyway, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). We propose to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then the more the merrier. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit `_ or `bento `_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. Synopsis and rationale ====================== To limit the scope of our design, we adopt several principles. First, we distinguish between a *source tree* (e.g., a VCS checkout) and a *source distribution* (e.g., an official snapshot release like ``lxml-3.4.4.zip``). There isn't a whole lot that *source trees* can be assumed to have in common. About all you know is that they can -- via some more or less Rube-Goldbergian process -- produce one or more binary distributions. In particular, you *cannot* tell via simple static inspection: - What version number will be attached to the resulting packages (e.g. it might be determined programmatically by consulting VCS metadata -- I have here a build of numpy version "1.11.0.dev0+4a9ad17") - What build- or run-time dependencies are required (e.g. these may depend on arbitrarily complex configuration settings that are determined via a mix of manual settings and auto-probing) - Or even how many distinct binary distributions will be produced (e.g. a source distribution may always produce wheel A, but only produce wheel B when built on Unix-like systems). Therefore, when dealing with source trees, our goal is just to provide a standard UX for the core operations that are commonly performed on other people's packages; anything fancier and more developer-centric we leave at the discretion of individual package developers. So our source trees just provide some simple hooks to let a tool like ``pip``: - query for build dependencies - run a build, producing wheels as output - set up the current source tree so that it can be placed on ``sys.path`` in "develop mode" and that's it. We teach users that the standard way to install a package from a VCS checkout is now ``pip install .`` instead of ``python setup.py install``. (This is already a good idea anyway -- e.g., pip can do reliable uninstall / upgrades.) Next, we note that pretty much all the operations that you might want to perform on a *source distribution* are also operations that you might want to perform on a source tree, and via the same UX. The only thing you do with source distributions that you don't do with source trees is, well, distribute them. There's all kind of metadata you could imagine including in a source distribution, but each piece of metadata puts an increased burden on source distribution generation tools, and most operations will still have to work without this metadata. So we only include extra metadata in source distributions if it helps solve specific problems that are unique to distribution. If you want wheel-style metadata, get a wheel and look at it -- they're great and getting better. Therefore, our source distributions are basically just source trees + a mechanism for signing. Finally: we explicitly do *not* have any concept of "depending on a source distribution". As in other systems like Debian, dependencies are always phrased in terms of binary distributions (wheels), and when a user runs something like ``pip install ``, then the long-run plan is that and all its transitive dependencies should be available as wheels in a package index. But this is not yet realistic, so as a transitional / backwards-compatibility measure, we provide a simple mechanism for ``pip install `` to handle cases where is provided only as a source distribution. Source trees ============ We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code of ``distutils``, ``setuptools``, ``pip``, and other tools. A version 1-or-greater format source tree can be identified by the presence of a file ``_pypackage/_pypackage.cfg``. If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. This is necessary because we anticipate that version 1+ source trees may want to contain a ``setup.py`` file for backwards compatibility, e.g.:: #!/usr/bin/env python import sys print("Don't call setup.py directly!") print("Use 'pip install .' instead!") print("(You might have to upgrade pip first.)") sys.exit(1) In the current version of the specification, the one file ``_pypackage/_pypackage.cfg`` is where pretty much all the action is (though see below). The motivation for putting it into a subdirectory is that: - the way of all standards is that cruft accumulates over time, so this way we pre-emptively have a place to put it, - real-world projects often accumulate build system cruft as well, so we might as well provide one obvious place to put it too. Of course this then creates the possibility of collisions between standard files and user files, and trying to teach arbitrary users not to scatter files around willy-nilly never works, so we adopt the convention that names starting with an underscore are reserved for official use, and non-underscored names are available for idiosyncratic use by individual projects. The alternative would be to simply place the main configuration file at the top-level, create the subdirectory only when specifically needed (most trees won't need it), and let users worry about finding their own place for their cruft. Not sure which is the best approach. Plus we can have a nice bikeshed about the names in general (FIXME). _pypackage.cfg -------------- The ``_pypackage.cfg`` file contains various settings. Another good bike-shed topic is which file format to use for storing these (FIXME), but for purposes of this draft I'll write examples using `toml `_, because you'll instantly be able to understand the semantics, it has similar expressivity to JSON while being more human-friendly (e.g., it supports comments and multi-line strings), it's better-specified than ConfigParser, and it's much simpler than YAML. Rust's package manager uses toml for similar purposes. Here's an example ``_pypackage/_pypackage.cfg``:: # Version of the "pypackage format" that this file uses. # Optional. If not present then 1 is assumed. # All version changes indicate incompatible changes; backwards # compatible changes are indicated by just having extra stuff in # the file. version = 1 [build] # An inline requirements file. Optional. # (FIXME: I guess this means we need a spec for requirements files?) requirements = """ mybuildtool >= 2.1 special_windows_tool ; sys_platform == "win32" """ # The path to an out-of-line requirements file. Optional. requirements-file = "build-requirements.txt" # A hook that will be called to query build requirements. Optional. requirements-dynamic = "mybuildtool:get_requirements" # A hook that will be called to build wheels. Required. build-wheels = "mybuildtool:do_build" # A hook that will be called to do an in-place build (see below). # Optional. build-in-place = "mybuildtool:do_inplace_build" # The "x" namespace is reserved for third-party extensions. # To use x.foo you should own the name "foo" on pypi. [x.mybuildtool] spam = ["spam", "spam", "spam"] All paths are relative to the ``_pypackage/`` directory (so e.g. the build.requirements-file value above refers to a file named ``_pypackage/build-requirements.txt``). A *hook* is a Python object that is looked up using the same rules as traditional setuptools entry_points: a dotted module name, followed by a colon, followed by a dotted name that is looked up within that module. *Running a hook* means: first, find or create a python interpreter which is executing in the current venv, whose working directory is set to the ``_pypackage/`` directory, and which has the ``_pypackage/`` directory on ``sys.path``. Then, inside this interpreter, look up the hook object, and call it, with arguments as specified below. A build command like ``pip wheel `` performs the following steps: 1) Validate the ``_pypackage.cfg`` version number. 2) Create an empty virtualenv / venv, that matches the environment that the installer is targeting (e.g. if you want wheels for CPython 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows venv). 3) If the build.requirements key is present, then in this venv run the equivalent of ``pip install -r ``, using whatever index settings are currently in effect. 4) If the build.requirements-file key is present, then in this venv run the equivalent of ``pip install -r ``, using whatever index settings are currently in effect. 5) If the build.requirements-dynamic key is present, then in this venv run the hook with no arguments, capture its stdout, and pipe it into ``pip install -r -``, using whatever index settings are currently in effect. If the hook raises an exception, then abort the build with an error. Note: because these steps are performed in sequence, the build.requirements-dynamic hook is allowed to use packages that are listed in build.requirements or build.requirements-file. 6) In this venv, run the build.build-wheels hook. This should be a Python function which takes one argument. This argument is an arbitrary dictionary intended to contain user-specified configuration, specified via some install-tool-specific mechanism. The intention is that tools like ``pip`` should provide some way for users to specify key/value settings that will be passed in here, analogous to the legacy ``--install-option`` and ``--global-option`` arguments. To make it easier for packages to transition from version 0 to version 1 sdists, we suggest that ``pip`` and other tools that have such existing option-setting interfaces SHOULD map them to entries in this dictionary when -- e.g.:: pip --global-option=a --install-option=b --install-option=c could produce a dict like:: {"--global-option": ["a"], "--install-option": ["b", "c"]} The hook's return value is a list of pathnames relative to the scratch directory. Each entry names a wheel file created by this build. Errors are signaled by raising an exception. When performing an in-place build (e.g. for ``pip install -e .``), then the same steps are followed, except that instead of the build.build-wheels hook, we call the build.build-in-place hook, and instead of returning a list of wheel files, it returns the name of a directory that should be placed onto ``sys.path`` (usually this will be the source tree itself, but may not be, e.g. if a build system wants to enforce a rule where the source is always kept pristine then it could symlink the .py files into a build directory, place the extension modules and dist-info there, and return that). This directory must contain importable versions of the code in the source tree, along with appropriate .dist-info directories. (FIXME: in-place builds are useful but intrinsically kinda broken -- e.g. extensions / source / metadata can all easily get out of sync -- so while I think this paragraph provides a reasonable hack that preserves current functionality, maybe we should defer specifying them to until after we've thought through the issues more?) When working with source trees, build tools like ``pip`` are encouraged to cache and re-use virtualenvs for performance. Other contents of _pypackage/ ----------------------------- _RECORD, _RECORD.jws, _RECORD.p7s: see below. _x//: reserved for use by tools (e.g. _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) Source distributions ==================== A *source distribution* is a file in a well-known archive format such as zip or tar.gz, which contains a single directory, and this directory is a source tree (in the sense defined in the previous section). The ``_pypackage/`` directory in a source distribution SHOULD also contain a _RECORD file, as defined in PEP 427, and MAY also contain _RECORD.jws and/or _RECORD.p7s signature files. For official releases, source distributions SHOULD be named as ``-.``, and the directory they contain SHOULD be named ``-``, and building this source tree SHOULD produce a wheel named ``--.whl`` (though it may produce other wheels as well). (FIXME: maybe we should add that if you want your sdist on PyPI then you MUST include a proper _RECORD file and use the proper naming convention?) Integration tools like ``pip`` SHOULD take advantage of this convention by applying the following heuristic: when seeking a package , if no appropriate wheel can be found, but an sdist named -. is found, then: 1) build the sdist 2) add the resulting wheels to the package search space 3) retry the original operation This handles a variety of simple and complex cases -- for example, if we need a package 'foo', and we find foo-1.0.zip which builds foo.whl and bar.whl, and foo.whl depends on bar.whl, then everything will work out. There remain other cases that are not handled, e.g. if we start out searching for bar.whl we will never discover foo-1.0.zip. We take the perspective that this is nonetheless sufficient for a transitional heuristic, and anyone who runs into this problem should just upload wheels already. If this turns out to be inadequate in practice, then it will be addressed by future extensions. Examples ======== **Example 1:** While we assume that installation tools will have to continue supporting version 0 sdists for the indefinite future, it's a useful check to make sure that our new format can continue to support packages using distutils / setuptools as their build system. We assume that a future version ``pip`` will take its existing knowledge of distutils internals and expose them as the appropriate hooks, and then existing distutils / setuptools packages can be ported forward by using the following ``_pypackage/_pypackage.cfg``:: [build] requirements = """ pip >= whatever wheel """ # Applies monkeypatches, then does 'setup.py dist_info' and # extracts the setup_requires requirements-dynamic = "pip.pypackage_hooks:setup_requirements" # Applies monkeypatches, then does 'setup.py wheel' build-wheels = "pip.pypackage_hooks:build_wheels" # Applies monkeypatches, then does: # setup.py dist_info && setup.py build_ext -i build-in-place = "pip.pypackage_hooks:build_in_place" This is also useful for any other installation tools that may want to support version 0 sdists without having to implement bug-for-bug compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is present, they can use this as a default. **Example 2:** For packages using numpy.distutils. This is identical to the distutils / setuptools example above, except that numpy is moved into the list of static build requirements. Right now, most projects using numpy.distutils don't bother trying to declare this dependency, and instead simply error out if numpy is not already installed. This is because currently the only way to declare a build dependency is via the ``setup_requires`` argument to the ``setup`` function, and in this case the ``setup`` function is ``numpy.distutils.setup``, which... obviously doesn't work very well. Drop this ``_pypackage.cfg`` into an existing project like this and it will become robustly pip-installable with no further changes:: [build] requirements = """ numpy pip >= whatever wheel """ requirements-dynamic = "pip.pypackage_hooks:setup_requirements" build-wheels = "pip.pypackage_hooks:build_wheels" build-in-place = "pip.pypackage_hooks:build_in_place" **Example 3:** `flit `_ is a tool designed to make distributing simple packages simple, but it currently has no support for sdists, and for convenience includes its own installation code that's redundant with that in pip. These 4 lines of boilerplate make any flit-using source tree pip-installable, and lets flit get out of the package installation business:: [build] requirements = "flit" build-wheels = "flit.pypackage_hooks:build_wheels" build-in-place = "flit.pypackage_hooks:build_in_place" FAQ === **Why is it version 1 instead of version 2?** Because the legacy sdist format is barely a format at all, and to `remind us to keep things simple `_. **What about cross-compilation?** Standardizing an interface for cross-compilation seems premature given how complicated the configuration required can be, the lack of an existing de facto standard, and the authors of this PEP's inexperience with cross-compilation. This would be a great target for future extensions, though. In the mean time, there's no requirement that ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a project's build system -- packages that want to support cross-compilation can still do so, they'll just need to include a README explaining how to do it. **PEP 426 says that the new sdist format will support automatically creating policy-compliant .deb/.rpm packages. What happened to that?** Step 1: enhance the wheel format as necessary so that a wheel can be automatically converted into a policy-compliant .deb/.rpm package (see PEP 491). Step 2: make it possible to automatically turn sdists into wheels (this PEP). Step 3: we're done. **What about automatically running tests?** Arguably this is another thing that should be pushed off to wheel metadata instead of sdist metadata: it's good practice to include tests inside your built distribution so that end-users can test their install (and see above re: our focus here being on stuff that end-users want to do, not dedicated package developers), there are lots of packages that have to be built before they can be tested anyway (e.g. because of binary extensions), and in any case it's good practice to test against an installed version in order to make sure your install code works properly. But even if we do want this in sdist, then it's hardly urgent (e.g. there is no ``pip test`` that people will miss), so we defer that for a future extension to avoid blocking the core functionality. -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Fri Oct 2 09:12:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 00:12:19 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 1, 2015 10:45 PM, "Marcus Smith" wrote: > > Can you clarify the relationship to PEP426 metadata? > There's no standard for metadata in here other than what's required to run a build hook. Right -- the idea is that discretely installable binary packages (i.e. wheels) are a different sort of thing than a source tree / source distribution, and need different metadata. PEP 426 is then taken as a draft of the next metadata spec *for wheels*, and this is a draft "simplest thing that could possibly work" metadata spec for source trees / source distributions. The "synopsis and rationale" section provides more motivation for this approach. > Does that imply you would have each build tool enforce their own convention for where metadata is found? I think you mean: "does that imply that each build tool would have its own way of determining what metadata to attach to the wheels it generates?", and the answer to that is yes -- already right now distutils does it via kwargs passed to the setup() function, flit does it via entries in its flit.ini file (though for simplicity it might later move this to an extension section in _pypackage.cfg -- and note that this purely declarative style is only possible because flit is designed to make simple things easy and complicated things someone else's problem), and in general it might require arbitrary code to be executed. I expect that some conventions will probably get sorted out once we have experimental build systems competing with each other and figuring out what works and what doesn't, but distutils-sig has historically had poor luck with trying to design this kind of thing a priori. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 13:58:50 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 07:58:50 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > We realized that actually as far as we could tell, it wouldn't > be that > hard at this point to clean up how sdists work so that it would be > possible to migrate away from distutils. So we wrote up a little > draft > proposal. > > The main question is, does this approach seem sound? I've just read over your proposal, but I've also just woken up so I might be a little slow still! After reading what you have, I don't think that this proposal is the right way to go about improving sdists. The first thing that immediately stood out to me, is that it's recommending that downstream redistributors like Debian, Fedora, etc utilize Wheels instead of the sdist to build their packages from. However, that is not really going to fly with most (all?) of the downstream redistributors. Debian for instance has policy that requires the use of building all of it's packages from Source, not from anything else and Wheels are not a source package. While it can theoretically work for pure python packages, it quickly devolves into a mess when you factor in packages that have any C code what so ever. Overall, this feels more like a sidegrade than an upgrade. One major theme throughout of the PEP is that we're going to push to rely heavily on wheels as the primary format of installation. While that works well for things like Debian, I don't think it's going to work as wheel for us. If we were only distributing pure python packages, then yes absolutely, however given that we are not, we have to worry about ABI issues. Given that there is so many different environments that a particular package might be installed into, all with different ABIs we have to assume that installing from source is still going to be a primary path for end users to install and that we are never going to have a world where we can assume a Wheel in a repository. One of the problems with the current system, is that we have no mechanism by which to determine dependencies of a source distribution without downloading the file and executing some potentially untrusted code. This makes dependency resolution harder and much much slower than if we could read that information statically from a source distribution. This PEP doesn't offer anything in the way of solving this problem. To a similar tune, this PEP also doesn't make it possible to really get at any other metadata without executing software. This makes it pratically impossible to safely inspect an unknown or untrusted package to determine what it is and to get information about it. Right now PyPI relies on the uploading tool to send that information alongside of the file it is uploading, but honestly what it should be doing is extracting that information from within the file. This is sort of possible right now since distutils and setuptools both create a static metadata file within the source distribution, but we don't rely on that within PyPI because that information may or may not be accurate and may or may not exist. However the twine uploading tool *does* rely on that, and this PEP would break the ability for twine to upload a package without executing arbitrary code. Overall, I don't think that this really solves most of the foundational problems with the current format. Largely it feels that what it achieves is shuffling around some logic (you need to create a hook that you reference from within a .cfg file instead of creating a setuptools extension or so) but without fixing most of the problems. The largest benefit I see to switching to this right now is that it would enable us to have build time dependencies that were controlled by pip rather than installed implicitly via the execution of the setup.py. That doesn't feel like a big enough benefit to me to do a mass shakeup of what we recommend and tell people to do. Having people adjust and change and do something new requires effort, and we need something to justify that effort to other people and I don't think that this PEP has something we can really use to justify that effort. I *do* think that there is a core of some ideas here that are valuable, and in fact are similar to some ideas I've had. The main flaw I see here is that it doesn't really fix sdists, it takes a solution that would work for VCS checkouts and then reuses it for sdists. In my mind, the supported flow for package installation would be: ? ? VCS/Bare Directory -> Source Distribution -> Wheel This would (eventually) be the only path that was supported for installation but you could "enter" the path at any stage. For example, if there is a Wheel already available, then you jump right on at the end and just install that, if there is a sdist available then pip first builds it into a wheel and then installs that, etc. I think your PEP is something like what the VCS/Bare Directory to sdist tooling could look like, but I don't think it's what the sdist to wheel path should look like.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Fri Oct 2 14:38:42 2015 From: cournape at gmail.com (David Cournapeau) Date: Fri, 2 Oct 2015 13:38:42 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 12:58 PM, Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > > We realized that actually as far as we could tell, it wouldn't > > be that > > hard at this point to clean up how sdists work so that it would be > > possible to migrate away from distutils. So we wrote up a little > > draft > > proposal. > > > > The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might > be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. Debian for instance > has > policy that requires the use of building all of it's packages from Source, > not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a > mess > when you factor in packages that have any C code what so ever. > > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on > wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that > we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, > all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never > going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism > by > which to determine dependencies of a source distribution without > downloading > the file and executing some potentially untrusted code. This makes > dependency > resolution harder and much much slower than if we could read that > information > statically from a source distribution. This PEP doesn't offer anything in > the > way of solving this problem. > > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine > what > it is and to get information about it. Right now PyPI relies on the > uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from > within the > file. This is sort of possible right now since distutils and setuptools > both > create a static metadata file within the source distribution, but we don't > rely > on that within PyPI because that information may or may not be accurate > and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. > > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference > from > within a .cfg file instead of creating a setuptools extension or so) but > without fixing most of the problems. The largest benefit I see to > switching to > this right now is that it would enable us to have build time dependencies > that > were controlled by pip rather than installed implicitly via the execution > of > the setup.py. That doesn't feel like a big enough benefit to me to do a > mass > shakeup of what we recommend and tell people to do. Having people adjust > and > change and do something new requires effort, and we need something to > justify > that effort to other people and I don't think that this PEP has something > we > can really use to justify that effort. > > I *do* think that there is a core of some ideas here that are valuable, > and in > fact are similar to some ideas I've had. The main flaw I see here is that > it > doesn't really fix sdists, it takes a solution that would work for VCS > checkouts and then reuses it for sdists. In my mind, the supported flow for > package installation would be: > > VCS/Bare Directory -> Source Distribution -> Wheel > > This would (eventually) be the only path that was supported for > installation > but you could "enter" the path at any stage. For example, if there is a > Wheel > already available, then you jump right on at the end and just install > that, if > there is a sdist available then pip first builds it into a wheel and then > installs that, etc. > > I think your PEP is something like what the VCS/Bare Directory to sdist > tooling > could look like, but I don't think it's what the sdist to wheel path should > look like. > A major feature of the proposal is to allow alternative build/packaging tools. If that proposal is not acceptable in its current form, how would you envision interoperability between pip and new systems. For example, is it realistic to encode which commands and options a setup.py would need to support to be pip-installable (without the setup.py using distutils) ? David -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 15:02:53 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 09:02:53 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 8:38:45 AM, David Cournapeau (cournape at gmail.com) wrote: > > A major feature of the proposal is to allow alternative build/packaging > tools. > > If that proposal is not acceptable in its current form, how would you > envision interoperability between pip and new systems. For example, is it > realistic to encode which commands and options a setup.py would need to > support to be pip-installable (without the setup.py using distutils) ? > I think it depends if you mean in the short term or the long term and what exactly you envision as the scope of an alternative build/packaging tool. In the short term we have two real options that I can think of off the top of my head. One is as you mentioned, define the interface to setup.py that any build tool is expected to follow. However I?m not sure if that?s a great idea because you end up having a bootstrapping problem since pip won?t know that it needs X available to execute a particular setup.py (a common problem with numpy.distutils, as I know you?re aware!). We could do a minimal extension and add another defacto-ish standard of allowing pip and setuptools to process additional setup_requires like arguments from a setup.cfg to solve that problem though. The flip side to this is that since it involves new capabilities in pip/setuptools/any other installer is that it you?ll have several years until you can depend on setup.cfg based setup_requires from being able to be depended on. Another short term option is to simply say that using something that isn?t distutils/setuptools isn?t supported, but that if you want to do something else, you should rely on setuptools ability to be extended. This means you?d need a very minimal ``setup.py`` [1], but that you could then do pretty much anything you wanted, since setuptools lets you override a good bit of it?s own logic, pbr uses this to have completely static metadata [2]. This option would just work and require no changes in pip or setuptools so then you could start depending on it right away and it would just work, the downside of course is that you?re tied to extending setuptools and the APIs it provides rather than being able to more freely do what you want, but the ability to do that in setuptools is pretty extensive. Longer term, I think the answer is sdist 2.0 which has proper metadata inside of it (name, version, dependencies, etc) but which also includes a hook like this PEP has to specify the build system that should be used to build a wheel out of this source distribution. [1] Example for pbr:?https://github.com/testing-cabal/mock/blob/master/setup.py [2]?https://github.com/testing-cabal/mock/blob/master/setup.cfg ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Fri Oct 2 07:45:51 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 1 Oct 2015 22:45:51 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Can you clarify the relationship to PEP426 metadata? There's no standard for metadata in here other than what's required to run a build hook. Does that imply you would have each build tool enforce their own convention for where metadata is found? On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith wrote: > Hi all, > > We realized that actually as far as we could tell, it wouldn't be that > hard at this point to clean up how sdists work so that it would be > possible to migrate away from distutils. So we wrote up a little draft > proposal. > > The main question is, does this approach seem sound? > > -n > > --- > > PEP: ?? > Title: Standard interface for interacting with source trees > and source distributions > Version: $Revision$ > Last-Modified: $Date$ > Author: Nathaniel J. Smith > Thomas Kluyver > Status: Draft > Type: Standards-Track > Content-Type: text/x-rst > Created: 30-Sep-2015 > Post-History: > Discussions-To: > > Abstract > ======== > > Distutils delenda est. > > > Extended abstract > ================= > > While ``distutils`` / ``setuptools`` have taken us a long way, they > suffer from three serious problems: (a) they're missing important > features like autoconfiguration and usable build-time dependency > declaration, (b) extending them is quirky, complicated, and fragile, > (c) you are forced to use them anyway, because they provide the > standard interface for installing python packages expected by both > users and installation tools like ``pip``. > > Previous efforts (e.g. distutils2 or setuptools itself) have attempted > to solve problems (a) and/or (b). We propose to solve (c). > > The goal of this PEP is get distutils-sig out of the business of being > a gatekeeper for Python build systems. If you want to use distutils, > great; if you want to use something else, then the more the merrier. > The difficulty of interfacing with distutils means that there aren't > many such systems right now, but to give a sense of what we're > thinking about see `flit `_ or > `bento > `_. Fortunately, wheels have now > solved many of the hard problems here -- e.g. it's no longer necessary > that a build system also know about every possible installation > configuration -- so pretty much all we really need from a build system > is that it have some way to spit out standard-compliant wheels. > > We therefore propose a new, relatively minimal interface for > installation tools like ``pip`` to interact with package source trees > and source distributions. > > > Synopsis and rationale > ====================== > > To limit the scope of our design, we adopt several principles. > > First, we distinguish between a *source tree* (e.g., a VCS checkout) > and a *source distribution* (e.g., an official snapshot release like > ``lxml-3.4.4.zip``). > > There isn't a whole lot that *source trees* can be assumed to have in > common. About all you know is that they can -- via some more or less > Rube-Goldbergian process -- produce one or more binary distributions. > In particular, you *cannot* tell via simple static inspection: > - What version number will be attached to the resulting packages (e.g. > it might be determined programmatically by consulting VCS metadata -- > I have here a build of numpy version "1.11.0.dev0+4a9ad17") > - What build- or run-time dependencies are required (e.g. these may > depend on arbitrarily complex configuration settings that are > determined via a mix of manual settings and auto-probing) > - Or even how many distinct binary distributions will be produced > (e.g. a source distribution may always produce wheel A, but only > produce wheel B when built on Unix-like systems). > > Therefore, when dealing with source trees, our goal is just to provide > a standard UX for the core operations that are commonly performed on > other people's packages; anything fancier and more developer-centric > we leave at the discretion of individual package developers. So our > source trees just provide some simple hooks to let a tool like > ``pip``: > > - query for build dependencies > - run a build, producing wheels as output > - set up the current source tree so that it can be placed on > ``sys.path`` in "develop mode" > > and that's it. We teach users that the standard way to install a > package from a VCS checkout is now ``pip install .`` instead of > ``python setup.py install``. (This is already a good idea anyway -- > e.g., pip can do reliable uninstall / upgrades.) > > Next, we note that pretty much all the operations that you might want > to perform on a *source distribution* are also operations that you > might want to perform on a source tree, and via the same UX. The only > thing you do with source distributions that you don't do with source > trees is, well, distribute them. There's all kind of metadata you > could imagine including in a source distribution, but each piece of > metadata puts an increased burden on source distribution generation > tools, and most operations will still have to work without this > metadata. So we only include extra metadata in source distributions if > it helps solve specific problems that are unique to distribution. If > you want wheel-style metadata, get a wheel and look at it -- they're > great and getting better. > > Therefore, our source distributions are basically just source trees + > a mechanism for signing. > > Finally: we explicitly do *not* have any concept of "depending on a > source distribution". As in other systems like Debian, dependencies > are always phrased in terms of binary distributions (wheels), and when > a user runs something like ``pip install ``, then the > long-run plan is that and all its transitive dependencies > should be available as wheels in a package index. But this is not yet > realistic, so as a transitional / backwards-compatibility measure, we > provide a simple mechanism for ``pip install `` to handle > cases where is provided only as a source distribution. > > > Source trees > ============ > > We retroactively declare the legacy source tree format involving > ``setup.py`` to be "version 0". We don't try to specify it further; > its de facto specification is encoded in the source code of > ``distutils``, ``setuptools``, ``pip``, and other tools. > > A version 1-or-greater format source tree can be identified by the > presence of a file ``_pypackage/_pypackage.cfg``. > > If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, > then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. > This is necessary because we anticipate that version 1+ source trees > may want to contain a ``setup.py`` file for backwards compatibility, > e.g.:: > > #!/usr/bin/env python > import sys > print("Don't call setup.py directly!") > print("Use 'pip install .' instead!") > print("(You might have to upgrade pip first.)") > sys.exit(1) > > In the current version of the specification, the one file > ``_pypackage/_pypackage.cfg`` is where pretty much all the action is > (though see below). The motivation for putting it into a subdirectory > is that: > - the way of all standards is that cruft accumulates over time, so > this way we pre-emptively have a place to put it, > - real-world projects often accumulate build system cruft as well, so > we might as well provide one obvious place to put it too. > > Of course this then creates the possibility of collisions between > standard files and user files, and trying to teach arbitrary users not > to scatter files around willy-nilly never works, so we adopt the > convention that names starting with an underscore are reserved for > official use, and non-underscored names are available for > idiosyncratic use by individual projects. > > The alternative would be to simply place the main configuration file > at the top-level, create the subdirectory only when specifically > needed (most trees won't need it), and let users worry about finding > their own place for their cruft. Not sure which is the best approach. > Plus we can have a nice bikeshed about the names in general (FIXME). > > _pypackage.cfg > -------------- > > The ``_pypackage.cfg`` file contains various settings. Another good > bike-shed topic is which file format to use for storing these (FIXME), > but for purposes of this draft I'll write examples using `toml > `_, because you'll instantly be > able to understand the semantics, it has similar expressivity to JSON > while being more human-friendly (e.g., it supports comments and > multi-line strings), it's better-specified than ConfigParser, and it's > much simpler than YAML. Rust's package manager uses toml for similar > purposes. > > Here's an example ``_pypackage/_pypackage.cfg``:: > > # Version of the "pypackage format" that this file uses. > # Optional. If not present then 1 is assumed. > # All version changes indicate incompatible changes; backwards > # compatible changes are indicated by just having extra stuff in > # the file. > version = 1 > > [build] > # An inline requirements file. Optional. > # (FIXME: I guess this means we need a spec for requirements files?) > requirements = """ > mybuildtool >= 2.1 > special_windows_tool ; sys_platform == "win32" > """ > # The path to an out-of-line requirements file. Optional. > requirements-file = "build-requirements.txt" > # A hook that will be called to query build requirements. Optional. > requirements-dynamic = "mybuildtool:get_requirements" > > # A hook that will be called to build wheels. Required. > build-wheels = "mybuildtool:do_build" > > # A hook that will be called to do an in-place build (see below). > # Optional. > build-in-place = "mybuildtool:do_inplace_build" > > # The "x" namespace is reserved for third-party extensions. > # To use x.foo you should own the name "foo" on pypi. > [x.mybuildtool] > spam = ["spam", "spam", "spam"] > > All paths are relative to the ``_pypackage/`` directory (so e.g. the > build.requirements-file value above refers to a file named > ``_pypackage/build-requirements.txt``). > > A *hook* is a Python object that is looked up using the same rules as > traditional setuptools entry_points: a dotted module name, followed by > a colon, followed by a dotted name that is looked up within that > module. *Running a hook* means: first, find or create a python > interpreter which is executing in the current venv, whose working > directory is set to the ``_pypackage/`` directory, and which has the > ``_pypackage/`` directory on ``sys.path``. Then, inside this > interpreter, look up the hook object, and call it, with arguments as > specified below. > > A build command like ``pip wheel `` performs the following > steps: > > 1) Validate the ``_pypackage.cfg`` version number. > > 2) Create an empty virtualenv / venv, that matches the environment > that the installer is targeting (e.g. if you want wheels for CPython > 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows > venv). > > 3) If the build.requirements key is present, then in this venv run the > equivalent of ``pip install -r ``, using > whatever index settings are currently in effect. > > 4) If the build.requirements-file key is present, then in this venv > run the equivalent of ``pip install -r ``, using > whatever index settings are currently in effect. > > 5) If the build.requirements-dynamic key is present, then in this venv > run the hook with no arguments, capture its stdout, and pipe it into > ``pip install -r -``, using whatever index settings are currently in > effect. If the hook raises an exception, then abort the build with an > error. > > Note: because these steps are performed in sequence, the > build.requirements-dynamic hook is allowed to use packages that are > listed in build.requirements or build.requirements-file. > > 6) In this venv, run the build.build-wheels hook. This should be a > Python function which takes one argument. > > This argument is an arbitrary dictionary intended to contain > user-specified configuration, specified via some install-tool-specific > mechanism. The intention is that tools like ``pip`` should provide > some way for users to specify key/value settings that will be passed > in here, analogous to the legacy ``--install-option`` and > ``--global-option`` arguments. > > To make it easier for packages to transition from version 0 to > version 1 sdists, we suggest that ``pip`` and other tools that have > such existing option-setting interfaces SHOULD map them to entries in > this dictionary when -- e.g.:: > > pip --global-option=a --install-option=b --install-option=c > > could produce a dict like:: > > {"--global-option": ["a"], "--install-option": ["b", "c"]} > > The hook's return value is a list of pathnames relative to the > scratch directory. Each entry names a wheel file created by this > build. > > Errors are signaled by raising an exception. > > When performing an in-place build (e.g. for ``pip install -e .``), > then the same steps are followed, except that instead of the > build.build-wheels hook, we call the build.build-in-place hook, and > instead of returning a list of wheel files, it returns the name of a > directory that should be placed onto ``sys.path`` (usually this will > be the source tree itself, but may not be, e.g. if a build system > wants to enforce a rule where the source is always kept pristine then > it could symlink the .py files into a build directory, place the > extension modules and dist-info there, and return that). This > directory must contain importable versions of the code in the source > tree, along with appropriate .dist-info directories. > > (FIXME: in-place builds are useful but intrinsically kinda broken -- > e.g. extensions / source / metadata can all easily get out of sync -- > so while I think this paragraph provides a reasonable hack that > preserves current functionality, maybe we should defer specifying them > to until after we've thought through the issues more?) > > When working with source trees, build tools like ``pip`` are > encouraged to cache and re-use virtualenvs for performance. > > > Other contents of _pypackage/ > ----------------------------- > > _RECORD, _RECORD.jws, _RECORD.p7s: see below. > > _x//: reserved for use by tools (e.g. > _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) > > > Source distributions > ==================== > > A *source distribution* is a file in a well-known archive format such > as zip or tar.gz, which contains a single directory, and this > directory is a source tree (in the sense defined in the previous > section). > > The ``_pypackage/`` directory in a source distribution SHOULD also > contain a _RECORD file, as defined in PEP 427, and MAY also contain > _RECORD.jws and/or _RECORD.p7s signature files. > > For official releases, source distributions SHOULD be named as > ``-.``, and the directory they contain SHOULD > be named ``-``, and building this source tree SHOULD > produce a wheel named ``--.whl`` > (though it may produce other wheels as well). > > (FIXME: maybe we should add that if you want your sdist on PyPI then > you MUST include a proper _RECORD file and use the proper naming > convention?) > > Integration tools like ``pip`` SHOULD take advantage of this > convention by applying the following heuristic: when seeking a package > , if no appropriate wheel can be found, but an sdist named > -. is found, then: > > 1) build the sdist > 2) add the resulting wheels to the package search space > 3) retry the original operation > > This handles a variety of simple and complex cases -- for example, if > we need a package 'foo', and we find foo-1.0.zip which builds foo.whl > and bar.whl, and foo.whl depends on bar.whl, then everything will work > out. There remain other cases that are not handled, e.g. if we start > out searching for bar.whl we will never discover foo-1.0.zip. We take > the perspective that this is nonetheless sufficient for a transitional > heuristic, and anyone who runs into this problem should just upload > wheels already. If this turns out to be inadequate in practice, then > it will be addressed by future extensions. > > > Examples > ======== > > **Example 1:** While we assume that installation tools will have to > continue supporting version 0 sdists for the indefinite future, it's a > useful check to make sure that our new format can continue to support > packages using distutils / setuptools as their build system. We assume > that a future version ``pip`` will take its existing knowledge of > distutils internals and expose them as the appropriate hooks, and then > existing distutils / setuptools packages can be ported forward by > using the following ``_pypackage/_pypackage.cfg``:: > > [build] > requirements = """ > pip >= whatever > wheel > """ > # Applies monkeypatches, then does 'setup.py dist_info' and > # extracts the setup_requires > requirements-dynamic = "pip.pypackage_hooks:setup_requirements" > # Applies monkeypatches, then does 'setup.py wheel' > build-wheels = "pip.pypackage_hooks:build_wheels" > # Applies monkeypatches, then does: > # setup.py dist_info && setup.py build_ext -i > build-in-place = "pip.pypackage_hooks:build_in_place" > > This is also useful for any other installation tools that may want to > support version 0 sdists without having to implement bug-for-bug > compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is > present, they can use this as a default. > > **Example 2:** For packages using numpy.distutils. This is identical > to the distutils / setuptools example above, except that numpy is > moved into the list of static build requirements. Right now, most > projects using numpy.distutils don't bother trying to declare this > dependency, and instead simply error out if numpy is not already > installed. This is because currently the only way to declare a build > dependency is via the ``setup_requires`` argument to the ``setup`` > function, and in this case the ``setup`` function is > ``numpy.distutils.setup``, which... obviously doesn't work very well. > Drop this ``_pypackage.cfg`` into an existing project like this and it > will become robustly pip-installable with no further changes:: > > [build] > requirements = """ > numpy > pip >= whatever > wheel > """ > requirements-dynamic = "pip.pypackage_hooks:setup_requirements" > build-wheels = "pip.pypackage_hooks:build_wheels" > build-in-place = "pip.pypackage_hooks:build_in_place" > > **Example 3:** `flit `_ is a tool > designed to make distributing simple packages simple, but it currently > has no support for sdists, and for convenience includes its own > installation code that's redundant with that in pip. These 4 lines of > boilerplate make any flit-using source tree pip-installable, and lets > flit get out of the package installation business:: > > [build] > requirements = "flit" > build-wheels = "flit.pypackage_hooks:build_wheels" > build-in-place = "flit.pypackage_hooks:build_in_place" > > > FAQ > === > > **Why is it version 1 instead of version 2?** Because the legacy sdist > format is barely a format at all, and to `remind us to keep things > simple < > https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect > >`_. > > **What about cross-compilation?** Standardizing an interface for > cross-compilation seems premature given how complicated the > configuration required can be, the lack of an existing de facto > standard, and the authors of this PEP's inexperience with > cross-compilation. This would be a great target for future extensions, > though. In the mean time, there's no requirement that > ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a > project's build system -- packages that want to support > cross-compilation can still do so, they'll just need to include a > README explaining how to do it. > > **PEP 426 says that the new sdist format will support automatically > creating policy-compliant .deb/.rpm packages. What happened to that?** > Step 1: enhance the wheel format as necessary so that a wheel can be > automatically converted into a policy-compliant .deb/.rpm package (see > PEP 491). Step 2: make it possible to automatically turn sdists into > wheels (this PEP). Step 3: we're done. > > **What about automatically running tests?** Arguably this is another > thing that should be pushed off to wheel metadata instead of sdist > metadata: it's good practice to include tests inside your built > distribution so that end-users can test their install (and see above > re: our focus here being on stuff that end-users want to do, not > dedicated package developers), there are lots of packages that have to > be built before they can be tested anyway (e.g. because of binary > extensions), and in any case it's good practice to test against an > installed version in order to make sure your install code works > properly. But even if we do want this in sdist, then it's hardly > urgent (e.g. there is no ``pip test`` that people will miss), so we > defer that for a future extension to avoid blocking the core > functionality. > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Oct 2 16:17:52 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 14:17:52 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Thank you for your work on this. We have to kill distutils to make progress in packaging. On Fri, Oct 2, 2015 at 10:12 AM Daniel Holth wrote: > One way to do sdist 2.0 would be to have the package-1.0.dist-info > directory in there (most sdists contain setuptools metadata) and to have a > flag static-metadata=1 in setup.cfg asserting that setup.py [if present] > does not alter the list of dependencies. > > In the old MEBS design the package could suggest a build system, but pip > would invoke a list of build plugins to inspect the directory and return > True if they were able to build the package. This would allow for ignoring > the package's suggested build system. Instead of defining a command line > interface for setup.py MEBS would define a set of methods on the build > plugin. > > I thought Robert Collins had a working setup-requires implementation > already? I have a worse but backwards compatible one too at > https://bitbucket.org/dholth/setup-requires/src/tip/setup.py > > On Fri, Oct 2, 2015 at 9:42 AM Marcus Smith wrote: > >> Can you clarify the relationship to PEP426 metadata? >> There's no standard for metadata in here other than what's required to >> run a build hook. >> Does that imply you would have each build tool enforce their own >> convention for where metadata is found? >> >> On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith wrote: >> >>> Hi all, >>> >>> We realized that actually as far as we could tell, it wouldn't be that >>> hard at this point to clean up how sdists work so that it would be >>> possible to migrate away from distutils. So we wrote up a little draft >>> proposal. >>> >>> The main question is, does this approach seem sound? >>> >>> -n >>> >>> --- >>> >>> PEP: ?? >>> Title: Standard interface for interacting with source trees >>> and source distributions >>> Version: $Revision$ >>> Last-Modified: $Date$ >>> Author: Nathaniel J. Smith >>> Thomas Kluyver >>> Status: Draft >>> Type: Standards-Track >>> Content-Type: text/x-rst >>> Created: 30-Sep-2015 >>> Post-History: >>> Discussions-To: >>> >>> Abstract >>> ======== >>> >>> Distutils delenda est. >>> >>> >>> Extended abstract >>> ================= >>> >>> While ``distutils`` / ``setuptools`` have taken us a long way, they >>> suffer from three serious problems: (a) they're missing important >>> features like autoconfiguration and usable build-time dependency >>> declaration, (b) extending them is quirky, complicated, and fragile, >>> (c) you are forced to use them anyway, because they provide the >>> standard interface for installing python packages expected by both >>> users and installation tools like ``pip``. >>> >>> Previous efforts (e.g. distutils2 or setuptools itself) have attempted >>> to solve problems (a) and/or (b). We propose to solve (c). >>> >>> The goal of this PEP is get distutils-sig out of the business of being >>> a gatekeeper for Python build systems. If you want to use distutils, >>> great; if you want to use something else, then the more the merrier. >>> The difficulty of interfacing with distutils means that there aren't >>> many such systems right now, but to give a sense of what we're >>> thinking about see `flit `_ or >>> `bento >>> `_. Fortunately, wheels have now >>> solved many of the hard problems here -- e.g. it's no longer necessary >>> that a build system also know about every possible installation >>> configuration -- so pretty much all we really need from a build system >>> is that it have some way to spit out standard-compliant wheels. >>> >>> We therefore propose a new, relatively minimal interface for >>> installation tools like ``pip`` to interact with package source trees >>> and source distributions. >>> >>> >>> Synopsis and rationale >>> ====================== >>> >>> To limit the scope of our design, we adopt several principles. >>> >>> First, we distinguish between a *source tree* (e.g., a VCS checkout) >>> and a *source distribution* (e.g., an official snapshot release like >>> ``lxml-3.4.4.zip``). >>> >>> There isn't a whole lot that *source trees* can be assumed to have in >>> common. About all you know is that they can -- via some more or less >>> Rube-Goldbergian process -- produce one or more binary distributions. >>> In particular, you *cannot* tell via simple static inspection: >>> - What version number will be attached to the resulting packages (e.g. >>> it might be determined programmatically by consulting VCS metadata -- >>> I have here a build of numpy version "1.11.0.dev0+4a9ad17") >>> - What build- or run-time dependencies are required (e.g. these may >>> depend on arbitrarily complex configuration settings that are >>> determined via a mix of manual settings and auto-probing) >>> - Or even how many distinct binary distributions will be produced >>> (e.g. a source distribution may always produce wheel A, but only >>> produce wheel B when built on Unix-like systems). >>> >>> Therefore, when dealing with source trees, our goal is just to provide >>> a standard UX for the core operations that are commonly performed on >>> other people's packages; anything fancier and more developer-centric >>> we leave at the discretion of individual package developers. So our >>> source trees just provide some simple hooks to let a tool like >>> ``pip``: >>> >>> - query for build dependencies >>> - run a build, producing wheels as output >>> - set up the current source tree so that it can be placed on >>> ``sys.path`` in "develop mode" >>> >>> and that's it. We teach users that the standard way to install a >>> package from a VCS checkout is now ``pip install .`` instead of >>> ``python setup.py install``. (This is already a good idea anyway -- >>> e.g., pip can do reliable uninstall / upgrades.) >>> >>> Next, we note that pretty much all the operations that you might want >>> to perform on a *source distribution* are also operations that you >>> might want to perform on a source tree, and via the same UX. The only >>> thing you do with source distributions that you don't do with source >>> trees is, well, distribute them. There's all kind of metadata you >>> could imagine including in a source distribution, but each piece of >>> metadata puts an increased burden on source distribution generation >>> tools, and most operations will still have to work without this >>> metadata. So we only include extra metadata in source distributions if >>> it helps solve specific problems that are unique to distribution. If >>> you want wheel-style metadata, get a wheel and look at it -- they're >>> great and getting better. >>> >>> Therefore, our source distributions are basically just source trees + >>> a mechanism for signing. >>> >>> Finally: we explicitly do *not* have any concept of "depending on a >>> source distribution". As in other systems like Debian, dependencies >>> are always phrased in terms of binary distributions (wheels), and when >>> a user runs something like ``pip install ``, then the >>> long-run plan is that and all its transitive dependencies >>> should be available as wheels in a package index. But this is not yet >>> realistic, so as a transitional / backwards-compatibility measure, we >>> provide a simple mechanism for ``pip install `` to handle >>> cases where is provided only as a source distribution. >>> >>> >>> Source trees >>> ============ >>> >>> We retroactively declare the legacy source tree format involving >>> ``setup.py`` to be "version 0". We don't try to specify it further; >>> its de facto specification is encoded in the source code of >>> ``distutils``, ``setuptools``, ``pip``, and other tools. >>> >>> A version 1-or-greater format source tree can be identified by the >>> presence of a file ``_pypackage/_pypackage.cfg``. >>> >>> If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, >>> then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. >>> This is necessary because we anticipate that version 1+ source trees >>> may want to contain a ``setup.py`` file for backwards compatibility, >>> e.g.:: >>> >>> #!/usr/bin/env python >>> import sys >>> print("Don't call setup.py directly!") >>> print("Use 'pip install .' instead!") >>> print("(You might have to upgrade pip first.)") >>> sys.exit(1) >>> >>> In the current version of the specification, the one file >>> ``_pypackage/_pypackage.cfg`` is where pretty much all the action is >>> (though see below). The motivation for putting it into a subdirectory >>> is that: >>> - the way of all standards is that cruft accumulates over time, so >>> this way we pre-emptively have a place to put it, >>> - real-world projects often accumulate build system cruft as well, so >>> we might as well provide one obvious place to put it too. >>> >>> Of course this then creates the possibility of collisions between >>> standard files and user files, and trying to teach arbitrary users not >>> to scatter files around willy-nilly never works, so we adopt the >>> convention that names starting with an underscore are reserved for >>> official use, and non-underscored names are available for >>> idiosyncratic use by individual projects. >>> >>> The alternative would be to simply place the main configuration file >>> at the top-level, create the subdirectory only when specifically >>> needed (most trees won't need it), and let users worry about finding >>> their own place for their cruft. Not sure which is the best approach. >>> Plus we can have a nice bikeshed about the names in general (FIXME). >>> >>> _pypackage.cfg >>> -------------- >>> >>> The ``_pypackage.cfg`` file contains various settings. Another good >>> bike-shed topic is which file format to use for storing these (FIXME), >>> but for purposes of this draft I'll write examples using `toml >>> `_, because you'll instantly be >>> able to understand the semantics, it has similar expressivity to JSON >>> while being more human-friendly (e.g., it supports comments and >>> multi-line strings), it's better-specified than ConfigParser, and it's >>> much simpler than YAML. Rust's package manager uses toml for similar >>> purposes. >>> >>> Here's an example ``_pypackage/_pypackage.cfg``:: >>> >>> # Version of the "pypackage format" that this file uses. >>> # Optional. If not present then 1 is assumed. >>> # All version changes indicate incompatible changes; backwards >>> # compatible changes are indicated by just having extra stuff in >>> # the file. >>> version = 1 >>> >>> [build] >>> # An inline requirements file. Optional. >>> # (FIXME: I guess this means we need a spec for requirements files?) >>> requirements = """ >>> mybuildtool >= 2.1 >>> special_windows_tool ; sys_platform == "win32" >>> """ >>> # The path to an out-of-line requirements file. Optional. >>> requirements-file = "build-requirements.txt" >>> # A hook that will be called to query build requirements. Optional. >>> requirements-dynamic = "mybuildtool:get_requirements" >>> >>> # A hook that will be called to build wheels. Required. >>> build-wheels = "mybuildtool:do_build" >>> >>> # A hook that will be called to do an in-place build (see below). >>> # Optional. >>> build-in-place = "mybuildtool:do_inplace_build" >>> >>> # The "x" namespace is reserved for third-party extensions. >>> # To use x.foo you should own the name "foo" on pypi. >>> [x.mybuildtool] >>> spam = ["spam", "spam", "spam"] >>> >>> All paths are relative to the ``_pypackage/`` directory (so e.g. the >>> build.requirements-file value above refers to a file named >>> ``_pypackage/build-requirements.txt``). >>> >>> A *hook* is a Python object that is looked up using the same rules as >>> traditional setuptools entry_points: a dotted module name, followed by >>> a colon, followed by a dotted name that is looked up within that >>> module. *Running a hook* means: first, find or create a python >>> interpreter which is executing in the current venv, whose working >>> directory is set to the ``_pypackage/`` directory, and which has the >>> ``_pypackage/`` directory on ``sys.path``. Then, inside this >>> interpreter, look up the hook object, and call it, with arguments as >>> specified below. >>> >>> A build command like ``pip wheel `` performs the following >>> steps: >>> >>> 1) Validate the ``_pypackage.cfg`` version number. >>> >>> 2) Create an empty virtualenv / venv, that matches the environment >>> that the installer is targeting (e.g. if you want wheels for CPython >>> 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows >>> venv). >>> >>> 3) If the build.requirements key is present, then in this venv run the >>> equivalent of ``pip install -r ``, using >>> whatever index settings are currently in effect. >>> >>> 4) If the build.requirements-file key is present, then in this venv >>> run the equivalent of ``pip install -r ``, using >>> whatever index settings are currently in effect. >>> >>> 5) If the build.requirements-dynamic key is present, then in this venv >>> run the hook with no arguments, capture its stdout, and pipe it into >>> ``pip install -r -``, using whatever index settings are currently in >>> effect. If the hook raises an exception, then abort the build with an >>> error. >>> >>> Note: because these steps are performed in sequence, the >>> build.requirements-dynamic hook is allowed to use packages that are >>> listed in build.requirements or build.requirements-file. >>> >>> 6) In this venv, run the build.build-wheels hook. This should be a >>> Python function which takes one argument. >>> >>> This argument is an arbitrary dictionary intended to contain >>> user-specified configuration, specified via some install-tool-specific >>> mechanism. The intention is that tools like ``pip`` should provide >>> some way for users to specify key/value settings that will be passed >>> in here, analogous to the legacy ``--install-option`` and >>> ``--global-option`` arguments. >>> >>> To make it easier for packages to transition from version 0 to >>> version 1 sdists, we suggest that ``pip`` and other tools that have >>> such existing option-setting interfaces SHOULD map them to entries in >>> this dictionary when -- e.g.:: >>> >>> pip --global-option=a --install-option=b --install-option=c >>> >>> could produce a dict like:: >>> >>> {"--global-option": ["a"], "--install-option": ["b", "c"]} >>> >>> The hook's return value is a list of pathnames relative to the >>> scratch directory. Each entry names a wheel file created by this >>> build. >>> >>> Errors are signaled by raising an exception. >>> >>> When performing an in-place build (e.g. for ``pip install -e .``), >>> then the same steps are followed, except that instead of the >>> build.build-wheels hook, we call the build.build-in-place hook, and >>> instead of returning a list of wheel files, it returns the name of a >>> directory that should be placed onto ``sys.path`` (usually this will >>> be the source tree itself, but may not be, e.g. if a build system >>> wants to enforce a rule where the source is always kept pristine then >>> it could symlink the .py files into a build directory, place the >>> extension modules and dist-info there, and return that). This >>> directory must contain importable versions of the code in the source >>> tree, along with appropriate .dist-info directories. >>> >>> (FIXME: in-place builds are useful but intrinsically kinda broken -- >>> e.g. extensions / source / metadata can all easily get out of sync -- >>> so while I think this paragraph provides a reasonable hack that >>> preserves current functionality, maybe we should defer specifying them >>> to until after we've thought through the issues more?) >>> >>> When working with source trees, build tools like ``pip`` are >>> encouraged to cache and re-use virtualenvs for performance. >>> >>> >>> Other contents of _pypackage/ >>> ----------------------------- >>> >>> _RECORD, _RECORD.jws, _RECORD.p7s: see below. >>> >>> _x//: reserved for use by tools (e.g. >>> _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) >>> >>> >>> Source distributions >>> ==================== >>> >>> A *source distribution* is a file in a well-known archive format such >>> as zip or tar.gz, which contains a single directory, and this >>> directory is a source tree (in the sense defined in the previous >>> section). >>> >>> The ``_pypackage/`` directory in a source distribution SHOULD also >>> contain a _RECORD file, as defined in PEP 427, and MAY also contain >>> _RECORD.jws and/or _RECORD.p7s signature files. >>> >>> For official releases, source distributions SHOULD be named as >>> ``-.``, and the directory they contain SHOULD >>> be named ``-``, and building this source tree SHOULD >>> produce a wheel named ``--.whl`` >>> (though it may produce other wheels as well). >>> >>> (FIXME: maybe we should add that if you want your sdist on PyPI then >>> you MUST include a proper _RECORD file and use the proper naming >>> convention?) >>> >>> Integration tools like ``pip`` SHOULD take advantage of this >>> convention by applying the following heuristic: when seeking a package >>> , if no appropriate wheel can be found, but an sdist named >>> -. is found, then: >>> >>> 1) build the sdist >>> 2) add the resulting wheels to the package search space >>> 3) retry the original operation >>> >>> This handles a variety of simple and complex cases -- for example, if >>> we need a package 'foo', and we find foo-1.0.zip which builds foo.whl >>> and bar.whl, and foo.whl depends on bar.whl, then everything will work >>> out. There remain other cases that are not handled, e.g. if we start >>> out searching for bar.whl we will never discover foo-1.0.zip. We take >>> the perspective that this is nonetheless sufficient for a transitional >>> heuristic, and anyone who runs into this problem should just upload >>> wheels already. If this turns out to be inadequate in practice, then >>> it will be addressed by future extensions. >>> >>> >>> Examples >>> ======== >>> >>> **Example 1:** While we assume that installation tools will have to >>> continue supporting version 0 sdists for the indefinite future, it's a >>> useful check to make sure that our new format can continue to support >>> packages using distutils / setuptools as their build system. We assume >>> that a future version ``pip`` will take its existing knowledge of >>> distutils internals and expose them as the appropriate hooks, and then >>> existing distutils / setuptools packages can be ported forward by >>> using the following ``_pypackage/_pypackage.cfg``:: >>> >>> [build] >>> requirements = """ >>> pip >= whatever >>> wheel >>> """ >>> # Applies monkeypatches, then does 'setup.py dist_info' and >>> # extracts the setup_requires >>> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >>> # Applies monkeypatches, then does 'setup.py wheel' >>> build-wheels = "pip.pypackage_hooks:build_wheels" >>> # Applies monkeypatches, then does: >>> # setup.py dist_info && setup.py build_ext -i >>> build-in-place = "pip.pypackage_hooks:build_in_place" >>> >>> This is also useful for any other installation tools that may want to >>> support version 0 sdists without having to implement bug-for-bug >>> compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is >>> present, they can use this as a default. >>> >>> **Example 2:** For packages using numpy.distutils. This is identical >>> to the distutils / setuptools example above, except that numpy is >>> moved into the list of static build requirements. Right now, most >>> projects using numpy.distutils don't bother trying to declare this >>> dependency, and instead simply error out if numpy is not already >>> installed. This is because currently the only way to declare a build >>> dependency is via the ``setup_requires`` argument to the ``setup`` >>> function, and in this case the ``setup`` function is >>> ``numpy.distutils.setup``, which... obviously doesn't work very well. >>> Drop this ``_pypackage.cfg`` into an existing project like this and it >>> will become robustly pip-installable with no further changes:: >>> >>> [build] >>> requirements = """ >>> numpy >>> pip >= whatever >>> wheel >>> """ >>> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >>> build-wheels = "pip.pypackage_hooks:build_wheels" >>> build-in-place = "pip.pypackage_hooks:build_in_place" >>> >>> **Example 3:** `flit `_ is a tool >>> designed to make distributing simple packages simple, but it currently >>> has no support for sdists, and for convenience includes its own >>> installation code that's redundant with that in pip. These 4 lines of >>> boilerplate make any flit-using source tree pip-installable, and lets >>> flit get out of the package installation business:: >>> >>> [build] >>> requirements = "flit" >>> build-wheels = "flit.pypackage_hooks:build_wheels" >>> build-in-place = "flit.pypackage_hooks:build_in_place" >>> >>> >>> FAQ >>> === >>> >>> **Why is it version 1 instead of version 2?** Because the legacy sdist >>> format is barely a format at all, and to `remind us to keep things >>> simple < >>> https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect >>> >`_. >>> >>> **What about cross-compilation?** Standardizing an interface for >>> cross-compilation seems premature given how complicated the >>> configuration required can be, the lack of an existing de facto >>> standard, and the authors of this PEP's inexperience with >>> cross-compilation. This would be a great target for future extensions, >>> though. In the mean time, there's no requirement that >>> ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a >>> project's build system -- packages that want to support >>> cross-compilation can still do so, they'll just need to include a >>> README explaining how to do it. >>> >>> **PEP 426 says that the new sdist format will support automatically >>> creating policy-compliant .deb/.rpm packages. What happened to that?** >>> Step 1: enhance the wheel format as necessary so that a wheel can be >>> automatically converted into a policy-compliant .deb/.rpm package (see >>> PEP 491). Step 2: make it possible to automatically turn sdists into >>> wheels (this PEP). Step 3: we're done. >>> >>> **What about automatically running tests?** Arguably this is another >>> thing that should be pushed off to wheel metadata instead of sdist >>> metadata: it's good practice to include tests inside your built >>> distribution so that end-users can test their install (and see above >>> re: our focus here being on stuff that end-users want to do, not >>> dedicated package developers), there are lots of packages that have to >>> be built before they can be tested anyway (e.g. because of binary >>> extensions), and in any case it's good practice to test against an >>> installed version in order to make sure your install code works >>> properly. But even if we do want this in sdist, then it's hardly >>> urgent (e.g. there is no ``pip test`` that people will miss), so we >>> defer that for a future extension to avoid blocking the core >>> functionality. >>> >>> -- >>> Nathaniel J. Smith -- http://vorpus.org >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Oct 2 16:12:27 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 14:12:27 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: One way to do sdist 2.0 would be to have the package-1.0.dist-info directory in there (most sdists contain setuptools metadata) and to have a flag static-metadata=1 in setup.cfg asserting that setup.py [if present] does not alter the list of dependencies. In the old MEBS design the package could suggest a build system, but pip would invoke a list of build plugins to inspect the directory and return True if they were able to build the package. This would allow for ignoring the package's suggested build system. Instead of defining a command line interface for setup.py MEBS would define a set of methods on the build plugin. I thought Robert Collins had a working setup-requires implementation already? I have a worse but backwards compatible one too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py On Fri, Oct 2, 2015 at 9:42 AM Marcus Smith wrote: > Can you clarify the relationship to PEP426 metadata? > There's no standard for metadata in here other than what's required to run > a build hook. > Does that imply you would have each build tool enforce their own > convention for where metadata is found? > > On Thu, Oct 1, 2015 at 9:53 PM, Nathaniel Smith wrote: > >> Hi all, >> >> We realized that actually as far as we could tell, it wouldn't be that >> hard at this point to clean up how sdists work so that it would be >> possible to migrate away from distutils. So we wrote up a little draft >> proposal. >> >> The main question is, does this approach seem sound? >> >> -n >> >> --- >> >> PEP: ?? >> Title: Standard interface for interacting with source trees >> and source distributions >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Nathaniel J. Smith >> Thomas Kluyver >> Status: Draft >> Type: Standards-Track >> Content-Type: text/x-rst >> Created: 30-Sep-2015 >> Post-History: >> Discussions-To: >> >> Abstract >> ======== >> >> Distutils delenda est. >> >> >> Extended abstract >> ================= >> >> While ``distutils`` / ``setuptools`` have taken us a long way, they >> suffer from three serious problems: (a) they're missing important >> features like autoconfiguration and usable build-time dependency >> declaration, (b) extending them is quirky, complicated, and fragile, >> (c) you are forced to use them anyway, because they provide the >> standard interface for installing python packages expected by both >> users and installation tools like ``pip``. >> >> Previous efforts (e.g. distutils2 or setuptools itself) have attempted >> to solve problems (a) and/or (b). We propose to solve (c). >> >> The goal of this PEP is get distutils-sig out of the business of being >> a gatekeeper for Python build systems. If you want to use distutils, >> great; if you want to use something else, then the more the merrier. >> The difficulty of interfacing with distutils means that there aren't >> many such systems right now, but to give a sense of what we're >> thinking about see `flit `_ or >> `bento >> `_. Fortunately, wheels have now >> solved many of the hard problems here -- e.g. it's no longer necessary >> that a build system also know about every possible installation >> configuration -- so pretty much all we really need from a build system >> is that it have some way to spit out standard-compliant wheels. >> >> We therefore propose a new, relatively minimal interface for >> installation tools like ``pip`` to interact with package source trees >> and source distributions. >> >> >> Synopsis and rationale >> ====================== >> >> To limit the scope of our design, we adopt several principles. >> >> First, we distinguish between a *source tree* (e.g., a VCS checkout) >> and a *source distribution* (e.g., an official snapshot release like >> ``lxml-3.4.4.zip``). >> >> There isn't a whole lot that *source trees* can be assumed to have in >> common. About all you know is that they can -- via some more or less >> Rube-Goldbergian process -- produce one or more binary distributions. >> In particular, you *cannot* tell via simple static inspection: >> - What version number will be attached to the resulting packages (e.g. >> it might be determined programmatically by consulting VCS metadata -- >> I have here a build of numpy version "1.11.0.dev0+4a9ad17") >> - What build- or run-time dependencies are required (e.g. these may >> depend on arbitrarily complex configuration settings that are >> determined via a mix of manual settings and auto-probing) >> - Or even how many distinct binary distributions will be produced >> (e.g. a source distribution may always produce wheel A, but only >> produce wheel B when built on Unix-like systems). >> >> Therefore, when dealing with source trees, our goal is just to provide >> a standard UX for the core operations that are commonly performed on >> other people's packages; anything fancier and more developer-centric >> we leave at the discretion of individual package developers. So our >> source trees just provide some simple hooks to let a tool like >> ``pip``: >> >> - query for build dependencies >> - run a build, producing wheels as output >> - set up the current source tree so that it can be placed on >> ``sys.path`` in "develop mode" >> >> and that's it. We teach users that the standard way to install a >> package from a VCS checkout is now ``pip install .`` instead of >> ``python setup.py install``. (This is already a good idea anyway -- >> e.g., pip can do reliable uninstall / upgrades.) >> >> Next, we note that pretty much all the operations that you might want >> to perform on a *source distribution* are also operations that you >> might want to perform on a source tree, and via the same UX. The only >> thing you do with source distributions that you don't do with source >> trees is, well, distribute them. There's all kind of metadata you >> could imagine including in a source distribution, but each piece of >> metadata puts an increased burden on source distribution generation >> tools, and most operations will still have to work without this >> metadata. So we only include extra metadata in source distributions if >> it helps solve specific problems that are unique to distribution. If >> you want wheel-style metadata, get a wheel and look at it -- they're >> great and getting better. >> >> Therefore, our source distributions are basically just source trees + >> a mechanism for signing. >> >> Finally: we explicitly do *not* have any concept of "depending on a >> source distribution". As in other systems like Debian, dependencies >> are always phrased in terms of binary distributions (wheels), and when >> a user runs something like ``pip install ``, then the >> long-run plan is that and all its transitive dependencies >> should be available as wheels in a package index. But this is not yet >> realistic, so as a transitional / backwards-compatibility measure, we >> provide a simple mechanism for ``pip install `` to handle >> cases where is provided only as a source distribution. >> >> >> Source trees >> ============ >> >> We retroactively declare the legacy source tree format involving >> ``setup.py`` to be "version 0". We don't try to specify it further; >> its de facto specification is encoded in the source code of >> ``distutils``, ``setuptools``, ``pip``, and other tools. >> >> A version 1-or-greater format source tree can be identified by the >> presence of a file ``_pypackage/_pypackage.cfg``. >> >> If both ``_pypackage/_pypackage.cfg`` and ``setup.py`` are present, >> then we have a version 1+ source tree, i.e., ``setup.py`` is ignored. >> This is necessary because we anticipate that version 1+ source trees >> may want to contain a ``setup.py`` file for backwards compatibility, >> e.g.:: >> >> #!/usr/bin/env python >> import sys >> print("Don't call setup.py directly!") >> print("Use 'pip install .' instead!") >> print("(You might have to upgrade pip first.)") >> sys.exit(1) >> >> In the current version of the specification, the one file >> ``_pypackage/_pypackage.cfg`` is where pretty much all the action is >> (though see below). The motivation for putting it into a subdirectory >> is that: >> - the way of all standards is that cruft accumulates over time, so >> this way we pre-emptively have a place to put it, >> - real-world projects often accumulate build system cruft as well, so >> we might as well provide one obvious place to put it too. >> >> Of course this then creates the possibility of collisions between >> standard files and user files, and trying to teach arbitrary users not >> to scatter files around willy-nilly never works, so we adopt the >> convention that names starting with an underscore are reserved for >> official use, and non-underscored names are available for >> idiosyncratic use by individual projects. >> >> The alternative would be to simply place the main configuration file >> at the top-level, create the subdirectory only when specifically >> needed (most trees won't need it), and let users worry about finding >> their own place for their cruft. Not sure which is the best approach. >> Plus we can have a nice bikeshed about the names in general (FIXME). >> >> _pypackage.cfg >> -------------- >> >> The ``_pypackage.cfg`` file contains various settings. Another good >> bike-shed topic is which file format to use for storing these (FIXME), >> but for purposes of this draft I'll write examples using `toml >> `_, because you'll instantly be >> able to understand the semantics, it has similar expressivity to JSON >> while being more human-friendly (e.g., it supports comments and >> multi-line strings), it's better-specified than ConfigParser, and it's >> much simpler than YAML. Rust's package manager uses toml for similar >> purposes. >> >> Here's an example ``_pypackage/_pypackage.cfg``:: >> >> # Version of the "pypackage format" that this file uses. >> # Optional. If not present then 1 is assumed. >> # All version changes indicate incompatible changes; backwards >> # compatible changes are indicated by just having extra stuff in >> # the file. >> version = 1 >> >> [build] >> # An inline requirements file. Optional. >> # (FIXME: I guess this means we need a spec for requirements files?) >> requirements = """ >> mybuildtool >= 2.1 >> special_windows_tool ; sys_platform == "win32" >> """ >> # The path to an out-of-line requirements file. Optional. >> requirements-file = "build-requirements.txt" >> # A hook that will be called to query build requirements. Optional. >> requirements-dynamic = "mybuildtool:get_requirements" >> >> # A hook that will be called to build wheels. Required. >> build-wheels = "mybuildtool:do_build" >> >> # A hook that will be called to do an in-place build (see below). >> # Optional. >> build-in-place = "mybuildtool:do_inplace_build" >> >> # The "x" namespace is reserved for third-party extensions. >> # To use x.foo you should own the name "foo" on pypi. >> [x.mybuildtool] >> spam = ["spam", "spam", "spam"] >> >> All paths are relative to the ``_pypackage/`` directory (so e.g. the >> build.requirements-file value above refers to a file named >> ``_pypackage/build-requirements.txt``). >> >> A *hook* is a Python object that is looked up using the same rules as >> traditional setuptools entry_points: a dotted module name, followed by >> a colon, followed by a dotted name that is looked up within that >> module. *Running a hook* means: first, find or create a python >> interpreter which is executing in the current venv, whose working >> directory is set to the ``_pypackage/`` directory, and which has the >> ``_pypackage/`` directory on ``sys.path``. Then, inside this >> interpreter, look up the hook object, and call it, with arguments as >> specified below. >> >> A build command like ``pip wheel `` performs the following >> steps: >> >> 1) Validate the ``_pypackage.cfg`` version number. >> >> 2) Create an empty virtualenv / venv, that matches the environment >> that the installer is targeting (e.g. if you want wheels for CPython >> 3.4 on 64-bit windows, then you make a CPython 3.4 64-bit windows >> venv). >> >> 3) If the build.requirements key is present, then in this venv run the >> equivalent of ``pip install -r ``, using >> whatever index settings are currently in effect. >> >> 4) If the build.requirements-file key is present, then in this venv >> run the equivalent of ``pip install -r ``, using >> whatever index settings are currently in effect. >> >> 5) If the build.requirements-dynamic key is present, then in this venv >> run the hook with no arguments, capture its stdout, and pipe it into >> ``pip install -r -``, using whatever index settings are currently in >> effect. If the hook raises an exception, then abort the build with an >> error. >> >> Note: because these steps are performed in sequence, the >> build.requirements-dynamic hook is allowed to use packages that are >> listed in build.requirements or build.requirements-file. >> >> 6) In this venv, run the build.build-wheels hook. This should be a >> Python function which takes one argument. >> >> This argument is an arbitrary dictionary intended to contain >> user-specified configuration, specified via some install-tool-specific >> mechanism. The intention is that tools like ``pip`` should provide >> some way for users to specify key/value settings that will be passed >> in here, analogous to the legacy ``--install-option`` and >> ``--global-option`` arguments. >> >> To make it easier for packages to transition from version 0 to >> version 1 sdists, we suggest that ``pip`` and other tools that have >> such existing option-setting interfaces SHOULD map them to entries in >> this dictionary when -- e.g.:: >> >> pip --global-option=a --install-option=b --install-option=c >> >> could produce a dict like:: >> >> {"--global-option": ["a"], "--install-option": ["b", "c"]} >> >> The hook's return value is a list of pathnames relative to the >> scratch directory. Each entry names a wheel file created by this >> build. >> >> Errors are signaled by raising an exception. >> >> When performing an in-place build (e.g. for ``pip install -e .``), >> then the same steps are followed, except that instead of the >> build.build-wheels hook, we call the build.build-in-place hook, and >> instead of returning a list of wheel files, it returns the name of a >> directory that should be placed onto ``sys.path`` (usually this will >> be the source tree itself, but may not be, e.g. if a build system >> wants to enforce a rule where the source is always kept pristine then >> it could symlink the .py files into a build directory, place the >> extension modules and dist-info there, and return that). This >> directory must contain importable versions of the code in the source >> tree, along with appropriate .dist-info directories. >> >> (FIXME: in-place builds are useful but intrinsically kinda broken -- >> e.g. extensions / source / metadata can all easily get out of sync -- >> so while I think this paragraph provides a reasonable hack that >> preserves current functionality, maybe we should defer specifying them >> to until after we've thought through the issues more?) >> >> When working with source trees, build tools like ``pip`` are >> encouraged to cache and re-use virtualenvs for performance. >> >> >> Other contents of _pypackage/ >> ----------------------------- >> >> _RECORD, _RECORD.jws, _RECORD.p7s: see below. >> >> _x//: reserved for use by tools (e.g. >> _x/mybuildtool/build/, _x/pip/venv-cache/cp34-none-linux_x86_64/) >> >> >> Source distributions >> ==================== >> >> A *source distribution* is a file in a well-known archive format such >> as zip or tar.gz, which contains a single directory, and this >> directory is a source tree (in the sense defined in the previous >> section). >> >> The ``_pypackage/`` directory in a source distribution SHOULD also >> contain a _RECORD file, as defined in PEP 427, and MAY also contain >> _RECORD.jws and/or _RECORD.p7s signature files. >> >> For official releases, source distributions SHOULD be named as >> ``-.``, and the directory they contain SHOULD >> be named ``-``, and building this source tree SHOULD >> produce a wheel named ``--.whl`` >> (though it may produce other wheels as well). >> >> (FIXME: maybe we should add that if you want your sdist on PyPI then >> you MUST include a proper _RECORD file and use the proper naming >> convention?) >> >> Integration tools like ``pip`` SHOULD take advantage of this >> convention by applying the following heuristic: when seeking a package >> , if no appropriate wheel can be found, but an sdist named >> -. is found, then: >> >> 1) build the sdist >> 2) add the resulting wheels to the package search space >> 3) retry the original operation >> >> This handles a variety of simple and complex cases -- for example, if >> we need a package 'foo', and we find foo-1.0.zip which builds foo.whl >> and bar.whl, and foo.whl depends on bar.whl, then everything will work >> out. There remain other cases that are not handled, e.g. if we start >> out searching for bar.whl we will never discover foo-1.0.zip. We take >> the perspective that this is nonetheless sufficient for a transitional >> heuristic, and anyone who runs into this problem should just upload >> wheels already. If this turns out to be inadequate in practice, then >> it will be addressed by future extensions. >> >> >> Examples >> ======== >> >> **Example 1:** While we assume that installation tools will have to >> continue supporting version 0 sdists for the indefinite future, it's a >> useful check to make sure that our new format can continue to support >> packages using distutils / setuptools as their build system. We assume >> that a future version ``pip`` will take its existing knowledge of >> distutils internals and expose them as the appropriate hooks, and then >> existing distutils / setuptools packages can be ported forward by >> using the following ``_pypackage/_pypackage.cfg``:: >> >> [build] >> requirements = """ >> pip >= whatever >> wheel >> """ >> # Applies monkeypatches, then does 'setup.py dist_info' and >> # extracts the setup_requires >> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >> # Applies monkeypatches, then does 'setup.py wheel' >> build-wheels = "pip.pypackage_hooks:build_wheels" >> # Applies monkeypatches, then does: >> # setup.py dist_info && setup.py build_ext -i >> build-in-place = "pip.pypackage_hooks:build_in_place" >> >> This is also useful for any other installation tools that may want to >> support version 0 sdists without having to implement bug-for-bug >> compatibility with pip -- if no ``_pypackage/_pypackage.cfg`` is >> present, they can use this as a default. >> >> **Example 2:** For packages using numpy.distutils. This is identical >> to the distutils / setuptools example above, except that numpy is >> moved into the list of static build requirements. Right now, most >> projects using numpy.distutils don't bother trying to declare this >> dependency, and instead simply error out if numpy is not already >> installed. This is because currently the only way to declare a build >> dependency is via the ``setup_requires`` argument to the ``setup`` >> function, and in this case the ``setup`` function is >> ``numpy.distutils.setup``, which... obviously doesn't work very well. >> Drop this ``_pypackage.cfg`` into an existing project like this and it >> will become robustly pip-installable with no further changes:: >> >> [build] >> requirements = """ >> numpy >> pip >= whatever >> wheel >> """ >> requirements-dynamic = "pip.pypackage_hooks:setup_requirements" >> build-wheels = "pip.pypackage_hooks:build_wheels" >> build-in-place = "pip.pypackage_hooks:build_in_place" >> >> **Example 3:** `flit `_ is a tool >> designed to make distributing simple packages simple, but it currently >> has no support for sdists, and for convenience includes its own >> installation code that's redundant with that in pip. These 4 lines of >> boilerplate make any flit-using source tree pip-installable, and lets >> flit get out of the package installation business:: >> >> [build] >> requirements = "flit" >> build-wheels = "flit.pypackage_hooks:build_wheels" >> build-in-place = "flit.pypackage_hooks:build_in_place" >> >> >> FAQ >> === >> >> **Why is it version 1 instead of version 2?** Because the legacy sdist >> format is barely a format at all, and to `remind us to keep things >> simple < >> https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect >> >`_. >> >> **What about cross-compilation?** Standardizing an interface for >> cross-compilation seems premature given how complicated the >> configuration required can be, the lack of an existing de facto >> standard, and the authors of this PEP's inexperience with >> cross-compilation. This would be a great target for future extensions, >> though. In the mean time, there's no requirement that >> ``_pypackage/_pypackage.cfg`` contain the *only* entry points to a >> project's build system -- packages that want to support >> cross-compilation can still do so, they'll just need to include a >> README explaining how to do it. >> >> **PEP 426 says that the new sdist format will support automatically >> creating policy-compliant .deb/.rpm packages. What happened to that?** >> Step 1: enhance the wheel format as necessary so that a wheel can be >> automatically converted into a policy-compliant .deb/.rpm package (see >> PEP 491). Step 2: make it possible to automatically turn sdists into >> wheels (this PEP). Step 3: we're done. >> >> **What about automatically running tests?** Arguably this is another >> thing that should be pushed off to wheel metadata instead of sdist >> metadata: it's good practice to include tests inside your built >> distribution so that end-users can test their install (and see above >> re: our focus here being on stuff that end-users want to do, not >> dedicated package developers), there are lots of packages that have to >> be built before they can be tested anyway (e.g. because of binary >> extensions), and in any case it's good practice to test against an >> installed version in order to make sure your install code works >> properly. But even if we do want this in sdist, then it's hardly >> urgent (e.g. there is no ``pip test`` that people will miss), so we >> defer that for a future extension to avoid blocking the core >> functionality. >> >> -- >> Nathaniel J. Smith -- http://vorpus.org >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Fri Oct 2 15:52:27 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 2 Oct 2015 14:52:27 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 12:58, Donald Stufft wrote: > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. > I can't now find that it in the draft text, but I don't think this is a problem. This proposal means there's a standardised way to turn a source tree into wheels. So downstream distributors can download an sdist - or even a tarball of a VCS tag, if they're being strict about it - build wheels from that using the config in this proposal, and then transform the wheels into their own package format. > Longer term, I think the answer is sdist 2.0 which has proper metadata inside of it (name, version, dependencies, etc) but which also includes a hook like this PEP has to specify the build system I hadn't heard of this before - is it something that's being worked on? Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Oct 2 18:30:42 2015 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 Oct 2015 09:30:42 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: > > > So downstream distributors can download an sdist - or even a tarball of a > VCS tag, if they're being strict about it - build wheels from that using > the config in this proposal, and then transform the wheels into their own > package format. > this has wheel itself being the interoperability standard. the going idea is that there would be a metadata artifact ("pydist.json" in PEP426), that provides the standard across different formats. > Longer term, I think the answer is sdist 2.0 which has proper metadata > inside of it (name, version, dependencies, etc) but which also includes a > hook like this PEP has to specify the build system > > I hadn't heard of this before - is it something that's being worked on? > I haven't seen any PEP's for it yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Fri Oct 2 19:38:47 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 2 Oct 2015 13:38:47 -0400 Subject: [Distutils] intelcompiler class in distutils In-Reply-To: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> References: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> Message-ID: On Wed, Sep 30, 2015 at 3:17 PM, Hogan, Christopher wrote: > Hello, > > > > We noticed there are classes in distutils for specific compilers like > Borland, Cygwin, MSVC, etc., and are interested in creating a class for the > Intel compiler. Is this something that the python developers/community > would like to see, or would such a patch have a low chance for acceptance? Hi, I don't have any weight to add one way or the other, but I have encountered a need for this in the past (though it was a long time ago and I can't recall the context). It might be tricky to get it accepted into distutils in the stdlib, if only because it would have to come with some promise of continued maintenance. And also getting anything added to distutils these days is hard (though last I recall the freeze may be liftable for non-backwards-compat-breaking new features where needed?) That said, you might have better luck adding it to setuptools instead, or a completely separate third-party package. Adding support for a new compiler that is *not* directly in the distutils package requires a bit of hackery though. Incidentally, d2to1 [1] actually supports packages that include their own compiler class, and allows them to use that compiler for builds. I don't know that this feature has ever actually been used, but my point in bringing it up is just to demonstrate that it can be done. And you can look at that code for an example of how to add a new compiler without it being directly in the distutils module (even if it's a bit of a hack). Best, Erik [1] https://github.com/embray/d2to1/blob/master/d2to1/util.py#L330 From brett at python.org Fri Oct 2 20:41:07 2015 From: brett at python.org (Brett Cannon) Date: Fri, 02 Oct 2015 18:41:07 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, 2 Oct 2015 at 05:08 Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > > We realized that actually as far as we could tell, it wouldn't > > be that > > hard at this point to clean up how sdists work so that it would be > > possible to migrate away from distutils. So we wrote up a little > > draft > > proposal. > > > > The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might > be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. Debian for instance > has > policy that requires the use of building all of it's packages from Source, > not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a > mess > when you factor in packages that have any C code what so ever. > So wouldn't they then download the sdist, build a wheel as an intermediate, and then generate the .deb file? I mean as long as people upload an sdist for those that want to build from source and a wheel for convenience -- which is probably what most people providing wheels do anyway -- then I don't see the problem. > > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on > wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that > we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, > all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never > going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism > by > which to determine dependencies of a source distribution without > downloading > the file and executing some potentially untrusted code. This makes > dependency > resolution harder and much much slower than if we could read that > information > statically from a source distribution. This PEP doesn't offer anything in > the > way of solving this problem. > Isn't that what the requirements and requirements-file fields in the _pypackage file provide? Only if you use that requirements-dynamic would it require execcuting arbitrary code to gather dependency information, or am I missing something? > > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine > what > it is and to get information about it. Right now PyPI relies on the > uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from > within the > file. This is sort of possible right now since distutils and setuptools > both > create a static metadata file within the source distribution, but we don't > rely > on that within PyPI because that information may or may not be accurate > and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. > Isn't that only if you use the dynamic fields? > > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference > from > within a .cfg file instead of creating a setuptools extension or so) but > without fixing most of the problems. The largest benefit I see to > switching to > this right now is that it would enable us to have build time dependencies > that > were controlled by pip rather than installed implicitly via the execution > of > the setup.py. That doesn't feel like a big enough benefit to me to do a > mass > shakeup of what we recommend and tell people to do. Having people adjust > and > change and do something new requires effort, and we need something to > justify > that effort to other people and I don't think that this PEP has something > we > can really use to justify that effort. > >From my naive perspective this proposal seems to help push forward a decoupling of building using distutils/setuptools as the only way you can properly build Python projects (which is what I think we are all after) and will hopefully eventually free pip up to simply do orchestration. > > I *do* think that there is a core of some ideas here that are valuable, > and in > fact are similar to some ideas I've had. The main flaw I see here is that > it > doesn't really fix sdists, it takes a solution that would work for VCS > checkouts and then reuses it for sdists. In my mind, the supported flow for > package installation would be: > > VCS/Bare Directory -> Source Distribution -> Wheel > > This would (eventually) be the only path that was supported for > installation > but you could "enter" the path at any stage. For example, if there is a > Wheel > already available, then you jump right on at the end and just install > that, if > there is a sdist available then pip first builds it into a wheel and then > installs that, etc. > > I think your PEP is something like what the VCS/Bare Directory to sdist > tooling > could look like, but I don't think it's what the sdist to wheel path should > look like. > Is there another proposal I'm unaware for the sdist -> wheel step that is build tool-agnostic? I'm all for going with the best solution but there has to be an actual alternative to compare against and I don't know of any others right now and this proposal does seem to move things forward in a reasonable fashion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Oct 2 21:02:44 2015 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 Oct 2015 12:02:44 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: > >> The first thing that immediately stood out to me, is that it's >> recommending >> that downstream redistributors like Debian, Fedora, etc utilize Wheels >> instead >> of the sdist to build their packages from. However, that is not really >> going to >> fly with most (all?) of the downstream redistributors. Debian for >> instance has >> policy that requires the use of building all of it's packages from >> Source, not >> from anything else and Wheels are not a source package. While it can >> theoretically work for pure python packages, it quickly devolves into a >> mess >> when you factor in packages that have any C code what so ever. >> > > So wouldn't they then download the sdist, build a wheel as an > intermediate, and then generate the .deb file? > the new goal I think was to have standardized metadata immediately available in an sdist, and get away from the model, that you had to run a build step, before you had a metadata artifact. so here, you'd have to build a wheel (potentially one with binary extensions) just to know what the metadata is? that doesn't sound right. > I mean as long as people upload an sdist for those that want to build from > source and a wheel for convenience -- which is probably what most people > providing wheels do anyway -- then I don't see the problem. > > >> >> Overall, this feels more like a sidegrade than an upgrade. One major theme >> throughout of the PEP is that we're going to push to rely heavily on >> wheels as >> the primary format of installation. While that works well for things like >> Debian, I don't think it's going to work as wheel for us. If we were only >> distributing pure python packages, then yes absolutely, however given >> that we >> are not, we have to worry about ABI issues. Given that there is so many >> different environments that a particular package might be installed into, >> all >> with different ABIs we have to assume that installing from source is still >> going to be a primary path for end users to install and that we are never >> going >> to have a world where we can assume a Wheel in a repository. >> >> One of the problems with the current system, is that we have no mechanism >> by >> which to determine dependencies of a source distribution without >> downloading >> the file and executing some potentially untrusted code. This makes >> dependency >> resolution harder and much much slower than if we could read that >> information >> statically from a source distribution. This PEP doesn't offer anything in >> the >> way of solving this problem. >> > > Isn't that what the requirements and requirements-file fields in the > _pypackage file provide? Only if you use that requirements-dynamic would it > require execcuting arbitrary code to gather dependency information, or am I > missing something? > those are just requirements to run the build hook, not run time dependencies. > Is there another proposal I'm unaware for the sdist -> wheel step that is > build tool-agnostic? > PEP426 talks about it some https://www.python.org/dev/peps/pep-0426/#metabuild-system -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 21:09:30 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 15:09:30 -0400 Subject: [Distutils] intelcompiler class in distutils In-Reply-To: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> References: <18A68EC651142A408A0A6AFDE7D1FC895C184A@ORSMSX103.amr.corp.intel.com> Message-ID: On October 2, 2015 at 9:42:22 AM, Hogan, Christopher (christopher.hogan at intel.com) wrote: > Hello, > > We noticed there are classes in distutils for specific compilers like Borland, Cygwin, > MSVC, etc., and are interested in creating a class for the Intel compiler. Is this something > that the python developers/community would like to see, or would such a patch have a low > chance for acceptance? > Is the Intel Compiler Free? (or OSS, but Free is the main thing). I think as long as there is a way for people to get a copy of the compiler to run tests and stuff against without spending $$$ there is very little reason not to add another compiler class. If it costs money always, then IDK what the policy for that is, I personally still wouldn?t have a problem as long as there was a commitment from intel to support it, but it?s not my decision either. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Fri Oct 2 21:45:25 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 15:45:25 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 2:41:20 PM, Brett Cannon (brett at python.org) wrote: > > Is there another proposal I'm unaware for the sdist -> wheel > step that is build tool-agnostic? I'm all for going with the best > solution but there has to be an actual alternative to compare > against and I don't know of any others right now and this proposal > does seem to move things forward in a reasonable fashion. As Marcus said, it was touched on in PEP 426 but there isn?t a fully fleshed out PEP (or even a sort of fleshed out PEP) but the high level idea isn?t super hard, take PEP 426 metadata + some sdist specific metadata and stick it in a tarball or something and then you have sdist 2.0. Part of that sdist specific metadata would be describing in some way how to build a wheel out of this sdist (rather than just assuming distutils/setuptools). I think we need to push back against partial solutions to problems that we have without at least some semblance of a plan for how to solve the rest of the problems we have, or to explicitly declare them unsolvable. That doesn't mean that we need to turn every thing into an overengineered mess where we try to solve every problem ever, but we need to be careful that we minimize churn where possible. An example just recently where we had this was we accepted PEP 438 2 years ago as a partial solution to try and "move things forward" and then it turned out that solution was, well partial, and when we removed the partial solution for a full solution we had the folks who decided to rely on the partial solution upset. IOW we need to balance between avoiding churn and making a change "worth" it, and in my opinion this idea here doesn't solve enough problems to make it worth it.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Fri Oct 2 21:48:47 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 19:48:47 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: The MEBS idea is inspired by heroku buildpacks where you just ask a list of tools whether they can build something. https://devcenter.heroku.com/articles/buildpacks . The idea would be that pip would use MEBS instead of its setup.py-focused builder. The first available MEBS plugin would notice setup.py and do what pip does now (force setuptools, build in a subprocess). You should know about flit https://github.com/takluyver/flit and Bento http://cournape.github.io/Bento/ which have their own lightweight metadata formats, which are transformed into standard Python formats by the respective tools. requires.txt is popular but I'm not a fan of it, it seems like it was invented by people who didn't want to have a proper setup.py for their project. We have to come up with something simpler than setup.py if we want to get some of the people who don't understand how to write setup.py. Ideally any required new user-editable "which build system" metadata could be boiled down to a single line in setup.cfg. There would be 3 stages: VCS checkout (minimal metadata, a "generate machine readable metadata" step equivalent to "setup.py egg_info") -> new sdist (PEP 376 style static metadata that can be trusted) -> wheel. (How pip builds a package from source: 1. download sdist; .egg-info directory is almost always present 2. run setup.py egg_info to get dependencies, because the static one is not reliable, because too many requirements lists have 'if' statements 3. compile) For all the talk about static metadata, the build script in general needs to remain a Turing-complete script. Build systems everywhere are programs to build other programs. I really like your idea about returning a list of built artifacts. Python packaging is strictly 1:1 source package -> output package but rpm, deb, can generate many packages from a single source package. I don't think we have to worry that much about Debian & RHEL. They will get over it if setup.py is no longer there. Change brings work but stagnation brings death. On Fri, Oct 2, 2015 at 2:41 PM Brett Cannon wrote: > On Fri, 2 Oct 2015 at 05:08 Donald Stufft wrote: > >> On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: >> > > We realized that actually as far as we could tell, it wouldn't >> > be that >> > hard at this point to clean up how sdists work so that it would be >> > possible to migrate away from distutils. So we wrote up a little >> > draft >> > proposal. >> > >> > The main question is, does this approach seem sound? >> >> I've just read over your proposal, but I've also just woken up so I might >> be >> a little slow still! After reading what you have, I don't think that this >> proposal is the right way to go about improving sdists. >> >> The first thing that immediately stood out to me, is that it's >> recommending >> that downstream redistributors like Debian, Fedora, etc utilize Wheels >> instead >> of the sdist to build their packages from. However, that is not really >> going to >> fly with most (all?) of the downstream redistributors. Debian for >> instance has >> policy that requires the use of building all of it's packages from >> Source, not >> from anything else and Wheels are not a source package. While it can >> theoretically work for pure python packages, it quickly devolves into a >> mess >> when you factor in packages that have any C code what so ever. >> > > So wouldn't they then download the sdist, build a wheel as an > intermediate, and then generate the .deb file? I mean as long as people > upload an sdist for those that want to build from source and a wheel for > convenience -- which is probably what most people providing wheels do > anyway -- then I don't see the problem. > > >> >> Overall, this feels more like a sidegrade than an upgrade. One major theme >> throughout of the PEP is that we're going to push to rely heavily on >> wheels as >> the primary format of installation. While that works well for things like >> Debian, I don't think it's going to work as wheel for us. If we were only >> distributing pure python packages, then yes absolutely, however given >> that we >> are not, we have to worry about ABI issues. Given that there is so many >> different environments that a particular package might be installed into, >> all >> with different ABIs we have to assume that installing from source is still >> going to be a primary path for end users to install and that we are never >> going >> to have a world where we can assume a Wheel in a repository. >> >> One of the problems with the current system, is that we have no mechanism >> by >> which to determine dependencies of a source distribution without >> downloading >> the file and executing some potentially untrusted code. This makes >> dependency >> resolution harder and much much slower than if we could read that >> information >> statically from a source distribution. This PEP doesn't offer anything in >> the >> way of solving this problem. >> > > Isn't that what the requirements and requirements-file fields in the > _pypackage file provide? Only if you use that requirements-dynamic would it > require execcuting arbitrary code to gather dependency information, or am I > missing something? > > >> >> To a similar tune, this PEP also doesn't make it possible to really get at >> any other metadata without executing software. This makes it pratically >> impossible to safely inspect an unknown or untrusted package to determine >> what >> it is and to get information about it. Right now PyPI relies on the >> uploading >> tool to send that information alongside of the file it is uploading, but >> honestly what it should be doing is extracting that information from >> within the >> file. This is sort of possible right now since distutils and setuptools >> both >> create a static metadata file within the source distribution, but we >> don't rely >> on that within PyPI because that information may or may not be accurate >> and may >> or may not exist. However the twine uploading tool *does* rely on that, >> and >> this PEP would break the ability for twine to upload a package without >> executing arbitrary code. >> > > Isn't that only if you use the dynamic fields? > > >> >> Overall, I don't think that this really solves most of the foundational >> problems with the current format. Largely it feels that what it achieves >> is >> shuffling around some logic (you need to create a hook that you reference >> from >> within a .cfg file instead of creating a setuptools extension or so) but >> without fixing most of the problems. The largest benefit I see to >> switching to >> this right now is that it would enable us to have build time dependencies >> that >> were controlled by pip rather than installed implicitly via the execution >> of >> the setup.py. That doesn't feel like a big enough benefit to me to do a >> mass >> shakeup of what we recommend and tell people to do. Having people adjust >> and >> change and do something new requires effort, and we need something to >> justify >> that effort to other people and I don't think that this PEP has something >> we >> can really use to justify that effort. >> > > From my naive perspective this proposal seems to help push forward a > decoupling of building using distutils/setuptools as the only way you can > properly build Python projects (which is what I think we are all after) and > will hopefully eventually free pip up to simply do orchestration. > > >> >> I *do* think that there is a core of some ideas here that are valuable, >> and in >> fact are similar to some ideas I've had. The main flaw I see here is that >> it >> doesn't really fix sdists, it takes a solution that would work for VCS >> checkouts and then reuses it for sdists. In my mind, the supported flow >> for >> package installation would be: >> >> VCS/Bare Directory -> Source Distribution -> Wheel >> >> This would (eventually) be the only path that was supported for >> installation >> but you could "enter" the path at any stage. For example, if there is a >> Wheel >> already available, then you jump right on at the end and just install >> that, if >> there is a sdist available then pip first builds it into a wheel and then >> installs that, etc. >> >> I think your PEP is something like what the VCS/Bare Directory to sdist >> tooling >> could look like, but I don't think it's what the sdist to wheel path >> should >> look like. >> > > Is there another proposal I'm unaware for the sdist -> wheel step that is > build tool-agnostic? I'm all for going with the best solution but there has > to be an actual alternative to compare against and I don't know of any > others right now and this proposal does seem to move things forward in a > reasonable fashion. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 2 22:03:07 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 21:03:07 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 20:02, Marcus Smith wrote: >> So wouldn't they then download the sdist, build a wheel as an >> intermediate, and then generate the .deb file? > > the new goal I think was to have standardized metadata immediately available > in an sdist, and get away from the model, that you had to run a build step, > before you had a metadata artifact. > so here, you'd have to build a wheel (potentially one with binary > extensions) just to know what the metadata is? that doesn't sound right. I'm uncomfortable with the fact that the proposed sdist format has more or less no metadata of its own (even the filename format is only a recommendation) so (for example) if someone does "pip install foo==1.0" I don't see how pip can find a suitable sdist, if no wheel is available. I would rather see an sdist format that can be introspected *without* running code or a build tool. Installers and packaging tools like pip need to be able to do that - one of the biggest issues with pip's current sdist handling is that it can't make any meaningful decisions before building at least the egg-info. Ultimately the question for me is at what point do we require packaging tools like pip (and ad-hoc distribution analysis scripts - I write a lot of those!) to run code from the package in order to continue? I'd like to be able to know, at a minimum, the package name and version, as those are needed to make decisions on whether there is a conflict with an already installed version. Basically, why aren't you using a PEP 426-compatible metadata format in the sdist (there's no reason not to, you just have to mandate that tools used to build sdists generate that form of metadata file)? You could also say that source trees SHOULD store metadata in the _pypackage directory (in an appropriate defined format, maybe one more suited to human editing than JSON) and tools that work on source trees (build tools, things that create sdists) SHOULD use that data as the definitive source of metadata, rather than using their own configuration. I don't see a problem with allowing source trees to have some flexibility, but sdists are tool-generated and so could easily be required to contain static metadata in a standard format. >> Is there another proposal I'm unaware for the sdist -> wheel step that is >> build tool-agnostic? > > PEP426 talks about it some > https://www.python.org/dev/peps/pep-0426/#metabuild-system While the metabuild system is a good long-term goal, I'll take something that's being developed now over a great idea that no-one has time to work on... Wheel came about because Daniel just got on with it. Having said that, I very strongly prefer a sdist proposal that's compatible with PEP 426 (at least to the extent that tools like wheel already support it). Throwing away all of the work already done on PEP 426 doesn't seem like a good plan. Paul From p.f.moore at gmail.com Fri Oct 2 22:14:28 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 21:14:28 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 20:45, Donald Stufft wrote: > n October 2, 2015 at 2:41:20 PM, Brett Cannon (brett at python.org) wrote: >> > Is there another proposal I'm unaware for the sdist -> wheel >> step that is build tool-agnostic? I'm all for going with the best >> solution but there has to be an actual alternative to compare >> against and I don't know of any others right now and this proposal >> does seem to move things forward in a reasonable fashion. > > As Marcus said, it was touched on in PEP 426 but there isn?t a fully fleshed > out PEP (or even a sort of fleshed out PEP) but the high level idea isn?t super > hard, take PEP 426 metadata + some sdist specific metadata and stick it in a > tarball or something and then you have sdist 2.0. Part of that sdist specific > metadata would be describing in some way how to build a wheel out of this sdist > (rather than just assuming distutils/setuptools). > > I think we need to push back against partial solutions to problems that we > have without at least some semblance of a plan for how to solve the rest of > the problems we have, or to explicitly declare them unsolvable. That doesn't > mean that we need to turn every thing into an overengineered mess where we try > to solve every problem ever, but we need to be careful that we minimize churn > where possible. An example just recently where we had this was we accepted > PEP 438 2 years ago as a partial solution to try and "move things forward" and > then it turned out that solution was, well partial, and when we removed the > partial solution for a full solution we had the folks who decided to rely on > the partial solution upset. > > IOW we need to balance between avoiding churn and making a change "worth" it, > and in my opinion this idea here doesn't solve enough problems to make it > worth it. That's a fair point. But I don't see any reason Nathaniel's proposal *couldn't* be that solution. I'd want to see the sdist format required to include static metadata, and the metadata format to be PEP 426, but neither of those seem incompatible with the ideas behind the proposal. Maybe I'm missing something massive, but I don't see a *huge* gap between this proposal and the basic ideas behind the metabuild concept - the sdist should define how to build it (in terms of required packages/tools for the build, and commands/hooks to call to do the specific build steps - make a wheel, do an in-place install). The biggest sticking point would be if Nathaniel is adamantly opposed to static metadata. Is there a good enumeration anywhere of the problems sdist 2.0 / the metabuild system needs to solve? I can think of: 1. Metadata needs to be discoverable without running code. 2. Packages should be able to specify the tools needed to build them. 3. Installers should only interact with sdists through well-defined entry points. 4. Build processes required are (a) create a wheel, (b) do a "develop" style in-place installation. 5. (Maybe) There needs to be better means of handling build errors. The proposal here seems to at least move towards better solutions for 2, 3, and 4. Paul From njs at pobox.com Fri Oct 2 22:19:56 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 13:19:56 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 4:58 AM, Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: >> > We realized that actually as far as we could tell, it wouldn't >> be that >> hard at this point to clean up how sdists work so that it would be >> possible to migrate away from distutils. So we wrote up a little >> draft >> proposal. >> >> The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels instead > of the sdist to build their packages from. However, that is not really going to > fly with most (all?) of the downstream redistributors. Debian for instance has > policy that requires the use of building all of it's packages from Source, not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a mess > when you factor in packages that have any C code what so ever. I think this was addressed downthread -- the idea would be that Debian would build from sdist, with a two step process: convert sdist to wheels, repack wheels into binary .deb. > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism by > which to determine dependencies of a source distribution without downloading > the file and executing some potentially untrusted code. This makes dependency > resolution harder and much much slower than if we could read that information > statically from a source distribution. This PEP doesn't offer anything in the > way of solving this problem. What are the "dependencies of a source distribution"? Do you mean the runtime dependencies of the wheels that will be built from a source distribution? If you need that metadata to be statically in the sdist, then you might as well give up now because it's simply impossible. As the very simplest example, every package that uses the numpy C API gets a runtime dependency on "numpy >= [whatever version happened to be installed on the *build* machine]". There are plenty of more complex examples too (e.g. ones that involve build/configure-time decisions about whether to rely on particular system libraries, or build/configure-time decisions about whether particular packages should even be built). For comparison, here's the Debian source package metadata: https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles Note that the only mandatory fields are format version / package name / package version / maintainer / checksums. The closest they come to making promises about the built packages are the Package-List and Binary fields which provide a optional hint about what binary packages will be built, and are allowed to contain lies (e.g. they explicitly don't guarantee that all the binary packages named will actually be produced on every architecture). The only kind of dependencies that a source package can declare are build-depends. > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine what > it is and to get information about it. Right now PyPI relies on the uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from within the > file. This is sort of possible right now since distutils and setuptools both > create a static metadata file within the source distribution, but we don't rely > on that within PyPI because that information may or may not be accurate and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. Okay, what metadata do you need? We certainly could put name / version kind of stuff in there. We left it out because we weren't sure what was necessary and it's easy to add later, but anything that's needed by twine fits neatly into the existing text saying that we should "include extra metadata in source distributions if it helps solve specific problems that are unique to distribution" -- twine uploads definitely count. > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference from > within a .cfg file instead of creating a setuptools extension or so) but numpy.distutils is the biggest distutils/setuptools extension around, and everyone involved in maintaining it wants to kill it with fire :-). That's a problem... > without fixing most of the problems. The largest benefit I see to switching to > this right now is that it would enable us to have build time dependencies that > were controlled by pip rather than installed implicitly via the execution of > the setup.py. Yes, this problem means that literally every numerical python package currently has a broken setup.py. > That doesn't feel like a big enough benefit to me to do a mass > shakeup of what we recommend and tell people to do. Having people adjust and > change and do something new requires effort, and we need something to justify > that effort to other people and I don't think that this PEP has something we > can really use to justify that effort. The end-user adjustment is teaching people to switch to always using pip to install packages -- this seems like something we will certainly do sooner or later, so we might as well get started. And it's already actually the right thing to do -- if you use 'setup.py install' then you get a timebomb in your venv where later upgrades may leave you with a broken package :-(. (This is orthogonal to the actual PEP.) In the long run, the idea that every package has to contain code that knows how to implement installation in very possible configuration (--user? --single-version-externally-managed?) is clearly broken, and teaching people to use 'pip install' is obviously the only sensible alternative. -n -- Nathaniel J. Smith -- http://vorpus.org From dholth at gmail.com Fri Oct 2 22:24:25 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 02 Oct 2015 20:24:25 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: We need to embrace partial solutions and the fine folks who propose them so the whole packaging ecosystem can have some progress. PEP 438 may not be a good analogue to adding a new sdist format since the latter only adds new things that you can do. A new sdist format will inconvenience a much more limited set of people, mainly the pip authors and the OS package maintainers. Sorry but the section of the PEP that prefixes filenames with _ has distracted the discussion away from the general idea. Instead of multiple hooks why not a single object exposed through an entry point that has several optional methods? NO [build] requirements = "flit" build-wheels = "flit.pypackage_hooks:build_wheels" build-in-place = "flit.pypackage_hooks:build_in_place" YES [build] build-system=flit class ABuildHook: def build_wheels(self): ... entry_points = {'new.sdist.build.hooks': ['flit= some_module:ABuildHook']} On Fri, Oct 2, 2015 at 3:48 PM Daniel Holth wrote: > The MEBS idea is inspired by heroku buildpacks where you just ask a list > of tools whether they can build something. > https://devcenter.heroku.com/articles/buildpacks . The idea would be that > pip would use MEBS instead of its setup.py-focused builder. The first > available MEBS plugin would notice setup.py and do what pip does now (force > setuptools, build in a subprocess). > > You should know about flit https://github.com/takluyver/flit and Bento > http://cournape.github.io/Bento/ which have their own lightweight > metadata formats, which are transformed into standard Python formats by the > respective tools. > > requires.txt is popular but I'm not a fan of it, it seems like it was > invented by people who didn't want to have a proper setup.py for their > project. > > We have to come up with something simpler than setup.py if we want to get > some of the people who don't understand how to write setup.py. Ideally > any required new user-editable "which build system" metadata could be > boiled down to a single line in setup.cfg. There would be 3 stages: VCS > checkout (minimal metadata, a "generate machine readable metadata" step > equivalent to "setup.py egg_info") -> new sdist (PEP 376 style static > metadata that can be trusted) -> wheel. > > (How pip builds a package from source: 1. download sdist; .egg-info > directory is almost always present 2. run setup.py egg_info to get > dependencies, because the static one is not reliable, because too many > requirements lists have 'if' statements 3. compile) > > For all the talk about static metadata, the build script in general needs > to remain a Turing-complete script. Build systems everywhere are programs > to build other programs. > > I really like your idea about returning a list of built artifacts. Python > packaging is strictly 1:1 source package -> output package but rpm, deb, > can generate many packages from a single source package. > > I don't think we have to worry that much about Debian & RHEL. They will > get over it if setup.py is no longer there. Change brings work but > stagnation brings death. > > On Fri, Oct 2, 2015 at 2:41 PM Brett Cannon wrote: > >> On Fri, 2 Oct 2015 at 05:08 Donald Stufft wrote: >> >>> On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) >>> wrote: >>> > > We realized that actually as far as we could tell, it wouldn't >>> > be that >>> > hard at this point to clean up how sdists work so that it would be >>> > possible to migrate away from distutils. So we wrote up a little >>> > draft >>> > proposal. >>> > >>> > The main question is, does this approach seem sound? >>> >>> I've just read over your proposal, but I've also just woken up so I >>> might be >>> a little slow still! After reading what you have, I don't think that this >>> proposal is the right way to go about improving sdists. >>> >>> The first thing that immediately stood out to me, is that it's >>> recommending >>> that downstream redistributors like Debian, Fedora, etc utilize Wheels >>> instead >>> of the sdist to build their packages from. However, that is not really >>> going to >>> fly with most (all?) of the downstream redistributors. Debian for >>> instance has >>> policy that requires the use of building all of it's packages from >>> Source, not >>> from anything else and Wheels are not a source package. While it can >>> theoretically work for pure python packages, it quickly devolves into a >>> mess >>> when you factor in packages that have any C code what so ever. >>> >> >> So wouldn't they then download the sdist, build a wheel as an >> intermediate, and then generate the .deb file? I mean as long as people >> upload an sdist for those that want to build from source and a wheel for >> convenience -- which is probably what most people providing wheels do >> anyway -- then I don't see the problem. >> >> >>> >>> Overall, this feels more like a sidegrade than an upgrade. One major >>> theme >>> throughout of the PEP is that we're going to push to rely heavily on >>> wheels as >>> the primary format of installation. While that works well for things like >>> Debian, I don't think it's going to work as wheel for us. If we were only >>> distributing pure python packages, then yes absolutely, however given >>> that we >>> are not, we have to worry about ABI issues. Given that there is so many >>> different environments that a particular package might be installed >>> into, all >>> with different ABIs we have to assume that installing from source is >>> still >>> going to be a primary path for end users to install and that we are >>> never going >>> to have a world where we can assume a Wheel in a repository. >>> >>> One of the problems with the current system, is that we have no >>> mechanism by >>> which to determine dependencies of a source distribution without >>> downloading >>> the file and executing some potentially untrusted code. This makes >>> dependency >>> resolution harder and much much slower than if we could read that >>> information >>> statically from a source distribution. This PEP doesn't offer anything >>> in the >>> way of solving this problem. >>> >> >> Isn't that what the requirements and requirements-file fields in the >> _pypackage file provide? Only if you use that requirements-dynamic would it >> require execcuting arbitrary code to gather dependency information, or am I >> missing something? >> >> >>> >>> To a similar tune, this PEP also doesn't make it possible to really get >>> at >>> any other metadata without executing software. This makes it pratically >>> impossible to safely inspect an unknown or untrusted package to >>> determine what >>> it is and to get information about it. Right now PyPI relies on the >>> uploading >>> tool to send that information alongside of the file it is uploading, but >>> honestly what it should be doing is extracting that information from >>> within the >>> file. This is sort of possible right now since distutils and setuptools >>> both >>> create a static metadata file within the source distribution, but we >>> don't rely >>> on that within PyPI because that information may or may not be accurate >>> and may >>> or may not exist. However the twine uploading tool *does* rely on that, >>> and >>> this PEP would break the ability for twine to upload a package without >>> executing arbitrary code. >>> >> >> Isn't that only if you use the dynamic fields? >> >> >>> >>> Overall, I don't think that this really solves most of the foundational >>> problems with the current format. Largely it feels that what it achieves >>> is >>> shuffling around some logic (you need to create a hook that you >>> reference from >>> within a .cfg file instead of creating a setuptools extension or so) but >>> without fixing most of the problems. The largest benefit I see to >>> switching to >>> this right now is that it would enable us to have build time >>> dependencies that >>> were controlled by pip rather than installed implicitly via the >>> execution of >>> the setup.py. That doesn't feel like a big enough benefit to me to do a >>> mass >>> shakeup of what we recommend and tell people to do. Having people adjust >>> and >>> change and do something new requires effort, and we need something to >>> justify >>> that effort to other people and I don't think that this PEP has >>> something we >>> can really use to justify that effort. >>> >> >> From my naive perspective this proposal seems to help push forward a >> decoupling of building using distutils/setuptools as the only way you can >> properly build Python projects (which is what I think we are all after) and >> will hopefully eventually free pip up to simply do orchestration. >> >> >>> >>> I *do* think that there is a core of some ideas here that are valuable, >>> and in >>> fact are similar to some ideas I've had. The main flaw I see here is >>> that it >>> doesn't really fix sdists, it takes a solution that would work for VCS >>> checkouts and then reuses it for sdists. In my mind, the supported flow >>> for >>> package installation would be: >>> >>> VCS/Bare Directory -> Source Distribution -> Wheel >>> >>> This would (eventually) be the only path that was supported for >>> installation >>> but you could "enter" the path at any stage. For example, if there is a >>> Wheel >>> already available, then you jump right on at the end and just install >>> that, if >>> there is a sdist available then pip first builds it into a wheel and then >>> installs that, etc. >>> >>> I think your PEP is something like what the VCS/Bare Directory to sdist >>> tooling >>> could look like, but I don't think it's what the sdist to wheel path >>> should >>> look like. >>> >> >> Is there another proposal I'm unaware for the sdist -> wheel step that is >> build tool-agnostic? I'm all for going with the best solution but there has >> to be an actual alternative to compare against and I don't know of any >> others right now and this proposal does seem to move things forward in a >> reasonable fashion. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 22:25:14 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 16:25:14 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: <3A3FFB47-34F3-4F46-8B7F-FDFD184BFE8A@stufft.io> I'm speaking to the proposal as currently written. It's not completely off base for what I think a solution could be. I think part of the problem though is we don't have all the building blocks figured out and standardized yet. PEP426 has stalled (I plan to pick it up once Warehouse is deployed but someone else could do that) and we should probably get the environment markers sorted out because they are going to be even more important for a static sdist. I think that the current proposal conflates a bcs checkout with a sdist too much. As Paul said, sdists are generated and are not generally for human consumption or creation. We should strictly define what it looks like l, but have pluggable build systems. I don't think we need anything more complex than the ability for a sdist to say that it gets built using X hook. Give that hook a standard API and then any tool can be a first class build tool. Sent from my iPhone > On Oct 2, 2015, at 4:14 PM, Paul Moore wrote: > > That's a fair point. But I don't see any reason Nathaniel's proposal > *couldn't* be that solution. I'd want to see the sdist format required > to include static metadata, and the metadata format to be PEP 426, but > neither of those seem incompatible with the ideas behind the proposal. > > Maybe I'm missing something massive, but I don't see a *huge* gap > between this proposal and the basic ideas behind the metabuild concept From njs at pobox.com Fri Oct 2 22:27:49 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 13:27:49 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 1:24 PM, Daniel Holth wrote: > Instead of multiple hooks why not a single object exposed through an entry > point that has several optional methods? > > NO > > [build] > requirements = "flit" > build-wheels = "flit.pypackage_hooks:build_wheels" > build-in-place = "flit.pypackage_hooks:build_in_place" > > YES > > [build] > build-system=flit > > class ABuildHook: > def build_wheels(self): ... > > entry_points = {'new.sdist.build.hooks': ['flit= some_module:ABuildHook']} Mostly because this rules out the possibility of shipping the build hook inside the package being built, which seems like a useful option to leave open (e.g. some people will probably want to vendor their build system for more-or-less legitimate reasons). Notice that the _pypackage directory is added to sys.path before resolving hooks. -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Fri Oct 2 22:42:13 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 21:42:13 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 21:19, Nathaniel Smith wrote: >> One of the problems with the current system, is that we have no mechanism by >> which to determine dependencies of a source distribution without downloading >> the file and executing some potentially untrusted code. This makes dependency >> resolution harder and much much slower than if we could read that information >> statically from a source distribution. This PEP doesn't offer anything in the >> way of solving this problem. > > What are the "dependencies of a source distribution"? Do you mean the > runtime dependencies of the wheels that will be built from a source > distribution? > > If you need that metadata to be statically in the sdist, then you > might as well give up now because it's simply impossible. > > As the very simplest example, every package that uses the numpy C API > gets a runtime dependency on "numpy >= [whatever version happened to > be installed on the *build* machine]". There are plenty of more > complex examples too (e.g. ones that involve build/configure-time > decisions about whether to rely on particular system libraries, or > build/configure-time decisions about whether particular packages > should even be built). I'm really not at all clear what you're saying here. It's quite possible that those of us who don't understand the complexities of the scientific/numpy world are missing something important, but if so it would be useful if you could spell out the problems in detail. >From my point of view, it's not a source distribution or a binary distribution that depends on something (numpy or whatever) - it's the *project*. If project foo needs numpy to work, it depends on numpy. If it depends on features in numpy 1.9, it depends on numpy>=1.9. Optional dependencies are covered by extras, and environment specific dependencies are covered by environment markers.[1] That remains true for all wheels that are built from that project, for whatever platform using whatever tools. It should also be true for the source distribution, precisely *because* it's independent of the build process. I can understand that a binary wheel may need a certain set of libraries installed - but that's about the platform tags that are part of the wheel definition, not about dependencies. Platform tags are an ongoing discussion, and a good example of a partial solution that needs to be extended, certainly, but they aren't really relevant in any way that I can see to how the build chain works. You seem to be saying that wheels need a dependency on "the version of numpy they were built against". That sounds to me like a binary compatibility requirement that platform tags are intended to cover. It may well be a requirement that platform tags need significant enhancement (maybe even redesign) to cover, but it's not a dependency in the sense that pip and the packaging PEPs use the term. And if my understanding is correct, I'm against trying to fit that information into a dependency simply to work around the current limitations of the platform tag mechanism. I'm all in favour of new initiatives to make progress in areas that are currently stalled (we definitely need people willing to contribute) but we really don't have the resources to throw away the progress we've already made. Even if some of the packaging PEPs are still works in progress, what is there represents an investment we need to build on, not bypass. Paul [1] If extras and environment markers don't cover the needs of scientific modules, we need some input into their design from the scientific community. But again, let's not throw away the work that's already done. From chris.barker at noaa.gov Fri Oct 2 23:03:33 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 14:03:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: I'm not sure if I understand what Nathaniel is getting at, but... > As the very simplest example, every package that uses the numpy C API > > gets a runtime dependency on "numpy >= [whatever version happened to > > be installed on the *build* machine]". > From my point of view, it's not a source distribution or a binary > distribution that depends on something (numpy or whatever) - it's the > *project*. If project foo needs numpy to work, it depends on numpy. If > it depends on features in numpy 1.9, it depends on numpy>=1.9. > Here is the gap (I'm a bit confused about what a "project" is -- so I"ll use the term "package", meaning a python package). A given package might depend on numpy, as you say, and it may work with all numpy versions 1.6 to 1.9. Fine, so we specify that in install_requires. And this shodl be the dependency in the sdist, too. If the package is pur python, this is fine and done. But if the package has some extensions code that used the numpy C API ( a very common occurrence), then when it is built, it will only work (reliably) with the version of numpy it was built with. So the project itself, and the sdist depend on numpy >=1.6, but a build binary wheel depends on numpy == 1.7 (for instance). Which requires a binary (wheel) dependency that is somewhat different than the source dependency. In a way, this is a lot like a python package that may work fine on py2.7 to py3.5, but a binary wheel is for py3.4 specifically (yes?) Note that conda, being developed originally for scipy, has packages like: gsw-3.0.3-np19py27_0.tar.bz2 so this binary depends specifically on py2.7 and numpy1.9 but once you get beyond python_numpy, you cold get a REALLY BIG matrix of possible version combnations! nevertheless, I think it would be really helpful to have the concept of a binary dependency that is distinct from the "package" dependency. Maybe down to "this wheel depends on this other particular wheel" - not only version, but also wheel specific. That would open the door to making wheels for binary non-python dependencies like libpng: then other wheels would depend on the libpng wheel, but different wheels built from the same package might have those dependencies handled differently, particularly on different platforms. I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that > needs to be extended, certainly, but they aren't really relevant in > any way that I can see to how the build chain works. > I haven't followed the discussion well, but I really think platform tags are never going to be specific enough to handle these use-cases And these really ARE dependencies, even if they aren't pure-python ones... And if my > understanding is correct, I'm against trying to fit that information > into a dependency simply to work around the current limitations of the > platform tag mechanism. > I can't imagine how you could extend platform tags to cover all of this -- but maybe I'm being unimaginative... BTW, numpy is nearly unique here -- I can't think of any other package that is a python package with a C API that is widely used -- most other cases are either a python package OR a regular old compiled lib. Which is probably why conda can get away with essentially special casing it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 23:30:29 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 17:30:29 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 4:20:00 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Fri, Oct 2, 2015 at 4:58 AM, Donald Stufft wrote: > > > > One of the problems with the current system, is that we have no mechanism by > > which to determine dependencies of a source distribution without downloading > > the file and executing some potentially untrusted code. This makes dependency > > resolution harder and much much slower than if we could read that information > > statically from a source distribution. This PEP doesn't offer anything in the > > way of solving this problem. > > What are the "dependencies of a source distribution"? Do you mean the > runtime dependencies of the wheels that will be built from a source > distribution? > > If you need that metadata to be statically in the sdist, then you > might as well give up now because it's simply impossible. I don?t believe this is impossible. > > As the very simplest example, every package that uses the numpy C API > gets a runtime dependency on "numpy >= [whatever version happened to > be installed on the *build* machine]?. A quick, off the cuff idea here is to allow additional ABI declarations and stop trying to use the same system for API and ABI. A source distribution can't have ABI dependencies, only wheels can and an installation has to be valid for both API and any relevant ABI requirements. > There are plenty of more > complex examples too (e.g. ones that involve build/configure-time > decisions about whether to rely on particular system libraries, or > build/configure-time decisions about whether particular packages > should even be built). I don't think build/configure-time decisions are great ideas as it's near impossible to actually depend on them. For example, take Pillow, Pillow will conditionally compile against libraries that enable it to much around with PNGs. However, if I *need* Pillow with PNG support, I don't have any mechanism to declare that. If instead, builds were *not* conditional and Pillow instead split it's PNG capabilities out into it's own package called say, Pillow-PNG which also did not conditionally compile against anything, but unconditionally did, then we could add in something like having Pillow declare a "weak" dependency on Pillow-PNG where we attempt to get it by default if possible, but we will skip installing it if we can't locate/build it. If you combine this with Extras, you could then easily make it so that people can depend on particular conditional features by doing something like ``Pillow[PNG]`` in their dependency metadata. > > For comparison, here's the Debian source package metadata: > https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles > Note that the only mandatory fields are format version / package name > / package version / maintainer / checksums. The closest they come to > making promises about the built packages are the Package-List and > Binary fields which provide a optional hint about what binary packages > will be built, and are allowed to contain lies (e.g. they explicitly > don't guarantee that all the binary packages named will actually be > produced on every architecture). The only kind of dependencies that a > source package can declare are build-depends. Debian doesn't really have "source packages" like we do, but inside of the debian/ directory is the control file which lists all of the dependency information (or explicitly lists a placeholder where something can't be statically declared). > > > To a similar tune, this PEP also doesn't make it possible to really get at > > any other metadata without executing software. This makes it pratically > > impossible to safely inspect an unknown or untrusted package to determine what > > it is and to get information about it. Right now PyPI relies on the uploading > > tool to send that information alongside of the file it is uploading, but > > honestly what it should be doing is extracting that information from within the > > file. This is sort of possible right now since distutils and setuptools both > > create a static metadata file within the source distribution, but we don't rely > > on that within PyPI because that information may or may not be accurate and may > > or may not exist. However the twine uploading tool *does* rely on that, and > > this PEP would break the ability for twine to upload a package without > > executing arbitrary code. > > Okay, what metadata do you need? We certainly could put name / version > kind of stuff in there. We left it out because we weren't sure what > was necessary and it's easy to add later, but anything that's needed > by twine fits neatly into the existing text saying that we should > "include extra metadata in source distributions if it helps solve > specific problems that are unique to distribution" -- twine uploads > definitely count. Everything that isn't specific to a built wheel. Look at the previously accepted metadata specs as well as PEP 426. If you're not including a field that was included in one of those, there should be a rationale for why that field is no longer being included. > > > Overall, I don't think that this really solves most of the foundational > > problems with the current format. Largely it feels that what it achieves is > > shuffling around some logic (you need to create a hook that you reference from > > within a .cfg file instead of creating a setuptools extension or so) but > > numpy.distutils is the biggest distutils/setuptools extension around, > and everyone involved in maintaining it wants to kill it with fire > :-). That's a problem? Well, it's not really what I call setuptools extension because it doesn't use the extension points in setuptools to do it's work. It expects you to just ``import numpy.distutils`` at the top of your ``setup.py`` and use that, which means that it breaks things like pip because we don't have a way to know that we need to install numpy first. > > > without fixing most of the problems. The largest benefit I see to switching to > > this right now is that it would enable us to have build time dependencies that > > were controlled by pip rather than installed implicitly via the execution of > > the setup.py. > > Yes, this problem means that literally every numerical python package > currently has a broken setup.py. Because numpy.distutils wasn't written to plug into setuptools. If it had been they wouldn't be.? > > > That doesn't feel like a big enough benefit to me to do a mass > > shakeup of what we recommend and tell people to do. Having people adjust and > > change and do something new requires effort, and we need something to justify > > that effort to other people and I don't think that this PEP has something we > > can really use to justify that effort. > > The end-user adjustment is teaching people to switch to always using > pip to install packages -- this seems like something we will certainly > do sooner or later, so we might as well get started. > > And it's already actually the right thing to do -- if you use > 'setup.py install' then you get a timebomb in your venv where later > upgrades may leave you with a broken package :-(. (This is orthogonal > to the actual PEP.) In the long run, the idea that every package has > to contain code that knows how to implement installation in very > possible configuration (--user? --single-version-externally-managed?) > is clearly broken, and teaching people to use 'pip install' is > obviously the only sensible alternative. > Sorry, ambigous end-user in my original statement. I don't mean the end-end-users (e.g. the people executing ``pip install``), I mean the packagers. I'm going to re-read the originall proposal and try to point out more actionable feedback shortly. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Fri Oct 2 23:36:00 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 17:36:00 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 4:24:38 PM, Daniel Holth (dholth at gmail.com) wrote: > > We need to embrace partial solutions and the fine folks who propose > them so the whole packaging ecosystem can have some progress. > PEP 438 may not be a good analogue to adding a new sdist format since > the latter only adds new things that you can do. A new sdist format > will inconvenience a much more limited set of people, mainly > the pip authors and the OS package maintainers. Packaging formats are a bit like HTTP, "move fast and break things" isn't super great because anytime you add a new format, you have to support that *forever* (or long enough to basically be forever). It's unlike a software package where you can, for example, deprecate and remove something and if someone is still using that feature they just continue to use an older version of your library. Packagers have no control over what version of the tools people are using and they are discouraged (or even disallowed) to go back in time and correct their older packages. Being conservative in what we accept as an accepted standard and implement in the main tools is a good thing, being liberal in experiments and what we try and having people try out partial solutions and figure out what works and what doesn't without making it a standard or putting it in pip is also a good thing. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Fri Oct 2 23:37:29 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 14:37:29 -0700 Subject: [Distutils] pip install from source dir?? Message-ID: I can't seem to find a way to pip install from a source dir -- not a tarball. Why would I want to do that? In this case, it's because I'm trying to build conda packages for python packages that have compiled C code, and are not using setuptools. On Windows, you need to use setuptools in order to use the "MS compiler for Python 2.7". So a straight "setup.py build" fails. But pip injects setuptools, so that "pip install" does work. But conda has already unpacked the source distribution, so I'd like to point pip at that, and can't find a way to do that (I could have sworn it was possible...) Do I have to build an sdist, and then point pip at that?? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 2 23:39:05 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 17:39:05 -0400 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: ``pip install path/to/directory/with/setup.py/in/it/`` On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) wrote: > I can't seem to find a way to pip install from a source dir -- not a > tarball. > > Why would I want to do that? > > In this case, it's because I'm trying to build conda packages for python > packages that have compiled C code, and are not using setuptools. > > On Windows, you need to use setuptools in order to use the "MS compiler for > Python 2.7". So a straight "setup.py build" fails. > > But pip injects setuptools, so that "pip install" does work. But conda has > already unpacked the source distribution, so I'd like to point pip at that, > and can't find a way to do that (I could have sworn it was possible...) > > Do I have to build an sdist, and then point pip at that?? > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From njs at pobox.com Fri Oct 2 23:45:55 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 14:45:55 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 1:03 PM, Paul Moore wrote: > On 2 October 2015 at 20:02, Marcus Smith wrote: >>> So wouldn't they then download the sdist, build a wheel as an >>> intermediate, and then generate the .deb file? >> >> the new goal I think was to have standardized metadata immediately available >> in an sdist, and get away from the model, that you had to run a build step, >> before you had a metadata artifact. >> so here, you'd have to build a wheel (potentially one with binary >> extensions) just to know what the metadata is? that doesn't sound right. > > I'm uncomfortable with the fact that the proposed sdist format has > more or less no metadata of its own (even the filename format is only > a recommendation) so (for example) if someone does "pip install > foo==1.0" I don't see how pip can find a suitable sdist, if no wheel > is available. About the filename thing: The reason that the draft makes the inclusion of package/version info a SHOULD instead of a MUST is that regardless of what the spec says, all decent installation tools are going to support doing things like curl https://github.com/numpy/numpy/archive/master.zip -O numpy-master.zip pip install numpy-master.zip So we can either handle that by saying that "numpy-master.zip" is an sdist, just not one that we would allow on PyPI (which is what the current draft does), or we could handle it by saying that numpy-master.zip is almost-but-not-quite an sdist, and handling it is a commonly supported extension to the standard. Doesn't really matter that much either way -- just a matter of terminology. Either way the sdists are PyPI are obviously going to be named -.. For sdists that do have a name/version: it's not really crucial to the proposal that name/version are only in the filename -- they could be repeated inside the file as well. Given that the version number in particular is something that usually would need to be computed at sdist-build-time (from a __version__.py or whatever -- it's very common that the source-of-truth for version numbers is not static), then leaving it out of the static metadata is nice because it makes sdist-building-code much simpler -- 90% of the time you could just keep the static metadata file instead of having to rewrite it for each sdist. But that's just an engineering trade-off, it's not crucial to the concept. > I would rather see an sdist format that can be introspected *without* > running code or a build tool. Installers and packaging tools like pip > need to be able to do that - one of the biggest issues with pip's > current sdist handling is that it can't make any meaningful decisions > before building at least the egg-info. Another way to look at this is to say that pip's current handling is proof that the build-to-get-metadata strategy is viable :-). It would indeed be nice if this weren't necessary, but the python packaging ecosystem has a long history of trying to make simplifying assumptions that turn out to bite us later... I think this is one of those. Note that for making installation decisions, name + version aren't enough: you also need full dependency information. And dependency information is definitely not fixed at sdist-creation-time. > Ultimately the question for me is at what point do we require > packaging tools like pip (and ad-hoc distribution analysis scripts - I > write a lot of those!) to run code from the package in order to > continue? I'd like to be able to know, at a minimum, the package name > and version, as those are needed to make decisions on whether there is > a conflict with an already installed version. > > Basically, why aren't you using a PEP 426-compatible metadata format > in the sdist (there's no reason not to, you just have to mandate that > tools used to build sdists generate that form of metadata file)? You > could also say that source trees SHOULD store metadata in the > _pypackage directory (in an appropriate defined format, maybe one more > suited to human editing than JSON) and tools that work on source trees > (build tools, things that create sdists) SHOULD use that data as the > definitive source of metadata, rather than using their own > configuration. I don't see a problem with allowing source trees to > have some flexibility, but sdists are tool-generated and so could > easily be required to contain static metadata in a standard format. > >>> Is there another proposal I'm unaware for the sdist -> wheel step that is >>> build tool-agnostic? >> >> PEP426 talks about it some >> https://www.python.org/dev/peps/pep-0426/#metabuild-system > > While the metabuild system is a good long-term goal, I'll take > something that's being developed now over a great idea that no-one has > time to work on... Wheel came about because Daniel just got on with > it. > > Having said that, I very strongly prefer a sdist proposal that's > compatible with PEP 426 (at least to the extent that tools like wheel > already support it). Throwing away all of the work already done on PEP > 426 doesn't seem like a good plan. Nothing is being thrown away -- the proposal is just that sdists and wheels are different things, so we should think of PEP 426 as wheel metadata, rather than all metadata. -n -- Nathaniel J. Smith -- http://vorpus.org From chris.barker at noaa.gov Fri Oct 2 23:49:14 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 14:49:14 -0700 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:39 PM, Donald Stufft wrote: > ``pip install path/to/directory/with/setup.py/in/it/`` WOW!! Sorry for being such an idiot -- I tried that, I really did -- a couple times, and it failed. but now it works fine, just like you'd expect. I have no idea what was doing before. a small doc patch might be in order -- I actually couldn't find that anywhere. Though looking now, I see: pip install [options] [-e] ... I must have gotten that tied to the -e option in my head, and didn't notice that the -e is optional... Sorry for the noise: Thanks, -CHB > On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) > wrote: > > I can't seem to find a way to pip install from a source dir -- not a > > tarball. > > > > Why would I want to do that? > > > > In this case, it's because I'm trying to build conda packages for python > > packages that have compiled C code, and are not using setuptools. > > > > On Windows, you need to use setuptools in order to use the "MS compiler > for > > Python 2.7". So a straight "setup.py build" fails. > > > > But pip injects setuptools, so that "pip install" does work. But conda > has > > already unpacked the source distribution, so I'd like to point pip at > that, > > and can't find a way to do that (I could have sworn it was possible...) > > > > Do I have to build an sdist, and then point pip at that?? > > > > -CHB > > > > > > > > -- > > > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker at noaa.gov > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Oct 3 00:00:48 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 15:00:48 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:45 PM, Nathaniel Smith wrote: > > I would rather see an sdist format that can be introspected *without* > > running code or a build tool. indeed -- this has come up a lot on this list, for binary dists, too, of course. but "build script as turing complete" [someone posted that in this thread...] requirement is there, too... but maybe we can get a long way with convention, without changing the tools. A setup.py is typically a bunc of stuff that builds up the setup, and then a call to setup.py, passing in teh objects created to various paramters. But if we tried to establish a convention that your setup.py would do: a_bunch_of_stuff_that_bulit_up_a_dict_of_options then setup(**setup_options_dict) then the_options_dict could be introspected without actually createing a seteup object. and in the easy cases, the_options_dict could be completely declarative, and maybe even stored in another file. and in the complex cases, it could still have all the parts that could be pre-declared declarative, so setup.py would be: setup_options_dict = load_options_dict("setup_options.txt") [do assorted complex stuff to edit/add to setup_options_dict] setup(**setup_options) would this move us toward an as-declarative-as-possible setup.py? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 3 00:15:38 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 15:15:38 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 1:42 PM, Paul Moore wrote: > On 2 October 2015 at 21:19, Nathaniel Smith wrote: >>> One of the problems with the current system, is that we have no mechanism by >>> which to determine dependencies of a source distribution without downloading >>> the file and executing some potentially untrusted code. This makes dependency >>> resolution harder and much much slower than if we could read that information >>> statically from a source distribution. This PEP doesn't offer anything in the >>> way of solving this problem. >> >> What are the "dependencies of a source distribution"? Do you mean the >> runtime dependencies of the wheels that will be built from a source >> distribution? >> >> If you need that metadata to be statically in the sdist, then you >> might as well give up now because it's simply impossible. >> >> As the very simplest example, every package that uses the numpy C API >> gets a runtime dependency on "numpy >= [whatever version happened to >> be installed on the *build* machine]". There are plenty of more >> complex examples too (e.g. ones that involve build/configure-time >> decisions about whether to rely on particular system libraries, or >> build/configure-time decisions about whether particular packages >> should even be built). > > I'm really not at all clear what you're saying here. It's quite > possible that those of us who don't understand the complexities of the > scientific/numpy world are missing something important, but if so it > would be useful if you could spell out the problems in detail. > > From my point of view, it's not a source distribution or a binary > distribution that depends on something (numpy or whatever) - it's the > *project*. If project foo needs numpy to work, it depends on numpy. If > it depends on features in numpy 1.9, it depends on numpy>=1.9. > Optional dependencies are covered by extras, and environment specific > dependencies are covered by environment markers.[1] That remains true > for all wheels that are built from that project, for whatever platform > using whatever tools. It should also be true for the source > distribution, precisely *because* it's independent of the build > process. "Project" is a pretty messy concept. Obviously in simple cases there's a one-to-one mapping between project <-> wheel <-> importable package, but this breaks down quickly in edge cases. Consider a project that provides builds multiple wheels out of the same source tree. You obviously can't expect that all of these packages will have the same dependencies. This situation is not common today for Python packages, but the only reason for that is that distutils makes it really hard to do -- it's extremely common in other package ecosystems, and the advantages are obvious. E.g., maybe numpy.distutils should be split into a separately installable package from numpy -- there's no technical reason that this should mean we are now forced to move the code for it into its own VCS repository. > I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that > needs to be extended, certainly, but they aren't really relevant in > any way that I can see to how the build chain works. (I assume that by "platform tags" you mean what PEP 426 calls "environment markers".) Environment markers are really useful for extending the set of cases that can be handled by a single architecture-dependent wheel. And they're a good fit for that environment, given that wheels can't contain arbitrary code. But they're certainly never going to be adequate to provide a single static description of every possible build configuration of every possible project. And installing an sdist already requires arbitrary code execution, so it doesn't make sense to try to build some elaborate system to avoid arbitrary code execution just for the dependency specification. You're right that in a perfect future world numpy C API related dependencies would be handling by some separate ABI-tracking mechanism similar to how the CPython ABI is tracked, so here are some other examples of why environment markers are inadequate: In the future it will almost certainly be possible to build numpy in two different configurations: one where it expects to find BLAS inside a wheel distributed for this purpose (e.g. this is necessary to provide high-quality windows wheels), and one where it expects to find BLAS installed on the system. This decision will *not* be tied to the platform, but be selectable at build time. E.g., on OS X there is a system-provided BLAS library, but it has some issues. So the default wheels on PyPI will probably act like windows and depend on a BLAS-package that we control, but there will also be individual users who prefer to build numpy in the configuration where it uses the system BLAS, so we definitely need to support both options on OS X. Now the problem: There will never be a single environment marker that you can stick into a wheel or sdist that says "we depend on the 'pyblas' package if the system is OS X (ok) and the user set this flag in this configuration file during the build process (wait wut)". Similarly, I think someone was saying in a discussion recently that lxml supports being built either in a mode where it requires libxml be available on the system, or else it can be statically linked. Even if in the future we start having metadata that lets us describe dependencies on external system libraries, it's never going to be the case that we can put the *same* dependency metadata into wheels that are built using these two configurations. > You seem to be saying that wheels need a dependency on "the version of > numpy they were built against". That sounds to me like a binary > compatibility requirement that platform tags are intended to cover. It > may well be a requirement that platform tags need significant > enhancement (maybe even redesign) to cover, but it's not a dependency > in the sense that pip and the packaging PEPs use the term. And if my > understanding is correct, I'm against trying to fit that information > into a dependency simply to work around the current limitations of the > platform tag mechanism. > > I'm all in favour of new initiatives to make progress in areas that > are currently stalled (we definitely need people willing to > contribute) but we really don't have the resources to throw away the > progress we've already made. Even if some of the packaging PEPs are > still works in progress, what is there represents an investment we > need to build on, not bypass. > > Paul > > [1] If extras and environment markers don't cover the needs of > scientific modules, we need some input into their design from the > scientific community. But again, let's not throw away the work that's > already done. As far as sdists go, you can either cover 90% of the cases by building increasingly elaborate metadata formats, or you can cover 100% of the cases by keeping things simple... -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sat Oct 3 00:17:57 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 15:17:57 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:36 PM, Donald Stufft wrote: > On October 2, 2015 at 4:24:38 PM, Daniel Holth (dholth at gmail.com) wrote: >> > We need to embrace partial solutions and the fine folks who propose >> them so the whole packaging ecosystem can have some progress. >> PEP 438 may not be a good analogue to adding a new sdist format since >> the latter only adds new things that you can do. A new sdist format >> will inconvenience a much more limited set of people, mainly >> the pip authors and the OS package maintainers. > > Packaging formats are a bit like HTTP, "move fast and break things" isn't super > great because anytime you add a new format, you have to support that *forever* > (or long enough to basically be forever). Right: this is why it's important for me to make the case that putting full PEP 426 metadata in sdists is not just temporarily inconvenient, but actually conceptually the wrong thing to do. -n -- Nathaniel J. Smith -- http://vorpus.org From wes.turner at gmail.com Sat Oct 3 00:26:07 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 2 Oct 2015 17:26:07 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 2, 2015 5:18 PM, "Nathaniel Smith" wrote: > > On Fri, Oct 2, 2015 at 2:36 PM, Donald Stufft wrote: > > On October 2, 2015 at 4:24:38 PM, Daniel Holth (dholth at gmail.com) wrote: > >> > We need to embrace partial solutions and the fine folks who propose > >> them so the whole packaging ecosystem can have some progress. > >> PEP 438 may not be a good analogue to adding a new sdist format since > >> the latter only adds new things that you can do. A new sdist format > >> will inconvenience a much more limited set of people, mainly > >> the pip authors and the OS package maintainers. > > > > Packaging formats are a bit like HTTP, "move fast and break things" isn't super > > great because anytime you add a new format, you have to support that *forever* > > (or long enough to basically be forever). > > Right: this is why it's important for me to make the case that putting > full PEP 426 metadata in sdists is not just temporarily inconvenient, > but actually conceptually the wrong thing to do. pydist.jsonld would be a helpful metadata file to add to an sdist, as well URIs to dependencies with rule/constraints in the reified edges drawn from e.g. - setup.py - requirements.txt - requirements.lock/versions/freeze.txt - requirements.peep.txt - requirements-dev/test/docs.txt - [versions.cfg] > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Oct 3 00:26:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 23:26:47 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 22:45, Nathaniel Smith wrote: >> I'm uncomfortable with the fact that the proposed sdist format has >> more or less no metadata of its own (even the filename format is only >> a recommendation) so (for example) if someone does "pip install >> foo==1.0" I don't see how pip can find a suitable sdist, if no wheel >> is available. > > About the filename thing: > > The reason that the draft makes the inclusion of package/version info > a SHOULD instead of a MUST is that regardless of what the spec says, > all decent installation tools are going to support doing things like > > curl https://github.com/numpy/numpy/archive/master.zip -O numpy-master.zip > pip install numpy-master.zip > > So we can either handle that by saying that "numpy-master.zip" is an > sdist, just not one that we would allow on PyPI (which is what the > current draft does), or we could handle it by saying that > numpy-master.zip is almost-but-not-quite an sdist, and handling it is > a commonly supported extension to the standard. Doesn't really matter > that much either way -- just a matter of terminology. Either way the > sdists are PyPI are obviously going to be named > -.. OK, that's a good point, and I never felt it was crucial that the name/version be encoded in the filename. But having them in some form of static metadata should be mandatory. Your _pypackage.cfg doesn't contain the package name or version, so how would I get them without running code? That's my real point. > Given that the version number in > particular is something that usually would need to be computed at > sdist-build-time (from a __version__.py or whatever -- it's very > common that the source-of-truth for version numbers is not static), > then leaving it out of the static metadata is nice because it makes > sdist-building-code much simpler -- 90% of the time you could just > keep the static metadata file instead of having to rewrite it for each > sdist. But that's just an engineering trade-off, it's not crucial to > the concept. I'm willing to allow for it being non-static in the source tree, but not in the sdist. >> I would rather see an sdist format that can be introspected *without* >> running code or a build tool. Installers and packaging tools like pip >> need to be able to do that - one of the biggest issues with pip's >> current sdist handling is that it can't make any meaningful decisions >> before building at least the egg-info. > > Another way to look at this is to say that pip's current handling is > proof that the build-to-get-metadata strategy is viable :-). Not if you look at the bug reports for pip that can be traced back to needing to run setup.py egg-info to get metadata, or other variations on not having static introspectable metadata in sdists. Paul From erik.m.bray at gmail.com Sat Oct 3 00:37:00 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 2 Oct 2015 18:37:00 -0400 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 5:39 PM, Donald Stufft wrote: > ``pip install path/to/directory/with/setup.py/in/it/`` In particular I install most of my packages from source these days by cd'ing into the source and $ pip install . That's all it takes. Works with -e too. Erik > On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) wrote: >> I can't seem to find a way to pip install from a source dir -- not a >> tarball. >> >> Why would I want to do that? >> >> In this case, it's because I'm trying to build conda packages for python >> packages that have compiled C code, and are not using setuptools. >> >> On Windows, you need to use setuptools in order to use the "MS compiler for >> Python 2.7". So a straight "setup.py build" fails. >> >> But pip injects setuptools, so that "pip install" does work. But conda has >> already unpacked the source distribution, so I'd like to point pip at that, >> and can't find a way to do that (I could have sworn it was possible...) >> >> Do I have to build an sdist, and then point pip at that?? >> >> -CHB >> >> >> From njs at pobox.com Sat Oct 3 00:47:15 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 15:47:15 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 3:26 PM, Paul Moore wrote: > On 2 October 2015 at 22:45, Nathaniel Smith wrote: >>> I'm uncomfortable with the fact that the proposed sdist format has >>> more or less no metadata of its own (even the filename format is only >>> a recommendation) so (for example) if someone does "pip install >>> foo==1.0" I don't see how pip can find a suitable sdist, if no wheel >>> is available. >> >> About the filename thing: >> >> The reason that the draft makes the inclusion of package/version info >> a SHOULD instead of a MUST is that regardless of what the spec says, >> all decent installation tools are going to support doing things like >> >> curl https://github.com/numpy/numpy/archive/master.zip -O numpy-master.zip >> pip install numpy-master.zip >> >> So we can either handle that by saying that "numpy-master.zip" is an >> sdist, just not one that we would allow on PyPI (which is what the >> current draft does), or we could handle it by saying that >> numpy-master.zip is almost-but-not-quite an sdist, and handling it is >> a commonly supported extension to the standard. Doesn't really matter >> that much either way -- just a matter of terminology. Either way the >> sdists are PyPI are obviously going to be named >> -.. > > OK, that's a good point, and I never felt it was crucial that the > name/version be encoded in the filename. But having them in some form > of static metadata should be mandatory. Your _pypackage.cfg doesn't > contain the package name or version, so how would I get them without > running code? That's my real point. Well, first, it's just not possible for a devel snapshot like numpy-master.zip or a VCS checkout to contain static version metadata, since the actual version of the generated wheels *will* be determined by running arbitrary code (e.g. 'git rev-parse HEAD'). So we're only talking about tagged/released source trees. Then the argument would be, what are you going to do with that name/version information? If the answer is "decide to install it", then (a) if you want to support installation from VCS snapshots (and you do) then your tool already has to support running arbitrary code to get the version number, and (b) installing the package will also certainly require running arbitrary code even if you have a nice official-release sdist, so. OTOH the twine upload case that Donald mentioned is a good example of an operation that might actually want some metadata from release sdists specifically :-). I'm not opposed to adding it if there's a clear use case, I just don't think we should try to shove every piece of wheel metadata into the sdist without a clear understanding of how they make sense and solve a problem *for sdists*. [...] >>> I would rather see an sdist format that can be introspected *without* >>> running code or a build tool. Installers and packaging tools like pip >>> need to be able to do that - one of the biggest issues with pip's >>> current sdist handling is that it can't make any meaningful decisions >>> before building at least the egg-info. >> >> Another way to look at this is to say that pip's current handling is >> proof that the build-to-get-metadata strategy is viable :-). > > Not if you look at the bug reports for pip that can be traced back to > needing to run setup.py egg-info to get metadata, or other variations > on not having static introspectable metadata in sdists. That sounds interesting! Can you elaborate? Links? I know that one unpleasant aspect of the current design is that the split between egg-info and actual building creates the possibility for time-of-definition-to-time-of-use bugs, where the final wheel hopefully matches what egg-info said it would, but in practice there could be skew. (Of course this is true in any system which represents this information in more than one place -- e.g. in sdist metadata and also in wheel metadata -- but right now it's particularly bad in cases where you can't actually get all the arguments you want to pass to setup() without running some code, but the code you need to run needs to be fetched via setup_requires=..., so you have to lie to setup() during the egg-info operation and hope that everything will work out in the end.) This is the motivation for the draft PEP to dropping egg-info as a separate operation. But there are certainly people who know more about the internal details of what pip needs than I do, and I'd love to hear more. -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Sat Oct 3 00:52:40 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Oct 2015 23:52:40 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 2 October 2015 at 23:15, Nathaniel Smith wrote: > "Project" is a pretty messy concept. Obviously in simple cases there's > a one-to-one mapping between project <-> wheel <-> importable package, > but this breaks down quickly in edge cases. I mistakenly used "project" in an attempt to avoid confusion resulting from me using the word "distribution" as a more general term than the way you were using "source distribution" or "binary distribution". Clearly I failed and made things more confusing. I use the term "distribution" in the sense used here https://packaging.python.org/en/latest/glossary/#term-distribution-package. Note that this is in contrast to the terms "source distribution" and "binary distribution" or "built distribution" in the same page. Sorry for confusing things. I'll stick to the terminology as in the PUG glossary from now on. > Consider a project that provides builds multiple wheels out of the > same source tree. You obviously can't expect that all of these > packages will have the same dependencies. Correct. But a distribution can and should (I believe) have the same dependencies for all of the source and built distributions derived from it. > This situation is not common today for Python packages, but the only > reason for that is that distutils makes it really hard to do -- it's > extremely common in other package ecosystems, and the advantages are > obvious. E.g., maybe numpy.distutils should be split into a separately > installable package from numpy -- there's no technical reason that > this should mean we are now forced to move the code for it into its > own VCS repository. I'm lost here, I'm afraid. Could you rephrase this in terms of the definitions from the PUG glossary? It sounds to me like the VCS repository is the project, which contains multiple distributions. I don't see how that's particularly hard. Each distribution just has its own subdirectory (and setup.py) in the VCS repository... > (I assume that by "platform tags" you mean what PEP 426 calls > "environment markers".) Nope, I mean as defined in PEP 425. The platform tag is part of the compatibility tag. Maybe I meant the ABI tag, I don't really follow the distinctions. > Environment markers are really useful for extending the set of cases > that can be handled by a single architecture-dependent wheel. And > they're a good fit for that environment, given that wheels can't > contain arbitrary code. > > But they're certainly never going to be adequate to provide a single > static description of every possible build configuration of every > possible project. And installing an sdist already requires arbitrary > code execution, so it doesn't make sense to try to build some > elaborate system to avoid arbitrary code execution just for the > dependency specification. > > You're right that in a perfect future world numpy C API related > dependencies would be handling by some separate ABI-tracking mechanism > similar to how the CPython ABI is tracked, so here are some other > examples of why environment markers are inadequate: > > In the future it will almost certainly be possible to build numpy in > two different configurations: one where it expects to find BLAS inside > a wheel distributed for this purpose (e.g. this is necessary to > provide high-quality windows wheels), and one where it expects to find > BLAS installed on the system. This decision will *not* be tied to the > platform, but be selectable at build time. E.g., on OS X there is a > system-provided BLAS library, but it has some issues. So the default > wheels on PyPI will probably act like windows and depend on a > BLAS-package that we control, but there will also be individual users > who prefer to build numpy in the configuration where it uses the > system BLAS, so we definitely need to support both options on OS X. > Now the problem: There will never be a single environment marker that > you can stick into a wheel or sdist that says "we depend on the > 'pyblas' package if the system is OS X (ok) and the user set this flag > in this configuration file during the build process (wait wut)". > > Similarly, I think someone was saying in a discussion recently that > lxml supports being built either in a mode where it requires libxml be > available on the system, or else it can be statically linked. Even if > in the future we start having metadata that lets us describe > dependencies on external system libraries, it's never going to be the > case that we can put the *same* dependency metadata into wheels that > are built using these two configurations. This is precisely the very complex issue that's being discussed under the banner of extending compatibility tags in a way that gives a viable but practical way of distinguishing binary wheels. You can either see that as a discussion about "expanding compatibility tags" or "finding something better than compatibility tags". I don't have much of a stake in that discussion, as the current compatibility tags suit my needs fine, as a Windows user. The issues seem to be around Linux and possibly some of the complexities around binary dependencies for numerical libraries. But the key point here is that I see the solution for this as being about distinguishing the "right" wheel for the target environment. It's not about anything that should reach back to sdists. Maybe a solution will involve a PEP 426 metadata enhancement that adds metadata that's only valid in binary distributions and not in source distributions, but that's fine by me. But it won't replace the existing dependency data, which *is* valid at the sdist level. At least as far as I can see - I'm willing to be enlightened. But your argument seems to be that sdist-level dependency information should be omitted because more detailed ABI compatibility data *might* be needed at the wheel level for some packages. I don't agree with that - we still need the existing metadata, even if more might be required in specialist cases. >> [1] If extras and environment markers don't cover the needs of >> scientific modules, we need some input into their design from the >> scientific community. But again, let's not throw away the work that's >> already done. > > As far as sdists go, you can either cover 90% of the cases by building > increasingly elaborate metadata formats, or you can cover 100% of the > cases by keeping things simple... But your argument seems to be that having metadata generated from package build code is "simpler". My strong opinion, based on what I've seen of the problems caused by having metadata in an "exectable setup.py", is that static metadata is far simpler. I don't believe that the cost of changing to a new system can be justified *without* getting the benefits of static metadata. Paul From donald at stufft.io Sat Oct 3 01:04:38 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 2 Oct 2015 19:04:38 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: > > Distutils delenda est. > I think that you should drop (from this PEP) the handling of a VCS/arbitrary directories and focus solely on creating a format for source distributions. A source distribution can (and should) be fairly strict and well defined exactly where all of the files go, what files exist and don't exist, and things of that nature (more on this later). In addition, the metadata files should be optimizes for machines to read and parse them first, for humans to read them second, and humans to write them not at all. Given the Python standard library, your metadata inside of the source distribution should (probably) be JSON. This is another reason why this should focus on the source distribution as well, because the file you put into VCS needs to be able to be written by humans. Metadata 2.0 should probably get finished before or as part of a new sdist format happening. I fell off towards the end of that and it appears that it got a lot more complex since I last looked at it. It probably needs more work. The filename should be strictly defined similarly to what Wheels have, probably something like {name}-{version}.{ext}, and like wheel it should mandate that any - characters inside of any of the fields should be escaped to a _ so that we can unambigiously parse it. It should not support arbitrary filenames because they are (almost never) actually sdists. In another email you mentioned something like the tarballs that github produces, but those are not source distributions, they are vcs exports and shouldn't be covered by this PEP. I don't believe that Python should develop anything like the Debian ability to have a single source "package" create multiple binary packages. The metadata of the Wheel *must* strictly match the metadata of the sdist (except for things that are Wheel specific). This includes things like name, version, etc. Trying to go down this path I think will make things a lot more complicated since we have a segmented archive where people have to claim particular names, otherwise how do you prevent me from registering the name "foobar" on PyPI and saying it produces the "Django" wheel? Since I think this should only deal with source distributions, then the primary thing we need is an operation that will take an unpacked source distribution that is currently sitting on the filesystem and turn it into a wheel located in a specific location. The layout for a source distribution should be specified, I think something like: ? ? . ? ? ??? meta ? ? ? ? ??? DESCRIPTION.rst ? ? ? ? ??? FORMAT-VERSION ? ? ? ? ??? LICENSE.txt ? ? ? ? ??? METADATA.json ? ? ??? src ? ? ? ? ??? my-cool-build-tool.cfg ? ? ? ? ??? mypackage ? ? ? ? ? ? ??? __init__.py I don't particularly care about the exact names, but this layout gives us two top level directories (and only two), one is a place where all of the source distribution metadata goes, and one is a src directory where all of the files for the project should go, including any relevant configuration for the build tool in use by the project. Having two directories like this eliminates the need to worry about naming collisions between the metadata files and the project itself. We should probably give this a new name instead of "sdist" and give it a dedicated extension. Perhaps we should call them "source wheels" and have the extension be something like .swhl or .src.whl. This means we don't need to worry about making the same artifact compatible with both the legacy toolchain and a toolchain that supports "source wheels". We should also probably specify a particular container format to be used for a .whl/.src.whl. It probably makes sense to simply use zip since that is what wheels use and it supports different compression algorithms internally. We probably want to at least suggest limiting compression algorithms used to Deflate and None, if not mandate that one of those two are used. We should include absolutely as much metadata as part of the static metadata inside the sdist as we can. I don't think there is any case to be made for things like name, version, summary, description, classifiers, license, keywords, contact information (author/maintainers), project URLs, etc are Wheel specific. I think there are other things which are arguably able to be specified in the sdist, but I'd need to fiddle with it to be sure. Basically any metadata that isn't included as static information will not be able to be displayed on PyPI. The metada should directly include the specifiers inside of it and shouldn't propagate the meme that pip's requirements.txt format is anything but a way to recreate a specific environment with pip. Build requirements cannot be dynamic. We don't need a "build in place" hook, you don't build source distributions in place, you build wheels with them. Another PEP that handles a VCS/non sdist directory can add things like building in place. I don't think there's ever going to be a world where pip depends on virtualenv or pyvenv. The PEP shouldn't prescribe how the tool installs the build deps or executes the build hook, though I think it should mandate that it is called with a compatible Python to the Wheel that is desired to be produced. Cross compiling can be handled later. Possibly we might want to make the hooks calling an executable instead of doing something that involves importing the hook and calling it. This would make it easier to prevent the build tool from monkeypatching the installation tool *and* make it easier for downstream redistributors to use it. If you're interested, I'm happy to directly collaborate on this PEP if it's in a github repository somewhere or something. There's an interoptability repo you can use or your own or whatever. Or you can tell me to go pound sand too and I'll just comment on posts to the ML. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Sat Oct 3 01:38:38 2015 From: qwcode at gmail.com (Marcus Smith) Date: Fri, 2 Oct 2015 16:38:38 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: > > > If you're interested, I'm happy to directly collaborate on this PEP if > it's in > a github repository somewhere or something. There's an interoptability repo > btw, the repo he's talking about is here: https://github.com/pypa/interoperability-peps it has a convention about where to add pep ideas that have no number yet -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Oct 3 01:58:41 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 2 Oct 2015 18:58:41 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 2, 2015 5:53 PM, "Paul Moore" wrote: > > On 2 October 2015 at 23:15, Nathaniel Smith wrote: > > "Project" is a pretty messy concept. Obviously in simple cases there's > > a one-to-one mapping between project <-> wheel <-> importable package, > > but this breaks down quickly in edge cases. > > I mistakenly used "project" in an attempt to avoid confusion resulting > from me using the word "distribution" as a more general term than the > way you were using "source distribution" or "binary distribution". > Clearly I failed and made things more confusing. > > I use the term "distribution" in the sense used here > https://packaging.python.org/en/latest/glossary/#term-distribution-package . > Note that this is in contrast to the terms "source distribution" and > "binary distribution" or "built distribution" in the same page. > > Sorry for confusing things. I'll stick to the terminology as in the > PUG glossary from now on. > > > Consider a project that provides builds multiple wheels out of the > > same source tree. You obviously can't expect that all of these > > packages will have the same dependencies. > > Correct. But a distribution can and should (I believe) have the same > dependencies for all of the source and built distributions derived > from it. > > > This situation is not common today for Python packages, but the only > > reason for that is that distutils makes it really hard to do -- it's > > extremely common in other package ecosystems, and the advantages are > > obvious. E.g., maybe numpy.distutils should be split into a separately > > installable package from numpy -- there's no technical reason that > > this should mean we are now forced to move the code for it into its > > own VCS repository. > > I'm lost here, I'm afraid. Could you rephrase this in terms of the > definitions from the PUG glossary? It sounds to me like the VCS > repository is the project, which contains multiple distributions. I > don't see how that's particularly hard. Each distribution just has its > own subdirectory (and setup.py) in the VCS repository... > > > (I assume that by "platform tags" you mean what PEP 426 calls > > "environment markers".) > > Nope, I mean as defined in PEP 425. The platform tag is part of the > compatibility tag. Maybe I meant the ABI tag, I don't really follow > the distinctions. > > > Environment markers are really useful for extending the set of cases > > that can be handled by a single architecture-dependent wheel. And > > they're a good fit for that environment, given that wheels can't > > contain arbitrary code. > > > > But they're certainly never going to be adequate to provide a single > > static description of every possible build configuration of every > > possible project. And installing an sdist already requires arbitrary > > code execution, so it doesn't make sense to try to build some > > elaborate system to avoid arbitrary code execution just for the > > dependency specification. > > > > You're right that in a perfect future world numpy C API related > > dependencies would be handling by some separate ABI-tracking mechanism > > similar to how the CPython ABI is tracked, so here are some other > > examples of why environment markers are inadequate: > > > > In the future it will almost certainly be possible to build numpy in > > two different configurations: one where it expects to find BLAS inside > > a wheel distributed for this purpose (e.g. this is necessary to > > provide high-quality windows wheels), and one where it expects to find > > BLAS installed on the system. This decision will *not* be tied to the > > platform, but be selectable at build time. E.g., on OS X there is a > > system-provided BLAS library, but it has some issues. So the default > > wheels on PyPI will probably act like windows and depend on a > > BLAS-package that we control, but there will also be individual users > > who prefer to build numpy in the configuration where it uses the > > system BLAS, so we definitely need to support both options on OS X. > > Now the problem: There will never be a single environment marker that > > you can stick into a wheel or sdist that says "we depend on the > > 'pyblas' package if the system is OS X (ok) and the user set this flag > > in this configuration file during the build process (wait wut)". > > > > Similarly, I think someone was saying in a discussion recently that > > lxml supports being built either in a mode where it requires libxml be > > available on the system, or else it can be statically linked. Even if > > in the future we start having metadata that lets us describe > > dependencies on external system libraries, it's never going to be the > > case that we can put the *same* dependency metadata into wheels that > > are built using these two configurations. > > This is precisely the very complex issue that's being discussed under > the banner of extending compatibility tags in a way that gives a > viable but practical way of distinguishing binary wheels. You can > either see that as a discussion about "expanding compatibility tags" > or "finding something better than compatibility tags". I don't have > much of a stake in that discussion, as the current compatibility tags > suit my needs fine, as a Windows user. The issues seem to be around > Linux and possibly some of the complexities around binary dependencies > for numerical libraries. > > But the key point here is that I see the solution for this as being > about distinguishing the "right" wheel for the target environment. > It's not about anything that should reach back to sdists. Maybe a > solution will involve a PEP 426 metadata enhancement that adds > metadata that's only valid in binary distributions and not in source > distributions, but that's fine by me. But it won't replace the > existing dependency data, which *is* valid at the sdist level. this would be good to discuss here: PEP 426: Define a JSON-LD context as part of the proposal #31 https://github.com/pypa/interoperability-peps/issues/31 > > At least as far as I can see - I'm willing to be enlightened. But your > argument seems to be that sdist-level dependency information should be > omitted because more detailed ABI compatibility data *might* be needed > at the wheel level for some packages. I don't agree with that - we > still need the existing metadata, even if more might be required in > specialist cases. if sys.platform: extras_require = > > > >> [1] If extras and environment markers don't cover the needs of > >> scientific modules, we need some input into their design from the > >> scientific community. But again, let's not throw away the work that's > >> already done. > > > > As far as sdists go, you can either cover 90% of the cases by building > > increasingly elaborate metadata formats, or you can cover 100% of the > > cases by keeping things simple... everything in one (composed) 'pydist.jsonld' > > But your argument seems to be that having metadata generated from > package build code is "simpler". My strong opinion, based on what I've > seen of the problems caused by having metadata in an "exectable > setup.py", is that static metadata is far simpler. static JSONLD is easily indexable with warehouse (Postgresql), elasticsearch, triplestores > > I don't believe that the cost of changing to a new system can be > justified *without* getting the benefits of static metadata. external benefits of canonical package URIs + schema.org/Action > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 3 03:03:30 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 18:03:30 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 2:30 PM, Donald Stufft wrote: > On October 2, 2015 at 4:20:00 PM, Nathaniel Smith (njs at pobox.com) wrote: [...] >> There are plenty of more >> complex examples too (e.g. ones that involve build/configure-time >> decisions about whether to rely on particular system libraries, or >> build/configure-time decisions about whether particular packages >> should even be built). > > I don't think build/configure-time decisions are great ideas as it's near > impossible to actually depend on them. For example, take Pillow, Pillow will > conditionally compile against libraries that enable it to much around with > PNGs. However, if I *need* Pillow with PNG support, I don't have any mechanism > to declare that. If instead, builds were *not* conditional and Pillow instead > split it's PNG capabilities out into it's own package called say, Pillow-PNG > which also did not conditionally compile against anything, but unconditionally > did, then we could add in something like having Pillow declare a "weak" > dependency on Pillow-PNG where we attempt to get it by default if possible, but > we will skip installing it if we can't locate/build it. If you combine this > with Extras, you could then easily make it so that people can depend on > particular conditional features by doing something like ``Pillow[PNG]`` in > their dependency metadata. While I agree with the sentiment here, I don't think we can simply unconditionally rule out build/configure-time decisions. I gave an example in the other subthread of a numpy wheel, which depending on build configuration might depend implicitly on the system BLAS, might have BLAS statically linked, or might depend explicitly on a "BLAS wheel". (And note that when configured to use a "BLAS wheel" then this would actually be a build-dependency, not just a runtime-dependency.) As far as downstream users are concerned, all of these numpy wheels export exactly the same API -- how numpy finds BLAS is just an internal detail. So in this case the problems your paragraph above is worrying about just don't arise. And numpy absolutely will need the option to be built in these different ways. >> >> For comparison, here's the Debian source package metadata: >> https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-debiansourcecontrolfiles >> Note that the only mandatory fields are format version / package name >> / package version / maintainer / checksums. The closest they come to >> making promises about the built packages are the Package-List and >> Binary fields which provide a optional hint about what binary packages >> will be built, and are allowed to contain lies (e.g. they explicitly >> don't guarantee that all the binary packages named will actually be >> produced on every architecture). The only kind of dependencies that a >> source package can declare are build-depends. > > Debian doesn't really have "source packages" like we do, but inside of the > debian/ directory is the control file which lists all of the dependency > information (or explicitly lists a placeholder where something can't be > statically declared). Someone who is more of an expert on debian packaging can correct me if I'm wrong, but I'm 99% sure that this is incorrect, and in an important way. The actual interface between a build tool like dpkg-buildpackage and a source package is: (a) the .dsc file, with the required fields I mentioned, (b) the debian/rules file, which is an opaque executable that can be called to perform standard operations like "build" and "clean" -- basically the moral equivalent of the hooks in our sdist proposal. The debian/control file does have a conventional format, but this is just convention -- most-or-all debian/control scripts all use the same set of tools to work with this file, and expect it to be in the same place. But if, say, Debian decides that they need a new kind of placeholder to handle a situation that hasn't arisen before, then there's no need to change the definition of a source package: you just add support for the new placeholder to the tools that work with this file, and then packages that want to make use of the new placeholder just have to Build-Depend on the latest version of those tools. This is the idea motivating the sdist PEP's design: you can't specify all of a source distribution's metadata statically, and then given that you'll be specifying at least part of the metadata dynamically, you want to do it in a way that you can change without having to do a PEP and update pip etc. >> >> > To a similar tune, this PEP also doesn't make it possible to really get at >> > any other metadata without executing software. This makes it pratically >> > impossible to safely inspect an unknown or untrusted package to determine what >> > it is and to get information about it. Right now PyPI relies on the uploading >> > tool to send that information alongside of the file it is uploading, but >> > honestly what it should be doing is extracting that information from within the >> > file. This is sort of possible right now since distutils and setuptools both >> > create a static metadata file within the source distribution, but we don't rely >> > on that within PyPI because that information may or may not be accurate and may >> > or may not exist. However the twine uploading tool *does* rely on that, and >> > this PEP would break the ability for twine to upload a package without >> > executing arbitrary code. >> >> Okay, what metadata do you need? We certainly could put name / version >> kind of stuff in there. We left it out because we weren't sure what >> was necessary and it's easy to add later, but anything that's needed >> by twine fits neatly into the existing text saying that we should >> "include extra metadata in source distributions if it helps solve >> specific problems that are unique to distribution" -- twine uploads >> definitely count. > > Everything that isn't specific to a built wheel. Look at the previously > accepted metadata specs as well as PEP 426. If you're not including a field > that was included in one of those, there should be a rationale for why that > field is no longer being included. The default rationale is just "let's keep our options open" -- it's much easier to add than to subtract later. In particular I hesitate a little bit to just drop in everything from PEP 426 and friends, because previous specs haven't really thought through the distinction between sdists and wheels -- e.g. if an sdist generates two wheels, they probably won't have the same name, description, trove classifiers, etc. They may not even have the same version (e.g. if two different tools with existing numbering schemes get merged into a single distribution -- esp. if one of them needs an epoch marker). So it may well make sense to have an "sdist description field", but it's not immediately obvious that it's identical to a wheel's description field. I mean, in practice it's probably no big deal -- a description is some text for human readers, whatever, it's useful and it'll be fine. But given that we can trivially add more fields to the pypackage.cfg file later, and that current sdists don't have any of this metadata, I just don't want to risk blocking progress on one axis (enabling better build systems) while waiting to achieve maximal progress on another mostly-orthogonal axis (having nice metadata in sdists for tools like twine to take advantage of). Bottom line: If after further discussion we reach the point where the only thing blocking this is the addition of name and description and trove classifier fields, then of course we'll just add those to the PEP :-). -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sat Oct 3 03:06:32 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 18:06:32 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 3:52 PM, Paul Moore wrote: > On 2 October 2015 at 23:15, Nathaniel Smith wrote: [...] >> This situation is not common today for Python packages, but the only >> reason for that is that distutils makes it really hard to do -- it's >> extremely common in other package ecosystems, and the advantages are >> obvious. E.g., maybe numpy.distutils should be split into a separately >> installable package from numpy -- there's no technical reason that >> this should mean we are now forced to move the code for it into its >> own VCS repository. > > I'm lost here, I'm afraid. Could you rephrase this in terms of the > definitions from the PUG glossary? It sounds to me like the VCS > repository is the project, which contains multiple distributions. I > don't see how that's particularly hard. Each distribution just has its > own subdirectory (and setup.py) in the VCS repository... The problem is that projects tend to release the whole project together, rather than releasing individual subdirectories, and that usually you can't just rip those subdirectories out of the parent project and expect to build them on their own, because there's shared infrastructure for build, configuration, static libraries with utility code that get built once and then linked into each distribution... having a VCS checkout = one wheel rule blocks a lot of otherwise sensible project arrangements and forces awkward technical workarounds. But if you allow one VCS checkout to produce multiple wheels, then I can't see how you can avoid having one sdist produce multiple wheels. >> (I assume that by "platform tags" you mean what PEP 426 calls >> "environment markers".) > > Nope, I mean as defined in PEP 425. The platform tag is part of the > compatibility tag. Maybe I meant the ABI tag, I don't really follow > the distinctions. > >> Environment markers are really useful for extending the set of cases >> that can be handled by a single architecture-dependent wheel. And >> they're a good fit for that environment, given that wheels can't >> contain arbitrary code. >> >> But they're certainly never going to be adequate to provide a single >> static description of every possible build configuration of every >> possible project. And installing an sdist already requires arbitrary >> code execution, so it doesn't make sense to try to build some >> elaborate system to avoid arbitrary code execution just for the >> dependency specification. >> >> You're right that in a perfect future world numpy C API related >> dependencies would be handling by some separate ABI-tracking mechanism >> similar to how the CPython ABI is tracked, so here are some other >> examples of why environment markers are inadequate: >> >> In the future it will almost certainly be possible to build numpy in >> two different configurations: one where it expects to find BLAS inside >> a wheel distributed for this purpose (e.g. this is necessary to >> provide high-quality windows wheels), and one where it expects to find >> BLAS installed on the system. This decision will *not* be tied to the >> platform, but be selectable at build time. E.g., on OS X there is a >> system-provided BLAS library, but it has some issues. So the default >> wheels on PyPI will probably act like windows and depend on a >> BLAS-package that we control, but there will also be individual users >> who prefer to build numpy in the configuration where it uses the >> system BLAS, so we definitely need to support both options on OS X. >> Now the problem: There will never be a single environment marker that >> you can stick into a wheel or sdist that says "we depend on the >> 'pyblas' package if the system is OS X (ok) and the user set this flag >> in this configuration file during the build process (wait wut)". >> >> Similarly, I think someone was saying in a discussion recently that >> lxml supports being built either in a mode where it requires libxml be >> available on the system, or else it can be statically linked. Even if >> in the future we start having metadata that lets us describe >> dependencies on external system libraries, it's never going to be the >> case that we can put the *same* dependency metadata into wheels that >> are built using these two configurations. > > This is precisely the very complex issue that's being discussed under > the banner of extending compatibility tags in a way that gives a > viable but practical way of distinguishing binary wheels. You can > either see that as a discussion about "expanding compatibility tags" > or "finding something better than compatibility tags". I don't have > much of a stake in that discussion, as the current compatibility tags > suit my needs fine, as a Windows user. The issues seem to be around > Linux and possibly some of the complexities around binary dependencies > for numerical libraries. > > But the key point here is that I see the solution for this as being > about distinguishing the "right" wheel for the target environment. > It's not about anything that should reach back to sdists. Maybe a > solution will involve a PEP 426 metadata enhancement that adds > metadata that's only valid in binary distributions and not in source > distributions, but that's fine by me. But it won't replace the > existing dependency data, which *is* valid at the sdist level. Okay. I think this is the key question for me: you want sdist's to contain rich static metadata, presumably because you want to do something with that metadata -- you say that not having it causes problems. The obvious thing that pip might want to use that metadata for is so that it can look at an sdist and know whether the wheel it builds from that sdist will be useful in solving its current dependency goals. But to answer this question, you actually do need to know what the compatibility tag will be. So: what problems does it solve for pip to get access to static information about some, *but not all* of the eventual wheel's metadata? (It's also the case that in the numpy example I gave above, it isn't just the compatibility tag that can vary between wheels built from the same sdist, it's really truly the actual runtime dependency metadata that varies. But even if we ignore that, I think my question still stands.) -n -- Nathaniel J. Smith -- http://vorpus.org From chris.barker at noaa.gov Sat Oct 3 03:27:23 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 2 Oct 2015 18:27:23 -0700 Subject: [Distutils] pip install from source dir?? In-Reply-To: References: Message-ID: On Fri, Oct 2, 2015 at 3:37 PM, Erik Bray wrote: > On Fri, Oct 2, 2015 at 5:39 PM, Donald Stufft wrote: > > ``pip install path/to/directory/with/setup.py/in/it/`` > > In particular I install most of my packages from source these days by > cd'ing into the source and > > $ pip install . > > That's all it takes. Works with -e too. > yup -- plugged this into a conda recipe and it works fine. in fact, I'm going to try to do all of my python recipes that way. Actually: $PYTHON -m pip install ./ $PYTHON is the python in conda's current build environment -- this will assure it used the right pip. Thanks all, this is working great. -CHB > Erik > > > On October 2, 2015 at 5:38:29 PM, Chris Barker (chris.barker at noaa.gov) > wrote: > >> I can't seem to find a way to pip install from a source dir -- not a > >> tarball. > >> > >> Why would I want to do that? > >> > >> In this case, it's because I'm trying to build conda packages for python > >> packages that have compiled C code, and are not using setuptools. > >> > >> On Windows, you need to use setuptools in order to use the "MS compiler > for > >> Python 2.7". So a straight "setup.py build" fails. > >> > >> But pip injects setuptools, so that "pip install" does work. But conda > has > >> already unpacked the source distribution, so I'd like to point pip at > that, > >> and can't find a way to do that (I could have sworn it was possible...) > >> > >> Do I have to build an sdist, and then point pip at that?? > >> > >> -CHB > >> > >> > >> > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Oct 3 04:27:33 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 2 Oct 2015 19:27:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Hi Donald, Thanks for taking the time to make such detailed comments! Thoughts below. On Fri, Oct 2, 2015 at 4:04 PM, Donald Stufft wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (njs at pobox.com) wrote: >> >> Distutils delenda est. >> > > I think that you should drop (from this PEP) the handling of a VCS/arbitrary > directories and focus solely on creating a format for source distributions. A > source distribution can (and should) be fairly strict and well defined exactly > where all of the files go, what files exist and don't exist, and things of that > nature (more on this later). Hmm. Okay, I think this really helps clarify our key point of difference! For me, an important requirement is that there continue to be a single standard command that end-users can use to install a VCS checkout. This is a really important usability property -- everyone knows "setup.py install". Unfortunately we can't keep "setup.py install" given its entanglement with distutils and the desire to split building and installation, so the obvious answer is that this should become 'pip install ', and from that everything else follows. Having a standard way to install from a VCS checkout is also useful for things like requirements files... and in fact it's required by our current standards. PEP 440 has this as an example of a valid dependency specification: pip @ git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d So I'm extremely reluctant to give up on standardizing how to handle VCS checkouts. And if we're going to have a standard for that, then would sure be nice if we could share the work between this standard and the one for sdists, given how similar they are. [...] > I don't believe that Python should develop anything like the Debian ability to > have a single source "package" create multiple binary packages. The metadata of > the Wheel *must* strictly match the metadata of the sdist (except for things > that are Wheel specific). This includes things like name, version, etc. Trying > to go down this path I think will make things a lot more complicated since we > have a segmented archive where people have to claim particular names, otherwise > how do you prevent me from registering the name "foobar" on PyPI and saying it > produces the "Django" wheel? What prevents it in the current draft is that there's no way for foobar to say any such thing :-). If you ask for Django, then the only sdist it will look at is the one in the Django segment. This is an intentionally limited solution, based on the intuition that multiple wheels from a single sdist will tend to be a relatively rare case, when they do occur then there will generally be one "main" wheel that people will want to depend on, and that people should be uploading wheels anyway rather than relying on sdists. (Part of the intuition for the last part is that we also have a not-terribly-secret-conspiracy here for writing a PEP to get Linux wheels onto PyPI and at least achieve feature parity with Windows / OS X. Obviously there will always be weird platforms -- iOS and FreeBSD and Linux-without-glibc and ... -- but this should dramatically reduce the frequency with which people need sdist dependencies.) If that proves inadequate, then the obvious extension would be to add some metadata to the sdist similar to Debian's, where an sdist has a list of all the wheels that it (might) produce when built, PyPI would grow an API by which pip-or-whoever could query for all sdists that claim to be able to produce wheel X, and at the same time PyPI would start enforcing the rule that if you want to upload an sdist that claims to produce wheel X then you have to own the name X. (After all, you need to own that name anyway so you can upload the wheels.) Or alternatively people could just split up their packages, like would be required by your proposal anyway :-). So I sorta doubt it will be a problem in practice, but even if becomes one then it won't be hard to fix. (And to be clear, the multiple-wheels-from-one-sdist thing is not a primary goal of this proposal -- the main reason we put it in is that once you've given up on having static wheel metadata inside the sdist then supporting multiple-wheels-from-one-sdist is trivial, so you might as well do it, esp. since it's a case that does seem to come up with some regularity in real life and you don't want to make people fight with their tools when it's unnecessary.) > Since I think this should only deal with source distributions, then the primary > thing we need is an operation that will take an unpacked source distribution > that is currently sitting on the filesystem and turn it into a wheel located > in a specific location. > > The layout for a source distribution should be specified, I think something > like: > > . > ??? meta > ? ??? DESCRIPTION.rst > ? ??? FORMAT-VERSION > ? ??? LICENSE.txt > ? ??? METADATA.json > ??? src > ??? my-cool-build-tool.cfg > ??? mypackage > ??? __init__.py > > I don't particularly care about the exact names, but this layout gives us two > top level directories (and only two), one is a place where all of the source > distribution metadata goes, and one is a src directory where all of the files > for the project should go, including any relevant configuration for the build > tool in use by the project. Having two directories like this eliminates the > need to worry about naming collisions between the metadata files and the > project itself. > > We should probably give this a new name instead of "sdist" and give it a > dedicated extension. Perhaps we should call them "source wheels" and have the > extension be something like .swhl or .src.whl. This means we don't need to > worry about making the same artifact compatible with both the legacy toolchain > and a toolchain that supports "source wheels". > > We should also probably specify a particular container format to be used for > a .whl/.src.whl. It probably makes sense to simply use zip since that is what > wheels use and it supports different compression algorithms internally. We > probably want to at least suggest limiting compression algorithms used to > Deflate and None, if not mandate that one of those two are used. > > We should include absolutely as much metadata as part of the static metadata > inside the sdist as we can. I don't think there is any case to be made for > things like name, version, summary, description, classifiers, license, > keywords, contact information (author/maintainers), project URLs, etc are > Wheel specific. I think there are other things which are arguably able to be > specified in the sdist, but I'd need to fiddle with it to be sure. Basically > any metadata that isn't included as static information will not be able to be > displayed on PyPI. I feel like this idea of "source wheels" makes some sense if we want something that looks like a wheel, but without the ABI compatibility issues of wheels. I'm uncertain how well it can be made to work in practice, or how urgent it is once we have a 95% solution in place for linux wheels, but it's certainly an interesting idea. To me it feels rather different from a traditional sdist, and obviously there's still the problem of having a standard way to build from a VCS checkout. It might even make sense to have standard methods to go: VCS checkout -> sdist -> (wheels and/or source wheels) ? > The metada should directly include the specifiers inside of it and shouldn't > propagate the meme that pip's requirements.txt format is anything but a way > to recreate a specific environment with pip. Yeah, there's a big question mark next to the requirements.txt stuff in the draft PEP, because something more standard and structured would certainly be nice. But requirements.txt is wildly popular, and for a good reason -- it provides a simple terse syntax that does what people want. (By comparison, the PEP 426 JSON syntax for requirements with extras and environment specifiers is extremely cumbersome yet less featureful.) And semantically, what we want here is a way to say "to build *this* I need an environment that looks like *this*", which is pretty close to what requirements.txt is actually designed for. So I dunno -- instead of fighting the meme maybe we should embrace it :-). But obviously this is a tangent to the main questions. [...] > I don't think there's ever going to be a world where pip depends on virtualenv > or pyvenv. Huh, really? Can you elaborate on why not? The standard doesn't have to require the use of clean build environments (I was thinking of the text in the standard as applying with the "as if rule" -- a valid sdist is one that can be built the way described, if you have some way that will work to build such sdists then your way is valid too). But using clean environments by default is really the only way that we're going to get a world where most packages have accurate build requirements. -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Sat Oct 3 13:51:28 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Oct 2015 12:51:28 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 3 October 2015 at 02:03, Nathaniel Smith wrote: > In particular I hesitate a little bit to just drop in everything from > PEP 426 and friends, because previous specs haven't really thought > through the distinction between sdists and wheels -- e.g. if an sdist > generates two wheels, they probably won't have the same name, > description, trove classifiers, etc. They may not even have the same > version (e.g. if two different tools with existing numbering schemes > get merged into a single distribution -- esp. if one of them needs an > epoch marker). So it may well make sense to have an "sdist description > field", but it's not immediately obvious that it's identical to a > wheel's description field. I can only assume you're using the term sdist differently from the way it is normally used (as in the PUG glossary), because for me the idea that a sdist generates multiple wheels makes no sense. If I do "pip wheel " I expect to get a single wheel that can be installed to give the same results as "pip install " would, The wheel install will work on my build machine, and on any machine where the wheel's compatibility metadata (the compatibility tags, currently) says it's valid. The above behaviour is key to pip's mechanism for auto-generating and caching wheels when doing an install, so I don't see how it could easily be discarded. If what you're calling a "sdist" doesn't work like this, maybe you should invent a new term, so that people don't get confused? If it *does* work like this, I don't see what you mean by a sdist generating two wheels. Paul From dholth at gmail.com Sat Oct 3 18:45:31 2015 From: dholth at gmail.com (Daniel Holth) Date: Sat, 03 Oct 2015 16:45:31 +0000 Subject: [Distutils] Did https://mail.python.org/pipermail/python-list/2008-June/467441.html get an answer? In-Reply-To: References: Message-ID: In wheel we had to jump through some hoops to do rfc822 style documents with utf-8, a task which can also be accomplished more easily with newer versions of the standard Python 3 email library (but not the versions we had to support at the time). https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/pkginfo.py?at=default&fileviewer=file-view-default On Mon, Sep 21, 2015 at 9:44 PM Robert Collins wrote: > On 22 September 2015 at 13:07, Daniel Holth wrote: > > It's a bug if bdist_wheel generates anything other than utf-8 METADATA. > > Cool. And PKG-INFO ? > > -Rob > > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Oct 3 19:30:57 2015 From: brett at python.org (Brett Cannon) Date: Sat, 03 Oct 2015 17:30:57 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote: > On 3 October 2015 at 02:03, Nathaniel Smith wrote: > > In particular I hesitate a little bit to just drop in everything from > > PEP 426 and friends, because previous specs haven't really thought > > through the distinction between sdists and wheels -- e.g. if an sdist > > generates two wheels, they probably won't have the same name, > > description, trove classifiers, etc. They may not even have the same > > version (e.g. if two different tools with existing numbering schemes > > get merged into a single distribution -- esp. if one of them needs an > > epoch marker). So it may well make sense to have an "sdist description > > field", but it's not immediately obvious that it's identical to a > > wheel's description field. > > I can only assume you're using the term sdist differently from the way > it is normally used (as in the PUG glossary), because for me the idea > that a sdist generates multiple wheels makes no sense. > > If I do "pip wheel " I expect to get a single wheel that > can be installed to give the same results as "pip install > " would, The wheel install will work on my build machine, > and on any machine where the wheel's compatibility metadata (the > compatibility tags, currently) says it's valid. > > The above behaviour is key to pip's mechanism for auto-generating and > caching wheels when doing an install, so I don't see how it could > easily be discarded. > > If what you're calling a "sdist" doesn't work like this, maybe you > should invent a new term, so that people don't get confused? If it > *does* work like this, I don't see what you mean by a sdist generating > two wheels. > I think sdist is getting a bit overloaded in this discussion. From my understanding of what Paul and Donald are after, the process should be: VCS -> sdist/source wheel -> binary wheel. Here, "VCS" is literally a git/hg clone of some source tree. A "sdist/source wheel" is carefully constructed zip file that contains all the source code from the VCS necessary to build a binary wheel for a project as long as the proper dependencies exist (e.g., proper version of NumPy, BLAS in one of its various forms, etc.). The binary wheel is obviously the final artifact that can just be flat-out loaded by Python without any work. So Paul doesn't see sdist/source wheels producing more than one binary wheel because in its own way an sdist/source wheel is a "compiled" artifact of select source code whose only purpose is to generate a binary wheel for a single project. So while a VCS clone might have multiple subprojects, each project should generate a single sdist/source wheel. Now this isn't to say that an sdist/source wheel won't generate different *versions* of a binary wheel based on whether e.g. BLAS is system-linked, statically linked, or dynamically loaded from a Linux binary wheel. But the key point is that the sdist/source wheel still **only** makes one kind of project. >From this perspective I don't see Nathaniel's desire for installing from a VCS as something other than requiring a step to bundle up the source code into an sdist/source wheel that pip knows how to handle. But I think what Paul and Donald are saying is pip doesn't want to have anything to do with the VCS -> sdist/source wheel step and that is entirely up to the project to manage through whatever tooling they choose. I also view the sdist//source wheel as almost a mini-VCS checkout since it is just a controlled view of the source code with a bunch of helpful, static metadata for pip to use to execute whatever build steps are needed to get to a binary wheel. Or I'm totally wrong. =) But for me I actually like the idea of an sdist/source wheel being explained as "a blob of source in a structured way that can produce a binary wheel for the package" and a binary wheel as "a blob of bytes for a package that Python can import" and I'm totally fine having C extensions not just shuttle around a blind zip but a structured zip in the form of an sdist/source wheel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sat Oct 3 19:36:15 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 3 Oct 2015 13:36:15 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 3, 2015 at 1:31:48 PM, Brett Cannon (brett at python.org) wrote: > On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote: > > > On 3 October 2015 at 02:03, Nathaniel Smith wrote: > > > In particular I hesitate a little bit to just drop in everything from > > > PEP 426 and friends, because previous specs haven't really thought > > > through the distinction between sdists and wheels -- e.g. if an sdist > > > generates two wheels, they probably won't have the same name, > > > description, trove classifiers, etc. They may not even have the same > > > version (e.g. if two different tools with existing numbering schemes > > > get merged into a single distribution -- esp. if one of them needs an > > > epoch marker). So it may well make sense to have an "sdist description > > > field", but it's not immediately obvious that it's identical to a > > > wheel's description field. > > > > I can only assume you're using the term sdist differently from the way > > it is normally used (as in the PUG glossary), because for me the idea > > that a sdist generates multiple wheels makes no sense. > > > > If I do "pip wheel " I expect to get a single wheel that > > can be installed to give the same results as "pip install > > " would, The wheel install will work on my build machine, > > and on any machine where the wheel's compatibility metadata (the > > compatibility tags, currently) says it's valid. > > > > The above behaviour is key to pip's mechanism for auto-generating and > > caching wheels when doing an install, so I don't see how it could > > easily be discarded. > > > > If what you're calling a "sdist" doesn't work like this, maybe you > > should invent a new term, so that people don't get confused? If it > > *does* work like this, I don't see what you mean by a sdist generating > > two wheels. > > > > I think sdist is getting a bit overloaded in this discussion. From my > understanding of what Paul and Donald are after, the process should be: > > VCS -> sdist/source wheel -> binary wheel. > > Here, "VCS" is literally a git/hg clone of some source tree. A > "sdist/source wheel" is carefully constructed zip file that contains all > the source code from the VCS necessary to build a binary wheel for a > project as long as the proper dependencies exist (e.g., proper version of > NumPy, BLAS in one of its various forms, etc.). The binary wheel is > obviously the final artifact that can just be flat-out loaded by Python > without any work. > > So Paul doesn't see sdist/source wheels producing more than one binary > wheel because in its own way an sdist/source wheel is a "compiled" artifact > of select source code whose only purpose is to generate a binary wheel for > a single project. So while a VCS clone might have multiple subprojects, > each project should generate a single sdist/source wheel. > > Now this isn't to say that an sdist/source wheel won't generate different > *versions* of a binary wheel based on whether e.g. BLAS is system-linked, > statically linked, or dynamically loaded from a Linux binary wheel. But the > key point is that the sdist/source wheel still **only** makes one kind of > project. > > From this perspective I don't see Nathaniel's desire for installing from a > VCS as something other than requiring a step to bundle up the source code > into an sdist/source wheel that pip knows how to handle. But I think what > Paul and Donald are saying is pip doesn't want to have anything to do with > the VCS -> sdist/source wheel step and that is entirely up to the project > to manage through whatever tooling they choose. I also view the > sdist//source wheel as almost a mini-VCS checkout since it is just a > controlled view of the source code with a bunch of helpful, static metadata > for pip to use to execute whatever build steps are needed to get to a > binary wheel. > > Or I'm totally wrong. =) But for me I actually like the idea of an > sdist/source wheel being explained as "a blob of source in a structured way > that can produce a binary wheel for the package" and a binary wheel as "a > blob of bytes for a package that Python can import" and I'm totally fine > having C extensions not just shuttle around a blind zip but a structured > zip in the form of an sdist/source wheel. This is basically accurate, the only thing is that I think that we do need an answer for handling the VCS side of things, but I think that we should defer it until after this PEP. Right now I think the PEP is trying to tackle two different problems at once because on the surface they look similar but IMO a VCS/arbitrary directory and a sdist are two entirely different things that are only similar on the surface. So Yes the path would be: VCS -> sdist (aka Source Wheel or w/e) -> Wheel -> Install \ ?\-> "In Place" install (aka Development Install) Right now, we have the Wheel -> Install part, and I'd like for this PEP to focus on the sdist -> Wheel part, and then a future PEP focus on the VCS part. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sat Oct 3 19:50:38 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 3 Oct 2015 13:50:38 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 2, 2015 at 10:27:36 PM, Nathaniel Smith (njs at pobox.com) wrote: > > So I'm extremely reluctant to give up on standardizing how to handle > VCS checkouts. And if we're going to have a standard for that, then > would sure be nice if we could share the work between this standard > and the one for sdists, given how similar they are. Mentioned in another email, but I don't think we should give up on the VCS handling, I just think we should defer it to another PEP. I think this PEP is suffering from trying to use the same mechanism for VCS and sdists when they are different things and have different considerations. The Python packaging toolchain has a long history of suffering because it tried to reuse the same thing across multiple "phases" of the packaging life cycle. The problem with that is different phases have different needs, and when you're trying to use the same thing across multiple you suffer from something that is a sort of "Jack of All Trades, Master of None" kind of thing. > > What prevents it in the current draft is that there's no way for > foobar to say any such thing :-). If you ask for Django, then the only > sdist it will look at is the one in the Django segment. This is an > intentionally limited solution, based on the intuition that multiple > wheels from a single sdist will tend to be a relatively rare case, > when they do occur then there will generally be one "main" wheel that > people will want to depend on, and that people should be uploading > wheels anyway rather than relying on sdists. There *must* be a 1:1 mapping in name/version between sdist and wheels. This assumption is baked into basically every layer of the toolchain. > > I feel like this idea of "source wheels" makes some sense if we want > something that looks like a wheel, but without the ABI compatibility > issues of wheels. I'm uncertain how well it can be made to work in > practice, or how urgent it is once we have a 95% solution in place for > linux wheels, but it's certainly an interesting idea. To me it feels > rather different from a traditional sdist, and obviously there's still > the problem of having a standard way to build from a VCS checkout. I feel like you have some sort of "a sdist is jsut a tarball of a VCS" mentality and I don't think that idea of a sdist is generally useful. > > I don't think there's ever going to be a world where pip depends on virtualenv > > or pyvenv. > > Huh, really? Can you elaborate on why not? The standard doesn't have > to require the use of clean build environments (I was thinking of the > text in the standard as applying with the "as if rule" -- a valid > sdist is one that can be built the way described, if you have some way > that will work to build such sdists then your way is valid too). But > using clean environments by default is really the only way that we're > going to get a world where most packages have accurate build > requirements. > We'll most likely use a clean environment by explicitly emptying out the sys.path or something similar. We won't depend on virtual environments because it is a layering violation. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Sat Oct 3 20:18:01 2015 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 3 Oct 2015 13:18:01 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 3, 2015 12:36 PM, "Donald Stufft" wrote: > > > > On October 3, 2015 at 1:31:48 PM, Brett Cannon (brett at python.org) wrote: > > On Sat, 3 Oct 2015 at 04:51 Paul Moore wrote: > > > > > On 3 October 2015 at 02:03, Nathaniel Smith wrote: > > > > In particular I hesitate a little bit to just drop in everything from > > > > PEP 426 and friends, because previous specs haven't really thought > > > > through the distinction between sdists and wheels -- e.g. if an sdist > > > > generates two wheels, they probably won't have the same name, > > > > description, trove classifiers, etc. They may not even have the same > > > > version (e.g. if two different tools with existing numbering schemes > > > > get merged into a single distribution -- esp. if one of them needs an > > > > epoch marker). So it may well make sense to have an "sdist description > > > > field", but it's not immediately obvious that it's identical to a > > > > wheel's description field. > > > > > > I can only assume you're using the term sdist differently from the way > > > it is normally used (as in the PUG glossary), because for me the idea > > > that a sdist generates multiple wheels makes no sense. > > > > > > If I do "pip wheel " I expect to get a single wheel that > > > can be installed to give the same results as "pip install > > > " would, The wheel install will work on my build machine, > > > and on any machine where the wheel's compatibility metadata (the > > > compatibility tags, currently) says it's valid. > > > > > > The above behaviour is key to pip's mechanism for auto-generating and > > > caching wheels when doing an install, so I don't see how it could > > > easily be discarded. > > > > > > If what you're calling a "sdist" doesn't work like this, maybe you > > > should invent a new term, so that people don't get confused? If it > > > *does* work like this, I don't see what you mean by a sdist generating > > > two wheels. > > > > > > > I think sdist is getting a bit overloaded in this discussion. From my > > understanding of what Paul and Donald are after, the process should be: > > > > VCS -> sdist/source wheel -> binary wheel. > > > > Here, "VCS" is literally a git/hg clone of some source tree. A > > "sdist/source wheel" is carefully constructed zip file that contains all > > the source code from the VCS necessary to build a binary wheel for a > > project as long as the proper dependencies exist (e.g., proper version of > > NumPy, BLAS in one of its various forms, etc.). The binary wheel is > > obviously the final artifact that can just be flat-out loaded by Python > > without any work. > > > > So Paul doesn't see sdist/source wheels producing more than one binary > > wheel because in its own way an sdist/source wheel is a "compiled" artifact > > of select source code whose only purpose is to generate a binary wheel for > > a single project. So while a VCS clone might have multiple subprojects, > > each project should generate a single sdist/source wheel. > > > > Now this isn't to say that an sdist/source wheel won't generate different > > *versions* of a binary wheel based on whether e.g. BLAS is system-linked, > > statically linked, or dynamically loaded from a Linux binary wheel. But the > > key point is that the sdist/source wheel still **only** makes one kind of > > project. > > > > From this perspective I don't see Nathaniel's desire for installing from a > > VCS as something other than requiring a step to bundle up the source code > > into an sdist/source wheel that pip knows how to handle. But I think what > > Paul and Donald are saying is pip doesn't want to have anything to do with > > the VCS -> sdist/source wheel step and that is entirely up to the project > > to manage through whatever tooling they choose. I also view the > > sdist//source wheel as almost a mini-VCS checkout since it is just a > > controlled view of the source code with a bunch of helpful, static metadata > > for pip to use to execute whatever build steps are needed to get to a > > binary wheel. > > > > Or I'm totally wrong. =) But for me I actually like the idea of an > > sdist/source wheel being explained as "a blob of source in a structured way > > that can produce a binary wheel for the package" and a binary wheel as "a > > blob of bytes for a package that Python can import" and I'm totally fine > > having C extensions not just shuttle around a blind zip but a structured > > zip in the form of an sdist/source wheel. > > This is basically accurate, the only thing is that I think that we do need an > answer for handling the VCS side of things, but I think that we should defer > it until after this PEP. Right now I think the PEP is trying to tackle two > different problems at once because on the surface they look similar but IMO > a VCS/arbitrary directory and a sdist are two entirely different things that > are only similar on the surface. > > So Yes the path would be: > > VCS -> sdist (aka Source Wheel or w/e) -> Wheel -> Install > \ > \-> "In Place" install (aka Development Install) > > Right now, we have the Wheel -> Install part, and I'd like for this PEP to > focus on the sdist -> Wheel part, and then a future PEP focus on the VCS part. thanks! is there a diagram of this somewhere? Is this basically a FSM with URIs for each resource and transformation, where more specific attributes are added (e.g. to pydist.jsonld)? [PEP 426 JSONLD] differences between VCS and sdist: * MANIFEST.in * setup.py * egg-info metadata PEP 426 JSONLD: https://github.com/pypa/interoperability-peps/issues/31 > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From setuptools at bugs.python.org Sun Oct 4 06:45:46 2015 From: setuptools at bugs.python.org (Stephen) Date: Sun, 04 Oct 2015 04:45:46 +0000 Subject: [Distutils] [issue163] Resolved Message-ID: <1443933946.17.0.650887817144.issue163@psf.upfronthosting.co.za> New submission from Stephen: Resolved ---------- files: 4588238e23891u2uy12.html messages: 754 nosy: stephenedu priority: wish status: resolved title: Resolved Added file: http://bugs.python.org/setuptools/file176/4588238e23891u2uy12.html _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- Play Slots, Roulette, and Poker Online

Play Slots, Roulette, and Poker Online

Australian casinos and gaming online slot machines

If you'd like to perform massive money obtain a individual stand, if you afford after that it do not become in case you concept the planet. You're able to search for podiums wherever clean generally stands at, others simply possess a workplace. As soon as you change the right tactic and obtain knowledgeable about the operating of gambling, then you can probably play for higher stakes. If you too are interested it's better to have a around through the well-known directories which come up with the massive set of 21 websites and their wide opinions. Instead, associates could possibly be necessary to enter selected limitations for them to get into the bonuses. Just as it's having some other game, you'll find methods as you are able to produce to be able to boost your likelihood of profitable at Pay Gown. The magic to be able to disguise behind a screen to hide your feelings been employed by effectively for several people of the game nowadays.

The most effective online casino on the web today boast good design, good games, and several financial selections to choose from. It's one of the most liked outdoor recreation that have been happening for a long time. It's likewise good for squeeze with fingers that flop nicely, like suited connectors, in case you are called by way of a great palm. That could be perfect give, no real matter what the turn and water cards were. Guaranteed, it may for some people; but, knowing everything you are performing beforehand, it makes your experience a much easier and much more enjoyable one.

The game will start when every one of the people have previously positioned the wagers around the betting circle. Another good way to get forward in casinos is always to just utilize some typically common sense. Several poker sites include major competitions on Sundays that attract a great number of internet positives. Sure, it could to some people; but, once you learn that which you are undertaking beforehand, it creates your experience a and much more fulfilling one.

Nevertheless, the final motion is going to be based on the supplier through the complete length of the bets. These bonuses usually are paid-as a percentage of the primary deposit that is made into their fresh gamers consideration. The volume of negative defeats is relatively generally increased inside the online game because of this of what some individuals state the game to be rigged or set.

However, with all the limitations of engineering in those days, it absolutely was quite comprehensible if immediate perform was unable to produce. But to become a champion on a regular basis, it takes more than only capabilities and coaching. They have specialist and live sellers who provide you with a tailored experience-you won't forget. We invested a couple of hours suspended while in the lazy stream on Sunday evening and observed it quite enjoyable.

Like all other races of activities, is a neighborhood card game that requires talent and probability. But, if you'd like in on the true activity, and start using household some good cash, you're likely to should deposit your consideration. This removes the confusion and dilemma of being forced to initially convert an unknown currency to at least one's personal before making a bet. In this manner you can filter down the machines that can advantage anyone essentially the most and figure out how to keep away from the fool machines. You may make usage of any opinions and reviews that have been manufactured concerning the casino and it would additionally enable when the presents any form of promotions for gamers, specially people who play consistently. In Australian baccarat, the ball player must produce a couple of decisions before coping the cards - these entail determining the positioning for profitable around the following palm, and price the worth of the choice.

From njs at pobox.com Sun Oct 4 20:22:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 4 Oct 2015 11:22:47 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Sat, Oct 3, 2015 at 10:50 AM, Donald Stufft wrote: > On October 2, 2015 at 10:27:36 PM, Nathaniel Smith (njs at pobox.com) wrote: >> >> So I'm extremely reluctant to give up on standardizing how to handle >> VCS checkouts. And if we're going to have a standard for that, then >> would sure be nice if we could share the work between this standard >> and the one for sdists, given how similar they are. > > Mentioned in another email, but I don't think we should give up on the VCS > handling, I just think we should defer it to another PEP. I think this PEP is > suffering from trying to use the same mechanism for VCS and sdists when they > are different things and have different considerations. The Python packaging > toolchain has a long history of suffering because it tried to reuse the same > thing across multiple "phases" of the packaging life cycle. The problem with > that is different phases have different needs, and when you're trying to use > the same thing across multiple you suffer from something that is a sort of > "Jack of All Trades, Master of None" kind of thing. I guess to make progress in this conversation I need some more detailed explanations. I totally get that there's a long history of thought and conversations behind the various assertions here like "a sdist is fundamentally different from a VCS checkout", "there must be a 1-1 mapping between sdists and wheels", "pip needs sdists that have full wheel metadata in static form", and I'm barging in from the outside with no context, but I literally have no idea why the specific design features you're asking for are desirable or even viable. Right now if I were to try and write the PEP you're asking for, then the rationale section would just be "because Donald said so" over and over :-). I couldn't write the motivation section, because I don't know any problems that the PEP you're describing would fix for me as a package author (which doesn't mean they don't exist, but!). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Oct 4 22:02:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 4 Oct 2015 21:02:44 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Let me see if I can help clarify, so it's not just Donald who says so :-) It does feel as if we're trying to explain a lot of things that "everybody knows". Clearly not everybody knows, as you don't, but what we're trying to clarify here is the de facto realities of how sdists work, and how people expect them to work. Unfortunately, there's an awful lot of things in the packaging ecosystem that are defined by existing practice, and traditionally haven't been formally documented. I'm sure it feels as if we're just repeatedly saying "it has to be like that" - but in truth, it's more that what we're saying is the definition of a sdist, as established by existing practice. I wish we could point you to a formal definition of the requirements, but unfortunately they've never been written down. With luck, one of the outcomes here will be that someone will record what a sdist is - but we need to reflect current reality, and not end up reusing the term "sdist" to mean something different from what people currently use it for. On 4 October 2015 at 19:22, Nathaniel Smith wrote: > "a sdist is fundamentally different from a VCS checkout", Specifically, a sdist is built by the packaging tools - at the moment, by "setup.py sdist", but in future by whatever tool(s) may replace distutils/setuptools. So a sdist has a defined format, and we can mandate certain things about it. In particular, we can require files to be present which are in tool-friendly formats, because the tools will build them. On the other hand, a VCS checkout is fundamentally built by a human, for use by humans. File formats need to be human-editable, we have to be prepared to work with constraints imposed by workflows and processes *other* than Python packaging tools. So we have much less ability to dictate the format. Your proposal mandates a single directory "owned" by the packaging ecosystem, which follows the git/hg/subversion model, so it's lightweight and low-risk. But you still cant realistically ask the user to maintain package data in (for example) a JSON file in that directory. > "there must be a 1-1 mapping between sdists and wheels", The fundamental reason is one I know I've mentioned here before - pip implements "pip install " by first building a wheel and then installing it. If a sdist generates two wheels, how will pip know which one to install? Also, users expect "pip wheel " to produce the wheel corresponding to the sdist. You're proposing to change that expectation - the onus is on you to justify that change. You need to consider backward compatibility in the wider sense here too - right now, there *is* a one-to-one mapping between a sdist and a wheel. If you want to change that you need to justify it, it's not enough just to claim that no-one has come up with a persuasive argument to keep things as they are. Change is not a bad thing, and "because we've always done it that way" is not a good argument, but change needs to be justified even so. > "pip needs sdists that have full wheel metadata in static form" I assume here you're now OK with the distinction between a sdist and a VCS checkout? If you still think we're saying that pip needs static metadata in *VCS checkouts* then please review the comments already made about the difference between a sdist and a VCS checkout. But basically, a sdist is a tool-generated archive that captures the state of the project and allows for *reproducible* builds of that project. If your understanding of what a sdist is differs from this, we need to stop and agree on terminology before going any further. I will concede that https://packaging.python.org/en/latest/glossary/ doesn't mention the point that a sdist needs to provide reproducible builds. But that's certainly how sdists are used at present, and how people expect them to work. Certainly, if I lost the wheel I'd built from a sdist, I'd expect to just rebuild it from the sdist and get the same wheel. Pip needs metadata to do dependency resolution. This includes project name, version, and dependency information. We could debate about whether *full* metadata is needed, but I'm not sure what the point is. Once you are recording the stuff that pip needs, why *not* record everything? There are other tools (and ad-hoc scripts) that would benefit from having the full metadata, so why would you make it harder for them to work? You claim that you want to keep your options open - but to me, it's more important to leave the *user's* options open. If we don't provide certain values, a user who needs that data has to propose a change to the format, wait for it to be implemented, and even then they can't rely on it until all projects move to the new format. Better to just require everything from the start, then users can get at whatever they need. As far as why the metadata should be static, the current sdist format does actually include static metadata, in the PKG-INFO file. So again we have a case where it's up to you to justify the backward compatibility break. But it's a little less clear-cut here, because you are proposing a new sdist format, so you've already argued for a break with the old format. Also the old format is not typically introspected, it's just used to unpack and run setup.py. So you can reasonably argue that the current state of affairs is irrelevant. However, we're talking here about whether the metadata should be statically available, or dynamically generated. The key point here is that dynamic metadata requires the tool (pip, my one-off script, whatever) to *run arbitrary code* in order to get the metadata. OK, with signing we can ensure that it's *trusted* code, but it still could do anything the project author wanted, and we can make no assumptions about what it does. That makes a tool's job much harder. A common bug report for pip is users finding that their installs fail, because setup.py requires numpy to be installed in order to run, and yet pip is running setup.py egg-info precisely to find out what the requirements are. We tell the user that the setup.py is written incorrectly, and they should install numpy and retry the install, but it's not a good user experience. And from a selfish point of view, users blame *pip* for the consequences of a project whose code to generate the metadata is buggy. Those bug reports are a drain on the time of the pip developers, as well as a frustrating experience for the users. If you want to argue that a VCS checkout, or development directory, needs to generate metatata dynamically, I won't argue. That's fine. But the sdist is a tool-generated snapshot of a *specific* release of a project (maybe "the release I made at 1:15 today for my test build", but still a specific build) and it should be perfectly possible to capture the dynamically generated metadata from the VCS checkout and store it in the sdist when it is built. If you feel that there is metadata that cannot be stored statically in the sdist, could you please give a specific example? But do remember that a sdist is intended as a *snapshot* of a VCS checkout that can be used to reproducibly build the project - so "the version number needs to include the time of the build" isn't a valid example. Paul From donald at stufft.io Sun Oct 4 22:45:05 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 4 Oct 2015 16:45:05 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 4, 2015 at 2:22:51 PM, Nathaniel Smith (njs at pobox.com) wrote: > > I guess to make progress in this conversation I need some more > detailed explanations. I totally get that there's a long history > of thought and conversations behind the various assertions > here like "a sdist is fundamentally different from a VCS checkout", > "there must be a 1-1 mapping between sdists and wheels", "pip > needs sdists that have full wheel metadata in static form", and > I'm barging in from the outside with no context, but I literally > have no idea why the specific design features you're asking for > are desirable or even viable. Right now if I were to try and write > the PEP you're asking for, then the rationale section would just > be "because Donald said so" over and over :-). I couldn't write > the motivation section, because I don't know any problems that > the PEP you're describing would fix for me as a package author > (which doesn't mean they don't exist, but!). I don't mind going into more details! I'll do the things you specifically mentioned and then if there is other things, feel free to bring them up too. I should also mention, that these are my opinions from my experiences with the toolchain and ecosystem, others may agree or disagree with me. I have strong opinions, but that doesn't make them immutable laws of the universe, although "because Donald said so" sounds like a pretty good answer to me ;) "a sdist is fundamentally different from a VCS checkout" This one I have a hard time trying to explain. They are focused on different things. With an sdist you need to have a project name, a version, a list of files, things like that. The use cases and needs for each "phase" are different. For instance, in a VCS checkout you can derrive the list of files or the version by asking the VCS but a sdist doesn't have a VCS so it has to have that baked into it. A more C centric example, is that you often times have something like autogen.sh in a C project's VCS, but you don't have the output of that checked into the VCS, however when you prepare a tarball for distribution you run autogen.sh and then include the output there. There are other differences too, in a VCS we don't really need the ability to statically read any metadata except for build dependencies and how to invoke the build tool. Most everything else can be dynamically configured because you're not distributing that. However in a sdist, we need as much of the metadata to be static as possible. Something like PyPI needs to be able to inspect any of the files uploaded to it (sdist, wheels, etc) for certain information and anything that can't be statically and safely read from it might as well not even exist as far as PyPI is concerned. We currently have the situation where we have a single file that is used for all phases of the process, dev (``setup.py develop`` & ``setup.py sdist``), building of a wheel (``setup.py bdist_wheel``) and even installation sometimes (``setup.py install``). Throughout this there are a lot of common problems where some author tried to optimize their ``setup.py`` for their development use cases and broke it for the other cases. An example of this is version handling, where it's not unusual for someone's first forray into attempting to deduplication version involves importing their thing (which works fine on their machine) and passing it into the setup kwargs. This simple thing would generally work just fine if the output of ``setup.py sdist`` produced static metadata and ``setup.py`` was no longer being used. This also becomes expressed in what interfaces you give to the toolchain at each "phase". It's important for something inside of a VCS checkout to be able to be written by human beings. This leads to wanting to use formats like INI (which is ugly) or something like TOML or YAML or some other nice, human friendly format. These formats are great for humans to write and for humans to read but are not particularly great as data interchange formats. Looking at something like JSON, msgpack, etc are far better for data interchange for computers to talk to other computers, but are not great for humans to write, edit, or even really read in many cases. If we go back to distutils2, you can see this effect happening there, they had two similar keywords arguments in their setup.cfg statements, description and description-file, these both did the same things, but just pulled from different sources (inline or via a file) forcing every tool in the chain to have to support both of these options even though it could have easily made an sdist that was distinct from the VCS code and simplified code there. I see the blurring of lines between the various phases of a package one of the fundamental flaws of distutils and setuptools. "there must be a 1-1 mapping between sdists and wheels" This has technical and social reasons. In the techincal side, the 1-1 mapping between sdists and wheels (and all other bdists) is an assumption baked into all of the tools. From PyPI's enforcement mechanisms, to pip's caching, to things like devpi and the such breaking this assumption will break a lot of code. This is all code and code is not immutable so we could of course change that, however we wouldn't be able to rely on the fact that we've fixed that assumption for many years (probably at least 5 at the earliest, 10+ is more likely). The social side is a bit more interesting though. In Debian, end users almost *never* actually interact with source packages and in near 100% of the time they are interacting soley with built packages (in fact, unlike Python, you have to manually build a deb before you can even attempt to install something). There really aren't "source packages" in Debian, just sources that happen to produce a Debian package. In Python land, a source package is still a package and people have expectations around that, I think people would be very confused if a sdist "foo-1.0.tar.gz" could produce a wheel "bar-3.0.whl". In addition, systems like Debian don't really try to protect against a malicious DD at all. Things like "prevent foo from claiming to be bar" are enforced via societal conventions and the fact that it is not an open repo and there are gatekeepers keeping everything in place. On the flip side, we let anyone upload to PyPI and rely on things like ACLs to secure things. This means that we need to know ahead of time what names a package is going to produce. The simpliest mechanism for this is to enforce a 1:1 mapping between sdist and wheel because that is an immutable property and easy to understand. I could possibly envision something that allowed this, but it would require a project to explicitly declare up front what names it will produce, and require registering those names with PyPI before you could upload a sdist that could produce those named wheels. Ultimately, I don't think the very minor benefits are worth the additional complexity and pain of trying to adapt all of the tooling and human expectations to this. "pip needs sdists that have full wheel metadata in static form" I think I could come around to the idea that some metadata doesn't make sense for a sdist, and that it really needs to be a part of wheels but not a part of sdist. I think that the argument needs to be made in the other direction though, we should assume that all metadata will be included as part of the sdist and then make an argument for why each particular piece of metadata is Wheel specific not specific to a particular version of a project. Things like name, version, description, classifiers, etc are easily able to be classified into specific to a particular (name, version) tuple. Other things like "Python ABI" are easily able to be classified into specific to a particular wheel. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From njs at pobox.com Mon Oct 5 08:29:31 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 4 Oct 2015 23:29:31 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Sat, Oct 3, 2015 at 10:50 AM, Donald Stufft wrote: > I feel like you have some sort of "a sdist is jsut a tarball of a VCS" > mentality and I don't think that idea of a sdist is generally useful. Hmm, so, between thinking this over more on the plane today and reading your and Paul's great replies, I think I see where a lot of this disjunction might be arising. I'll try to reply to those in more detail later, but first let me try to lay this out and see if it makes things clearer. First, let's drop the word "sdist", it's confusing. I'm starting from the long history and conventions around how people make what I'll call "source releases" (and in a few paragraphs will contrast with "source wheels"). 'Everyone knows' that when you release a new version of some package (in pretty much any language), then one key step is to put together a file called -.. And 'everyone knows' that if you see a file that follows this naming convention, and you download it, then what you'll find inside is: a single directory called -/, and inside this directory will be something that's almost like a VCS checkout -- it'll probably contain a README, source files in convenient places to be edited or grepped, etc. The differences from a VCS checkout (if any) will be little convenience stuff -- like ./autogen.sh will have been run already, or there will be an extra file containing a fixed version number instead of it being autodetected, or -DNDEBUG will be in the default CFLAGS, or Cython files will have been pre-translated to C -- but fundamentally it will be similar to a VCS checkout, and switching back and forth between them won't be too jarring. 95% of the time there will be a standard way to build the thing ('./configure && make && make install', or 'python setup.py install', or similar). And these kind of source releases have a rich ecosystem around them and serve a wide range of uses: they provide a low-tech archival record (while VCS's come and go), they end up in deb and rpm "original source" bundles, they get downloaded by users and built by hand (maybe with weird configury on top, like a hack to enable cross-compilation) or poked around in by hand, etc. etc. When sdists were originally designed, then "source releases" is what the designers were thinking about. Then, easy_install came along, and pulled off a clever hack where when you asked for some package by name, then it would try to automagically go out and track down any relevant source releases and build them all. And it works great, except when it doesn't. And creates massive headaches for everyone trying to work on python packaging afterwards, because source releases were not designed to be used this way. My hypothesis is that the requirements that were confusing me are based around the idea that an sdist should be something designed to slot into this particular use case: i.e., something that pip can automatically grab and work with while solving a dependency resolution problem. Therefore it really needs to have a static name, and static version number, and static dependencies, and must produce exactly one binary wheel that shares all that metadata. Let's call this a "source wheel" -- what we're really looking for here is a way to ship extension modules inside something that acts like an architecture-neutral "none-any" wheel. So: the email that started this thread was a proposal for how to standardize the format of "source releases", and Donald's counter was a proposal for how to standardize the format of "source wheels". Does that sound right? If so, then some follow-up thoughts: 1) If we design a source wheel format, then I am 100% in favor of the suggestion of giving it a unique extension like "swhl". I'm still a young whippersnapper compared to some, but I've been downloading files named -. for 20 years, and AFAICR every one of those files unpacked to make a single directory that was laid out like a VCS checkout. Obviously we can change what goes inside, but we should change the naming convention at the same time because otherwise we're just going to confuse people. 2) I think there's a strong case to be made that Python actually needs standards for *both* source releases and source wheels. There's certainly no logical contradiction -- they're conceptually different things. It sounds like we all agree that "pip" should continue to have a way to build and install an arbitrary VCS checkout, and extending that standard to cover building and installing a classic "source release" would be... almost difficult *not* to do. And I think that there will continue to be a clear need for source releases even in a world where source wheels exist, because of all those places where source releases get used that aren't automagic-easy_install/pip-builds. For example, most pure Python packages (the ones that already make "none-any" wheels) have no need at all for source wheels, but they still need source releases to serve as archival snapshots. And more complex packages that need build-time configuration (e.g. numpy) will continue to require source releases that can be configured to build wheels that have a variety of different properties (e.g., different dependency metadata), so they can't get by with source wheels alone -- but you can imagine that such projects might reasonably *in addition* provide a source wheel that locks down the same default configuration that gets used for their uploaded binary wheel builds, and is designed for pip to use when trying to resolve dependencies on platforms where a regular binary wheel is unavailable. Pictorially, this world would look like: VCS checkout -> source release \ \ --------------------+--> in-place install | +--> wheels -> install | +--> source wheels -> wheels -> install 3) It sounds like we all agree that - 'pip install ' should work - that there is some crucial metadata that VCS checkouts won't be able to provide without running arbitrary code (e.g. dependencies and version numbers) - that what metadata they do provide (e.g., which arbitrary code to run) should be specified in a human-friendly configuration file Given this, in the big picture it sounds like the only really essentially controversial things about the original proposal might be: - that 'pip install ' should work the same as 'pip install ' (does anyone actually disagree?) - the 1 paragraph describing the "transitional plan" allowing pip to automatically install from these source releases *as part of a dependency resolution plan* (as opposed to when a VCS-checkout-or-equivalent is explicitly given as an install target). Which honestly I don't like either, it's just there as a backcompat measure so that these source releases don't create a regression versus existing sdists -- note that one of the goals of the design was that current sdists could be upgraded into this format by dropping in a single static file (or by an install tool "virtually" dropping in this file when it encounters an old-style sdist -- so you don't need to keep around code to handle both cases separately). Does that sound right? (Other features of the original proposal include stuff like the lack of trivial metadata like "name" and "description", and the support for generating multiple wheels from one directory. I am explicitly calling these "inessential".) -n -- Nathaniel J. Smith -- http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Oct 5 11:03:21 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 10:03:21 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 5 October 2015 at 07:29, Nathaniel Smith wrote: > First, let's drop the word "sdist", it's confusing. I'll read your full reply later, when I have the time, but please note we can't drop the term sdist - it's a well known concept in packaging. Having said that, I'm happy if you want to restate your proposal in terms of a new concept that's not a sdist (you'll need to come up with a suitable term, and make it clear that it's distinct from a sdist when you formalise the proposal, but that's OK for now). My immediate thought is that I'm against a proposal that throws out the sdist concept in favour of something new, as there's a lot of reworking that would need to be done to achieve that, and I don't know who's going to do that. So I'd need convincing that the proposal is practical. For example, PyPI would need to be able to host these new things, and distinguish between them and sdists. But I'll hold off on detailed comments until I've read your full email. Paul From p.f.moore at gmail.com Mon Oct 5 13:28:48 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 12:28:48 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: OK, I've had a better read of your email now. Responses inline. On 5 October 2015 at 07:29, Nathaniel Smith wrote: > First, let's drop the word "sdist", it's confusing. We can't (see below for details). We can deprecate the sdist concept, if that's what you want to propose. From what I gather, you're proposing deprecating it in favour of a "source wheel" concept. I don't have a huge issue with that other than that I don't see the necessity - the sdist concept pretty much covers what you want, except maybe that it's not clear enough to people outside the packaging community how it differs from a VCS checkout. > I'm starting from the long history and conventions around how people make > what I'll call "source releases" (and in a few paragraphs will contrast with > "source wheels"). 'Everyone knows' that when you release a new version of > some package (in pretty much any language), then one key step is to put > together a file called -.. And 'everyone > knows' that if you see a file that follows this naming convention, and you > download it, then what you'll find inside is: a single directory called > -/, and inside this directory will be something that's > almost like a VCS checkout -- it'll probably contain a README, source files > in convenient places to be edited or grepped, etc. The differences from a > VCS checkout (if any) will be little convenience stuff -- like ./autogen.sh > will have been run already, or there will be an extra file containing a > fixed version number instead of it being autodetected, or -DNDEBUG will be > in the default CFLAGS, or Cython files will have been pre-translated to C -- > but fundamentally it will be similar to a VCS checkout, and switching back > and forth between them won't be too jarring. 95% of the time there will be a > standard way to build the thing ('./configure && make && make install', or > 'python setup.py install', or similar). Breaking at this point, because that's frankly *not* the reality in the Python packaging world (at least not nowadays - I'm not clear to what extent you're just talking about history and background here, although your reference to Cython makes me think you're talking in terms of current practice). It may look like that, but there are some fundamental differences. First and foremost, nobody zips up and publishes their VCS checkout in the way you describe. (At least not if they are using the standard tools - distutils and setuptools). Instead, they create a "sdist" using the "python setup.py sdist" command. I'm sorry, but I'm going to carry on using the "sdist" term here, because I'm describing current practice and sdists *are* current practice. The difference is between a sdist and what you call a "source release" is subtle, precisely because the current sdist format is a bit of a mess, but the key point is that all sdists are created by a standard process, and conform to a standard naming convention and layout. The packaging tools rely on being able to make that assumption, in all sorts of ways which we're doing our best to clarify as part of this thread, but which honestly have been a little bit implicit up to this point. Further muddying the water is the fact that as you say, pip needs to be able to build from a VCS checkout (a directory on the user's local system) and we have code in pip that does that - mostly by assuming that you can treat a VCS checkout as an unpacked sdist, but there are hacks we need to do to make that work (we run setup.py egg-info to get the metadata we need, for example, which has implications as we only get that data at a later stage than we have it in the sdist case) and differences in functionality (develop mode). At this point I'm not saying that things have to be this way, or even that "make a source release however you choose as long as it follows these conventions" isn't a viable way forward, but I do think we need to agree on our picture of how things are now, or we'll continue talking past each other. > And these kind of source releases > have a rich ecosystem around them and serve a wide range of uses: they > provide a low-tech archival record (while VCS's come and go), they end up in > deb and rpm "original source" bundles, they get downloaded by users and > built by hand (maybe with weird configury on top, like a hack to enable > cross-compilation) or poked around in by hand, etc. etc. When sdists were > originally designed, then "source releases" is what the designers were > thinking about. This, on the other hand, I suspect is not far from the truth. When sdists were designed, they were a convenience for bundling the stuff needed to do setup.py install later, possibly on a different machine. But that's a long time ago, and not really relevant now. For better or worse. Unless you are suggesting that we go all the way back to that original point? Which you may be, but that means discarding the work that's been done based on the sdist concept since then. Which leads nicely on to... > Then, easy_install came along, and pulled off a clever hack where when you > asked for some package by name, then it would try to automagically go out > and track down any relevant source releases and build them all. And it works > great, except when it doesn't. And creates massive headaches for everyone > trying to work on python packaging afterwards, because source releases were > not designed to be used this way. > > My hypothesis is that the requirements that were confusing me are based > around the idea that an sdist should be something designed to slot into this > particular use case: i.e., something that pip can automatically grab and > work with while solving a dependency resolution problem. Therefore it really > needs to have a static name, and static version number, and static > dependencies, and must produce exactly one binary wheel that shares all that > metadata. Anyone who knows my history will know that I'm the last person to defend setuptools' hacks, but you hit the nail on the head above. When it works, it works great (meh, "sufficiently well" :-)) And pip *needs* to do static dependency resolution. We have enough bug reports and feature requests asking that we improve the dependency resolution process that nobody is going to be happy with anything that doesn't allow for at least as much static information as we currently have, and ultimately more. > Let's call this a "source wheel" -- what we're really looking for > here is a way to ship extension modules inside something that acts like an > architecture-neutral "none-any" wheel. I don't understand this statement. What do extension modules matter here? We need to be able to ship sources in a form that can participate in dependency resolution (and any other potential discovery processes that may turn up in future) without having to run code to do so. The reasons for this are: 1. Running code is a performance overhead, and possibly even a security risk (even trusted code may behave in ways you didn't anticipate). We want to do as little as possible of that as we can, and in particular we want to discard invalid candidate files without running any of their code. 2. Running code introduces the possibility of that code failing. We don't want end users to have installs fail because code in distributions we're going to discard is buggy. 3. Repositories like PyPI need to present project metadata, for both human and tool consumption - they can only do this if it's available statically. You seem to be thinking that binary wheels are sufficient for this for pure-Python code. Look at it the other way - we discard sdists from the dependency calculations whenever there's an equivalent binary wheel available. That's always for non-any wheels, but less often for architecture-dependent wheels. But in order to know that the wheel is equivalent, we need to match it with the sdist - so the sdist needs the metadata you're trying to argue against providing... > So: the email that started this thread was a proposal for how to standardize > the format of "source releases", and Donald's counter was a proposal for how > to standardize the format of "source wheels". Does that sound right? Well, essentially yes, although I read it as your original email being a proposal for a new format to replace sdists, and Donald's and my counter is that there's already a been a certain amount of thinking and design gone into how we move from the current ad-hoc sdist format to a better defined and specified "next version", so how does your proposal affect that? It seems that your answer is that you want to bypass that and offer an alternative. Is that fair? For the record, I don't like the term "source wheel" and would prefer to stick with "sdist" if appropriate, or choose a term that doesn't include the word "wheel" otherwise (as wheels seem to me to be strongly, and beneficially, linked in people's minds to the concept of a binary release format). > If so, then some follow-up thoughts: > > 1) If we design a source wheel format, then I am 100% in favor of the > suggestion of giving it a unique extension like "swhl". I'm still a young > whippersnapper compared to some, but I've been downloading files named > -. for 20 years, and AFAICR every one of > those files unpacked to make a single directory that was laid out like a VCS > checkout. Obviously we can change what goes inside, but we should change the > naming convention at the same time because otherwise we're just going to > confuse people. I have no problem with making it clear that "sdist version 2" or "source wheel" is not the same as a packed VCS checkout. I don't see the need for a new term, I'd be happy with "-.sdist" as the name. I'd also like to emphasize strongly that PyPI only hosts sdists, and *not* source releases - that source releases are typically only seen in Python in the form of a VCS checkout or development directory. (There's an implication there that we need to explore, that pip won't necessarily gain the ability to be pointed at a non-sdist format packed "source release" archive, and download it and process it. That's not a given, but I'd like to be sure we are happy with the potential re-introduction of confusion over the distinction between a sdist/source wheel and a source release that would result). > 2) I think there's a strong case to be made that Python actually needs > standards for *both* source releases and source wheels. There's certainly no > logical contradiction -- they're conceptually different things. It sounds > like we all agree that "pip" should continue to have a way to build and > install an arbitrary VCS checkout, and extending that standard to cover > building and installing a classic "source release" would be... almost > difficult *not* to do. As noted above, I'm happy for that discussion to occur. But I'm *not* sure the case is as strong as you think. Technically, it's certainly not too hard, but the social issues are what concern me. How will we explain to someone that they can't upload their file to PyPI because it's a "source release" not a "source wheel"? What is the implication on people's workflow? How would we explain why people might want to make "source releases" *at all*? Personally, I can only see a need in my personal experience for a VCS url that people can clone, a packaged source artifact that I can upload to PyPI for automatic consumption, and (binary) wheels. That second item is a "source wheel" - not a "source release". > And I think that there will continue to be a clear need for source releases > even in a world where source wheels exist, because of all those places where > source releases get used that aren't automagic-easy_install/pip-builds. For > example, most pure Python packages (the ones that already make "none-any" > wheels) have no need at all for source wheels, but they still need source > releases to serve as archival snapshots. And more complex packages that need > build-time configuration (e.g. numpy) will continue to require source > releases that can be configured to build wheels that have a variety of > different properties (e.g., different dependency metadata), so they can't > get by with source wheels alone -- but you can imagine that such projects > might reasonably *in addition* provide a source wheel that locks down the > same default configuration that gets used for their uploaded binary wheel > builds, and is designed for pip to use when trying to resolve dependencies > on platforms where a regular binary wheel is unavailable. > > Pictorially, this world would look like: > > VCS checkout -> source release > \ \ > --------------------+--> in-place install > | > +--> wheels -> install > | > +--> source wheels -> wheels -> install I don't see the need for source releases that you do. That's likely because I don't deal with the sorts of complex projects you do, though, so I'm not dismissing the issue. As I say, my objections are mostly non-technical. I do think you should consider how to document "what a source release is intended to achieve" in a way that explains it to people who don't need the complexity it adds - and with the explicit goal of making sure that you dissuade people who *don't* need source releases from thinking they do. > 3) It sounds like we all agree that > - 'pip install ' should work Yes. > - that there is some crucial metadata that VCS checkouts won't be able to > provide without running arbitrary code (e.g. dependencies and version > numbers) I'm still resisting this one, although I can live with "Nathanial tells me so" :-) > - that what metadata they do provide (e.g., which arbitrary code to run) > should be specified in a human-friendly configuration file I don't agree to that one particularly, in the sense that I don't really care. I'd be happy with a system that said something like that for a VCS checkout, pip expects "setup.py egg-info" and "setup.py sdist" to work, and produce respectively a set of static metadata in a known location, and a properly formatted "source wheel"/sdist file. Non-distutils build tools can write a wrapper setup.py that works however they prefer. (That's roughly what we have now, BTW). > Given this, in the big picture it sounds like the only really essentially > controversial things about the original proposal might be: > - that 'pip install ' should work the same as > 'pip install ' (does anyone actually disagree?) Yes, to the extent that I want to ensure it's clearly documented how, and why, this differs from a sdist/source wheel. But that's not a technical issue. > - the 1 paragraph describing the "transitional plan" allowing pip to > automatically install from these source releases *as part of a dependency > resolution plan* (as opposed to when a VCS-checkout-or-equivalent is > explicitly given as an install target). Which honestly I don't like either, > it's just there as a backcompat measure so that these source releases don't > create a regression versus existing sdists -- note that one of the goals of > the design was that current sdists could be upgraded into this format by > dropping in a single static file (or by an install tool "virtually" dropping > in this file when it encounters an old-style sdist -- so you don't need to > keep around code to handle both cases separately). Given that these source releases won't be hosted on PyPI (as the proposal currently stands) there's no real need for this - all you need to say is that you can point pip at any old URL and take your chances :-) > Does that sound right? Not really. For me the big controversy is whether we move forward from where we are with sdists, or we ignore the current sdist mechanism and start over. A key further question, which I don't think has been stated explicitly until I started this email, is what formats will be supported for hosting on PyPI. I am against hosting formats that don't support static metadata, such as your "source distribution", as I don't see how PyPI would be able to publish the metadata if it weren't static. And following on from that, we need to agree whether the key formats should be required to have a static version. I'm OK with a VCS checkout having a dynamically generated version, that's part of the "all bets are off" contract over such things (if you don't generate a version that reflects every change, you get to deal with the consequences) but I don't think that's a reasonable thing to allow in "published" formats. > (Other features of the original proposal include stuff like the lack of > trivial metadata like "name" and "description", and the support for > generating multiple wheels from one directory. I am explicitly calling these > "inessential".) Not sure what you mean by lack of name/description being "inessential". The double negative confuses me. Do you mean you're OK with requiring them? Fair enough. For multiple wheels, I'd tend to consider the opposite to be true - it's not that the capability is non-essential, but rather that in published formats (source wheel and later in the chain) it's essential that one source generates one target. Paul From donald at stufft.io Mon Oct 5 14:44:04 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 08:44:04 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 2:29:35 AM, Nathaniel Smith (njs at pobox.com) wrote: > > Does that sound right? Off the bat, I'm perfectly fine with `pip install archive-made-from-vcs.tar.gz` working the same as `pip install git+https://github.com/foo/bar.git``. The fact that one is a VCS and one isn't is immaterial to me that's just a transport mechanism for an arbitrary collection of files and directories. These arbitrary collections of file and directories can only provide a very minimal amount of information about the thing that is contained inside of them, and indeed there may even be multiple things inside of this rooted within different sub directories of the top level arbitrary collection of files. I agree with Paul though, I don't see any way this arbitrary collection of files/directories can be distributed on PyPI. PyPI is not just a place to dump whatever random archives you want for a package, it's specifically a place for Python package files (sdists, wheels, and eggs right now for the most part). It sounds like maybe you're just looking for a way to make it so that pip no longer makes these "arbitrary files/directories" installs work by treating them? like unpacked sdists and instead follows some supported path. If so, that is reasonable to me (and something I planned on getting around to). If that is what you're shooting for, I think it got confused by trying to mix in the sdist concept, as our sdists and wheels are not really for human consumption any more. I don't think that it makes sense for pip to go directly from a VCS[1] to a Wheel in the grand scheme of things. Right now we kind of do it, but that's because we just treat them like an unpacked sdist [2], long term though I don't think that is the correct way to do things. We (I?) want to minimize the different installation "paths" that can be taken and the main variable then when you do a ``pip install`` is how far long that path we already are. My ideal path looks something like this: ? ? VCS -> Source Wheel [3] -> Wheel -> Installed ? ? \-> Inplace Installation [4] So in that regards, I think that the only things (in my ideal world) people should be uploading to PyPI are source wheels (mandatory? [5]) and binary wheels (optional?). I don't know if downstream would prefer to use a source wheel or an arbitrary collection of files that may or may not be in a tarball, but I'm focused primarily on the needs of our toolchain, while still trying to make sure we don't end up in a situation that hurts downstream as well. I also don't think it makes sense for pip to ever really install these items, no matter what the transport mechanism is (VCS, tarball, unpacked on disk) without being explicitly pointed to by an URL or file path. There's obviously some backwards compatability issues here, because we can't just stop fetching .tar.gz links or anything, but I think the expectation should be that these items are only ever directly installed, not installed as part of the dependency resolution process. In that vein, they don't really participate in the dependency resolution process either (we'll install their dependencies and what not, but we'll assume that since you're pointing us to an explicit archive that we aren't going to resolve that particular dependency to anything other than what you've explicitly given us). If we're only installing these items when explicitly being given them by a direct URL or path, then a lot of the worries about non static metadata no longer exist, because as a user you're explicitly opting into installing something that isn't a real package, but is a VCS install of something that could be a package. Which we'll fetch, turn into a source wheel, and then turn that into a binary wheel (or do an inplace install). I also don't think it makes sense for these VCS installs to directly support outputting multiple different source wheels, but instead rely on the fact pip lets you say "install this artifact, but first CD into a particular directory". So you'd essentially just structure the filesystem of your VCS install to make independent little mini projects that could be independently packaged into their own VCS installs, but are all contained within some larger thing and we need to install a specific sub directory. I'm happy to be convinced otherwise, but given that this is a sort of edge case that a project will need this and we already have the subdirectory support I think it's simpler to just leverage that. Given my desire to reduce the number of installation paths we actually support, I think that trying to standardize this concept of a VCS install depends first on standardizing the concept of a source wheel (or sdist 2.0). This is because, in my mind, the main two things you can do with a VCS install, from pip's point of view, is do an in place installation or create a source wheel. I also agree with Paul that we should not be adding new formats to PyPI that do not support static metadata. What metadata in specific makes sense for that particular format (like ABI doesn't make sense for a source wheel/sdist) is going to be format dependent, but there should never be a "run this bit of code to find out X" situation. The closest we should get is "that type of metadata doesn't make sense for X, you need to build X into a Y for it". PyPI needs static metadata [6], and static metadata makes other tools easier to operate as well. [1] I'm going to just call these VCS installs, but I mean any install that is ? ? not a fully formed Python package and is instead just an arbitrary ? ? collection of files and directories where we can have a minimal amount of ? ? control over the structure. [2] This goes back to my belief that one of the "original sins" of distutils ? ? and setuptools was blurring the lines between the different phases in the ? ? life cycle of a package. [3] Unlike Paul, I think a Source Wheel is actually a decent name for this ? ? concept. It's similar to .rpm and .src.rpm in that world, and I think it ? ? makes it more obvious that this item isn't an installable item in it's own ? ? right, that it exists in order to produce binary wheels. However, this ? ? concept is currently being handled by the sdist "format", however ? ? ambigiously defined that currently is. I also think it makes it a bit ? ? easier to get rid of the ambigous "package", since we can just call them ? ? all "wheels" which is easier to say than "distribution". [4] Even though I want to reduce the number of paths we take, I don't think ? ? we'll ever be able to reasonably get rid of the inplace installation path. ? ? There will probably need to be some back and forth about exactly which ? ? parts of the inplace install the installer should be responsible for and ? ? which parts the build tool is responsible for. [5] Mandatory in the sense of, if you're going to upload any files to PyPI you ? ? must upload one of these, not mandatory in the absolute sense. In a future ? ? world, ideally the only thing people will need to upload is a source wheel ? ? and we'll have a build farm that will take that and automatically produce ? ? binary wheels from them. [6] In reality, we don't have static metadata today, and we get by. This is ? ? mostly because we force uploads to include static metadata alongside the ? ? upload and we present that instead. In the future I want to move us to a ? ? situation where you *just* upload the file, and PyPI inspects the file for ? ? all of the metadata it needs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Mon Oct 5 15:11:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 14:11:56 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 5 October 2015 at 13:44, Donald Stufft wrote: > [3] Unlike Paul, I think a Source Wheel is actually a decent name for this > concept. It's similar to .rpm and .src.rpm in that world, and I think it > makes it more obvious that this item isn't an installable item in it's own > right, that it exists in order to produce binary wheels. However, this > concept is currently being handled by the sdist "format", however > ambigiously defined that currently is. I also think it makes it a bit > easier to get rid of the ambigous "package", since we can just call them > all "wheels" which is easier to say than "distribution". Being a Windows user, I hadn't caught the parallel between Wheel-Source wheel and RPM-Source RPM (is there also a similar deb-source deb pair?) But if the concept works for people with a Linux background, I'm OK with it. (My main concern is that end users commonly make requests to projects saying "please provide wheels". I don't want that nice simple concept to get confused, if we can avoid it). Paul From solipsis at pitrou.net Mon Oct 5 15:18:37 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 15:18:37 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils References: Message-ID: <20151005151837.35894286@fsol> On Mon, 5 Oct 2015 14:11:56 +0100 Paul Moore wrote: > > Being a Windows user, I hadn't caught the parallel between > Wheel-Source wheel and RPM-Source RPM (is there also a similar > deb-source deb pair?) > > But if the concept works for people with a Linux background, I'm OK with it. The "source RPM" concept is actually confusing since, IIUC, a "source RPM" is nothing like a normal RPM (i.e. you can't install it using the "rpm" tool). It should actually be called a "RPM source" (again, IIUC). > (My main concern is that end users commonly make requests to projects > saying "please provide wheels". I don't want that nice simple concept > to get confused, if we can avoid it). +1 Regards Antoine. From solipsis at pitrou.net Mon Oct 5 15:22:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 15:22:01 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils References: Message-ID: <20151005152201.4433e9a6@fsol> On Mon, 5 Oct 2015 08:44:04 -0400 Donald Stufft wrote: > > I don't think that it makes sense for pip to go directly from a VCS[1] to a > Wheel in the grand scheme of things. Right now we kind of do it, but that's > because we just treat them like an unpacked sdist [2], long term though I don't > think that is the correct way to do things. We (I?) want to minimize the > different installation "paths" that can be taken and the main variable then > when you do a ``pip install`` is how far long that path we already are. My > ideal path looks something like this: > > ? ? VCS -> Source Wheel [3] -> Wheel -> Installed > ? ? \-> Inplace Installation [4] A valid use case may be to do an in-place installation from a sdist (although you may question the sanity of doing development from a source tree which isn't VCS-backed :-)). In any case, sdists being apt for human consumption is an important feature IMHO. Regards Antoine. From donald at stufft.io Mon Oct 5 15:25:39 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:25:39 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 9:11:59 AM, Paul Moore (p.f.moore at gmail.com) wrote: > On 5 October 2015 at 13:44, Donald Stufft wrote: > > [3] Unlike Paul, I think a Source Wheel is actually a decent name for this > > concept. It's similar to .rpm and .src.rpm in that world, and I think it > > makes it more obvious that this item isn't an installable item in it's own > > right, that it exists in order to produce binary wheels. However, this > > concept is currently being handled by the sdist "format", however > > ambigiously defined that currently is. I also think it makes it a bit > > easier to get rid of the ambigous "package", since we can just call them > > all "wheels" which is easier to say than "distribution". > > Being a Windows user, I hadn't caught the parallel between > Wheel-Source wheel and RPM-Source RPM (is there also a similar > deb-source deb pair?) > > But if the concept works for people with a Linux background, I'm OK with it. > > (My main concern is that end users commonly make requests to projects > saying "please provide wheels". I don't want that nice simple concept > to get confused, if we can avoid it). > > Paul > I'm not dead set on the source wheel name or anything either fwiw. I liked the parallels between that and .src.rpm and .rpm and I think the "sdist" name is kind of a mouthful to say. The debian analog to a source rpm is just a "source package", but that's really just a directory that you can run the debian build tools in, it's not really a package format like source RPMs are. Source RPMs have an "install" concept, but it's not really like pip's install, it just unpacks the file into place so that you can later build it. The main thing I wanted was something that wasn't a mouthful to say and that had a dedicated extension so we don't end up where the filename looks incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). I liked the idea of tying it into Wheels to keep the "new guard" of packaging formats related to each other, but I'm also happy to use a different term. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Mon Oct 5 15:26:30 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 05 Oct 2015 13:26:30 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: Axle? On Mon, Oct 5, 2015 at 9:25 AM Donald Stufft wrote: > On October 5, 2015 at 9:11:59 AM, Paul Moore (p.f.moore at gmail.com) wrote: > > On 5 October 2015 at 13:44, Donald Stufft wrote: > > > [3] Unlike Paul, I think a Source Wheel is actually a decent name for > this > > > concept. It's similar to .rpm and .src.rpm in that world, and I think > it > > > makes it more obvious that this item isn't an installable item in it's > own > > > right, that it exists in order to produce binary wheels. However, this > > > concept is currently being handled by the sdist "format", however > > > ambigiously defined that currently is. I also think it makes it a bit > > > easier to get rid of the ambigous "package", since we can just call > them > > > all "wheels" which is easier to say than "distribution". > > > > Being a Windows user, I hadn't caught the parallel between > > Wheel-Source wheel and RPM-Source RPM (is there also a similar > > deb-source deb pair?) > > > > But if the concept works for people with a Linux background, I'm OK with > it. > > > > (My main concern is that end users commonly make requests to projects > > saying "please provide wheels". I don't want that nice simple concept > > to get confused, if we can avoid it). > > > > Paul > > > > I'm not dead set on the source wheel name or anything either fwiw. I liked > the > parallels between that and .src.rpm and .rpm and I think the "sdist" name > is > kind of a mouthful to say. The debian analog to a source rpm is just a > "source > package", but that's really just a directory that you can run the debian > build > tools in, it's not really a package format like source RPMs are. Source > RPMs > have an "install" concept, but it's not really like pip's install, it just > unpacks the file into place so that you can later build it. > > The main thing I wanted was something that wasn't a mouthful to say and > that > had a dedicated extension so we don't end up where the filename looks > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). I liked the > idea > of tying it into Wheels to keep the "new guard" of packaging formats > related to > each other, but I'm also happy to use a different term. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 5 15:28:48 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:28:48 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005152201.4433e9a6@fsol> References: <20151005152201.4433e9a6@fsol> Message-ID: On October 5, 2015 at 9:25:24 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Mon, 5 Oct 2015 08:44:04 -0400 > Donald Stufft wrote: > > > > I don't think that it makes sense for pip to go directly from a VCS[1] to a > > Wheel in the grand scheme of things. Right now we kind of do it, but that's > > because we just treat them like an unpacked sdist [2], long term though I don't > > think that is the correct way to do things. We (I?) want to minimize the > > different installation "paths" that can be taken and the main variable then > > when you do a ``pip install`` is how far long that path we already are. My > > ideal path looks something like this: > > > > VCS -> Source Wheel [3] -> Wheel -> Installed > > \-> Inplace Installation [4] > > A valid use case may be to do an in-place installation from a sdist > (although you may question the sanity of doing development from a > source tree which isn't VCS-backed :-)). > > In any case, sdists being apt for human consumption is an important > feature IMHO. > I don't think so. Remember in my footnote I mentioned that I wasn't using VCS to mean literally "something checked into a VCS", but rather the more traditional (outside of Python) "source release" concept. This could be a simple tarball that just has the proper, for human consumption, files in it or it could be a VCS checkout, or it could just be some files sitting on disk. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Mon Oct 5 15:25:14 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 05 Oct 2015 13:25:14 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: The OP asks for a Python callable interface to pip instead of setup.py's command line interface. That could be accomplished now by figuring out all of the arguments that pip will send to setup.py (setup.py egg_info and setup.py bdist_wheel)?, and then by writing a setup.py emulator that implements those commands by invoking the discovered callables, but it's hard to know exactly what that command line interface is. It has been a packaging TODO to write the list of arguments pip can send to setup.py down... Suppose someone from the pip team wrote the generic setup.py. It would implement the command line interface, import a script called, say, build.py, and invoke Python callables to do egg_info and bdist_wheel. Then the flit author could implement a couple of functions instead of having to reverse-engineer a command line interface. This would be an improvement because build system authors do not know how to extend setuptools or reverse engineer the necessary setup.py command line interface. On Mon, Oct 5, 2015 at 7:28 AM Paul Moore wrote: > OK, I've had a better read of your email now. Responses inline. > > On 5 October 2015 at 07:29, Nathaniel Smith wrote: > > First, let's drop the word "sdist", it's confusing. > > We can't (see below for details). We can deprecate the sdist concept, > if that's what you want to propose. From what I gather, you're > proposing deprecating it in favour of a "source wheel" concept. I > don't have a huge issue with that other than that I don't see the > necessity - the sdist concept pretty much covers what you want, except > maybe that it's not clear enough to people outside the packaging > community how it differs from a VCS checkout. > > > I'm starting from the long history and conventions around how people make > > what I'll call "source releases" (and in a few paragraphs will contrast > with > > "source wheels"). 'Everyone knows' that when you release a new version of > > some package (in pretty much any language), then one key step is to put > > together a file called -.. And > 'everyone > > knows' that if you see a file that follows this naming convention, and > you > > download it, then what you'll find inside is: a single directory called > > -/, and inside this directory will be something that's > > almost like a VCS checkout -- it'll probably contain a README, source > files > > in convenient places to be edited or grepped, etc. The differences from a > > VCS checkout (if any) will be little convenience stuff -- like > ./autogen.sh > > will have been run already, or there will be an extra file containing a > > fixed version number instead of it being autodetected, or -DNDEBUG will > be > > in the default CFLAGS, or Cython files will have been pre-translated to > C -- > > but fundamentally it will be similar to a VCS checkout, and switching > back > > and forth between them won't be too jarring. 95% of the time there will > be a > > standard way to build the thing ('./configure && make && make install', > or > > 'python setup.py install', or similar). > > Breaking at this point, because that's frankly *not* the reality in > the Python packaging world (at least not nowadays - I'm not clear to > what extent you're just talking about history and background here, > although your reference to Cython makes me think you're talking in > terms of current practice). It may look like that, but there are some > fundamental differences. > > First and foremost, nobody zips up and publishes their VCS checkout in > the way you describe. (At least not if they are using the standard > tools - distutils and setuptools). Instead, they create a "sdist" > using the "python setup.py sdist" command. I'm sorry, but I'm going to > carry on using the "sdist" term here, because I'm describing current > practice and sdists *are* current practice. > > The difference is between a sdist and what you call a "source release" > is subtle, precisely because the current sdist format is a bit of a > mess, but the key point is that all sdists are created by a standard > process, and conform to a standard naming convention and layout. The > packaging tools rely on being able to make that assumption, in all > sorts of ways which we're doing our best to clarify as part of this > thread, but which honestly have been a little bit implicit up to this > point. > > Further muddying the water is the fact that as you say, pip needs to > be able to build from a VCS checkout (a directory on the user's local > system) and we have code in pip that does that - mostly by assuming > that you can treat a VCS checkout as an unpacked sdist, but there are > hacks we need to do to make that work (we run setup.py egg-info to get > the metadata we need, for example, which has implications as we only > get that data at a later stage than we have it in the sdist case) and > differences in functionality (develop mode). > > At this point I'm not saying that things have to be this way, or even > that "make a source release however you choose as long as it follows > these conventions" isn't a viable way forward, but I do think we need > to agree on our picture of how things are now, or we'll continue > talking past each other. > > > And these kind of source releases > > have a rich ecosystem around them and serve a wide range of uses: they > > provide a low-tech archival record (while VCS's come and go), they end > up in > > deb and rpm "original source" bundles, they get downloaded by users and > > built by hand (maybe with weird configury on top, like a hack to enable > > cross-compilation) or poked around in by hand, etc. etc. When sdists were > > originally designed, then "source releases" is what the designers were > > thinking about. > > This, on the other hand, I suspect is not far from the truth. When > sdists were designed, they were a convenience for bundling the stuff > needed to do setup.py install later, possibly on a different machine. > > But that's a long time ago, and not really relevant now. For better or > worse. Unless you are suggesting that we go all the way back to that > original point? Which you may be, but that means discarding the work > that's been done based on the sdist concept since then. Which leads > nicely on to... > > > Then, easy_install came along, and pulled off a clever hack where when > you > > asked for some package by name, then it would try to automagically go out > > and track down any relevant source releases and build them all. And it > works > > great, except when it doesn't. And creates massive headaches for everyone > > trying to work on python packaging afterwards, because source releases > were > > not designed to be used this way. > > > > My hypothesis is that the requirements that were confusing me are based > > around the idea that an sdist should be something designed to slot into > this > > particular use case: i.e., something that pip can automatically grab and > > work with while solving a dependency resolution problem. Therefore it > really > > needs to have a static name, and static version number, and static > > dependencies, and must produce exactly one binary wheel that shares all > that > > metadata. > > Anyone who knows my history will know that I'm the last person to > defend setuptools' hacks, but you hit the nail on the head above. When > it works, it works great (meh, "sufficiently well" :-)) > > And pip *needs* to do static dependency resolution. We have enough bug > reports and feature requests asking that we improve the dependency > resolution process that nobody is going to be happy with anything that > doesn't allow for at least as much static information as we currently > have, and ultimately more. > > > Let's call this a "source wheel" -- what we're really looking for > > here is a way to ship extension modules inside something that acts like > an > > architecture-neutral "none-any" wheel. > > I don't understand this statement. What do extension modules matter > here? We need to be able to ship sources in a form that can > participate in dependency resolution (and any other potential > discovery processes that may turn up in future) without having to run > code to do so. The reasons for this are: > > 1. Running code is a performance overhead, and possibly even a > security risk (even trusted code may behave in ways you didn't > anticipate). We want to do as little as possible of that as we can, > and in particular we want to discard invalid candidate files without > running any of their code. > 2. Running code introduces the possibility of that code failing. We > don't want end users to have installs fail because code in > distributions we're going to discard is buggy. > 3. Repositories like PyPI need to present project metadata, for both > human and tool consumption - they can only do this if it's available > statically. > > You seem to be thinking that binary wheels are sufficient for this for > pure-Python code. Look at it the other way - we discard sdists from > the dependency calculations whenever there's an equivalent binary > wheel available. That's always for non-any wheels, but less often for > architecture-dependent wheels. But in order to know that the wheel is > equivalent, we need to match it with the sdist - so the sdist needs > the metadata you're trying to argue against providing... > > > So: the email that started this thread was a proposal for how to > standardize > > the format of "source releases", and Donald's counter was a proposal for > how > > to standardize the format of "source wheels". Does that sound right? > > Well, essentially yes, although I read it as your original email being > a proposal for a new format to replace sdists, and Donald's and my > counter is that there's already a been a certain amount of thinking > and design gone into how we move from the current ad-hoc sdist format > to a better defined and specified "next version", so how does your > proposal affect that? > > It seems that your answer is that you want to bypass that and offer an > alternative. Is that fair? > > For the record, I don't like the term "source wheel" and would prefer > to stick with "sdist" if appropriate, or choose a term that doesn't > include the word "wheel" otherwise (as wheels seem to me to be > strongly, and beneficially, linked in people's minds to the concept of > a binary release format). > > > If so, then some follow-up thoughts: > > > > 1) If we design a source wheel format, then I am 100% in favor of the > > suggestion of giving it a unique extension like "swhl". I'm still a young > > whippersnapper compared to some, but I've been downloading files named > > -. for 20 years, and AFAICR every one of > > those files unpacked to make a single directory that was laid out like a > VCS > > checkout. Obviously we can change what goes inside, but we should change > the > > naming convention at the same time because otherwise we're just going to > > confuse people. > > I have no problem with making it clear that "sdist version 2" or > "source wheel" is not the same as a packed VCS checkout. I don't see > the need for a new term, I'd be happy with "-.sdist" > as the name. I'd also like to emphasize strongly that PyPI only hosts > sdists, and *not* source releases - that source releases are typically > only seen in Python in the form of a VCS checkout or development > directory. > > (There's an implication there that we need to explore, that pip won't > necessarily gain the ability to be pointed at a non-sdist format > packed "source release" archive, and download it and process it. > That's not a given, but I'd like to be sure we are happy with the > potential re-introduction of confusion over the distinction between a > sdist/source wheel and a source release that would result). > > > 2) I think there's a strong case to be made that Python actually needs > > standards for *both* source releases and source wheels. There's > certainly no > > logical contradiction -- they're conceptually different things. It sounds > > like we all agree that "pip" should continue to have a way to build and > > install an arbitrary VCS checkout, and extending that standard to cover > > building and installing a classic "source release" would be... almost > > difficult *not* to do. > > As noted above, I'm happy for that discussion to occur. But I'm *not* > sure the case is as strong as you think. Technically, it's certainly > not too hard, but the social issues are what concern me. How will we > explain to someone that they can't upload their file to PyPI because > it's a "source release" not a "source wheel"? What is the implication > on people's workflow? How would we explain why people might want to > make "source releases" *at all*? Personally, I can only see a need in > my personal experience for a VCS url that people can clone, a packaged > source artifact that I can upload to PyPI for automatic consumption, > and (binary) wheels. That second item is a "source wheel" - not a > "source release". > > > And I think that there will continue to be a clear need for source > releases > > even in a world where source wheels exist, because of all those places > where > > source releases get used that aren't automagic-easy_install/pip-builds. > For > > example, most pure Python packages (the ones that already make "none-any" > > wheels) have no need at all for source wheels, but they still need source > > releases to serve as archival snapshots. And more complex packages that > need > > build-time configuration (e.g. numpy) will continue to require source > > releases that can be configured to build wheels that have a variety of > > different properties (e.g., different dependency metadata), so they can't > > get by with source wheels alone -- but you can imagine that such projects > > might reasonably *in addition* provide a source wheel that locks down the > > same default configuration that gets used for their uploaded binary wheel > > builds, and is designed for pip to use when trying to resolve > dependencies > > on platforms where a regular binary wheel is unavailable. > > > > Pictorially, this world would look like: > > > > VCS checkout -> source release > > \ \ > > --------------------+--> in-place install > > | > > +--> wheels -> install > > | > > +--> source wheels -> wheels -> install > > I don't see the need for source releases that you do. That's likely > because I don't deal with the sorts of complex projects you do, > though, so I'm not dismissing the issue. As I say, my objections are > mostly non-technical. I do think you should consider how to document > "what a source release is intended to achieve" in a way that explains > it to people who don't need the complexity it adds - and with the > explicit goal of making sure that you dissuade people who *don't* need > source releases from thinking they do. > > > 3) It sounds like we all agree that > > - 'pip install ' should work > > Yes. > > > - that there is some crucial metadata that VCS checkouts won't be able > to > > provide without running arbitrary code (e.g. dependencies and version > > numbers) > > I'm still resisting this one, although I can live with "Nathanial > tells me so" :-) > > > - that what metadata they do provide (e.g., which arbitrary code to > run) > > should be specified in a human-friendly configuration file > > I don't agree to that one particularly, in the sense that I don't > really care. I'd be happy with a system that said something like that > for a VCS checkout, pip expects "setup.py egg-info" and "setup.py > sdist" to work, and produce respectively a set of static metadata in a > known location, and a properly formatted "source wheel"/sdist file. > Non-distutils build tools can write a wrapper setup.py that works > however they prefer. (That's roughly what we have now, BTW). > > > Given this, in the big picture it sounds like the only really essentially > > controversial things about the original proposal might be: > > - that 'pip install ' should work the same as > > 'pip install ' (does anyone actually disagree?) > > Yes, to the extent that I want to ensure it's clearly documented how, > and why, this differs from a sdist/source wheel. But that's not a > technical issue. > > > - the 1 paragraph describing the "transitional plan" allowing pip to > > automatically install from these source releases *as part of a dependency > > resolution plan* (as opposed to when a VCS-checkout-or-equivalent is > > explicitly given as an install target). Which honestly I don't like > either, > > it's just there as a backcompat measure so that these source releases > don't > > create a regression versus existing sdists -- note that one of the goals > of > > the design was that current sdists could be upgraded into this format by > > dropping in a single static file (or by an install tool "virtually" > dropping > > in this file when it encounters an old-style sdist -- so you don't need > to > > keep around code to handle both cases separately). > > Given that these source releases won't be hosted on PyPI (as the > proposal currently stands) there's no real need for this - all you > need to say is that you can point pip at any old URL and take your > chances :-) > > > Does that sound right? > > Not really. For me the big controversy is whether we move forward from > where we are with sdists, or we ignore the current sdist mechanism and > start over. > > A key further question, which I don't think has been stated explicitly > until I started this email, is what formats will be supported for > hosting on PyPI. I am against hosting formats that don't support > static metadata, such as your "source distribution", as I don't see > how PyPI would be able to publish the metadata if it weren't static. > > And following on from that, we need to agree whether the key formats > should be required to have a static version. I'm OK with a VCS > checkout having a dynamically generated version, that's part of the > "all bets are off" contract over such things (if you don't generate a > version that reflects every change, you get to deal with the > consequences) but I don't think that's a reasonable thing to allow in > "published" formats. > > > (Other features of the original proposal include stuff like the lack of > > trivial metadata like "name" and "description", and the support for > > generating multiple wheels from one directory. I am explicitly calling > these > > "inessential".) > > Not sure what you mean by lack of name/description being > "inessential". The double negative confuses me. Do you mean you're OK > with requiring them? Fair enough. > > For multiple wheels, I'd tend to consider the opposite to be true - > it's not that the capability is non-essential, but rather that in > published formats (source wheel and later in the chain) it's essential > that one source generates one target. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Oct 5 15:39:10 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 15:39:10 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> Message-ID: <20151005153910.7f921992@fsol> On Mon, 5 Oct 2015 09:28:48 -0400 Donald Stufft wrote: > > > > In any case, sdists being apt for human consumption is an important > > feature IMHO. > > > > I don't think so. Remember in my footnote I mentioned that I wasn't using VCS > to mean literally "something checked into a VCS", but rather the more > traditional (outside of Python) "source release" concept. This could be a > simple tarball that just has the proper, for human consumption, files in it > or it could be a VCS checkout, or it could just be some files sitting on disk. But why use two different formats for "source release" and "sdists"? Currently sdists fit the assumptions for a source release, why introduce some complexity and have the users deal with separate concepts (with all the confusion that will inevitably ensue)? Regards Antoine. From p.f.moore at gmail.com Mon Oct 5 15:41:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 5 Oct 2015 14:41:25 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 5 October 2015 at 14:25, Donald Stufft wrote: > The main thing I wanted was something that wasn't a mouthful to say and that > had a dedicated extension so we don't end up where the filename looks > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). You find "Source Wheel" easier to say / type than "sdist"? (I type it more than say it, but typing sdist is easier, and when I say it I usually go for "ess-dist, or suh-dist"). And ".sdist" seems like a perfectly good suffix to me. And the bikeshed should be a light purple colour :-) Paul From wes.turner at gmail.com Mon Oct 5 15:45:27 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 5 Oct 2015 08:45:27 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 5, 2015 8:41 AM, "Paul Moore" wrote: > > On 5 October 2015 at 14:25, Donald Stufft wrote: > > The main thing I wanted was something that wasn't a mouthful to say and that > > had a dedicated extension so we don't end up where the filename looks > > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). > > You find "Source Wheel" easier to say / type than "sdist"? (I type it > more than say it, but typing sdist is easier, and when I say it I > usually go for "ess-dist, or suh-dist"). And ".sdist" seems like a > perfectly good suffix to me. How about something like .sdist.whl.zip ? There are already many tools with e.g. ZIP MIME extensions so that one can open the file and browse the archive with a file explorer without a platform specific file type association install step > > And the bikeshed should be a light purple colour :-) > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 5 15:51:05 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:51:05 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005153910.7f921992@fsol> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> Message-ID: <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Because the status quo, which is a single format, sucks. I don't think changing it so we invoke a Python function instead of a script is going to make it not suck. The fundamental reason it sucks (in my mind) is that we have a single format trying to do way too many things and we need to break that out into smaller chunks. You can see these needs are different I think by looking at how what Nathaniel wants differs from what me and Paul want. He wants something that will make the human side easier and will support different tools, we want something that pip can consume more reasonably. Trying to do too much with a singular format just means it sucks for all the uses cases instead of being great for one use case. I also don't think it will be confusing. They'll associate the VCS thing (a source release) as something focused on development for most everyone. Most people won't explicitly make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with manually producing binary wheels or producing source releases other than what's checked into vcs will be something that I suspect only advanced users will do. Sent from my iPhone > On Oct 5, 2015, at 9:39 AM, Antoine Pitrou wrote: > > On Mon, 5 Oct 2015 09:28:48 -0400 > Donald Stufft wrote: >>> >>> In any case, sdists being apt for human consumption is an important >>> feature IMHO. >> >> I don't think so. Remember in my footnote I mentioned that I wasn't using VCS >> to mean literally "something checked into a VCS", but rather the more >> traditional (outside of Python) "source release" concept. This could be a >> simple tarball that just has the proper, for human consumption, files in it >> or it could be a VCS checkout, or it could just be some files sitting on disk. > > But why use two different formats for "source release" and "sdists"? > Currently sdists fit the assumptions for a source release, why > introduce some complexity and have the users deal with separate > concepts (with all the confusion that will inevitably ensue)? > > Regards > > Antoine. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Mon Oct 5 15:54:09 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 09:54:09 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 9:45:29 AM, Wes Turner (wes.turner at gmail.com) wrote: > > > How about something like > .sdist.whl.zip ? > > > There are already many tools with e.g. ZIP MIME extensions > so that one can open the file and browse the archive with a file > explorer > without a platform specific file type association install step >? -1 on using a .zip extension. I don?t really want people opening these up to inspect them unless they know what they are doing. For most people, they should just think of them as blobs of stuff. +0 on .sdist.whl instead of .swhl or .src.whl. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Mon Oct 5 16:01:30 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 10:01:30 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 9:41:28 AM, Paul Moore (p.f.moore at gmail.com) wrote: > On 5 October 2015 at 14:25, Donald Stufft wrote: > > The main thing I wanted was something that wasn't a mouthful to say and that > > had a dedicated extension so we don't end up where the filename looks > > incredibly generic (requests-1.0.swhl vs requests-1.0.tar.gz). > > You find "Source Wheel" easier to say / type than "sdist"? (I type it > more than say it, but typing sdist is easier, and when I say it I > usually go for "ess-dist, or suh-dist"). And ".sdist" seems like a > perfectly good suffix to me. Yea I do, but I have a stutter and "d" (particularly trying to transition to it from the "s") and "st" are hard for me to get the sounds out anyways (ironic given my name contains both a D and a St!). I don't know how hard it is for someone else to say, probably slightly harder but not by much. This is why I also hate the "distribution" term, I have a hard time saying the whole word without getting stuck in the "stribut" part. Anyways, name is the least important part to me as long as we get a dedicated extension out of it :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Mon Oct 5 16:52:02 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 5 Oct 2015 16:52:02 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: <20151005165202.0709e341@fsol> On Mon, 5 Oct 2015 09:51:05 -0400 Donald Stufft wrote: > > You can see these needs are different I think by looking at how what Nathaniel wants differs from what me and Paul want. He wants something that will make the human side easier and will support different tools, we want something that pip can consume more reasonably. Trying to do too much with a singular format just means it sucks for all the uses cases instead of being great for one use case. That doesn't seem to follow. You can have regular human-compatible content and a few machine-compatible files besides (perhaps in a dedicated subdirectory). I don't see how that "sucks". > I also don't think it will be confusing. They'll associate the VCS thing (a source release) as something focused on development for most everyone. Most people won't explicitly make one and nobody will be uploading it to PyPI. Well, what is the point of standardizing the concept of source releases if nobody produces them? Regards Antoine. From donald at stufft.io Mon Oct 5 17:10:53 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 11:10:53 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005165202.0709e341@fsol> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> <20151005165202.0709e341@fsol> Message-ID: On October 5, 2015 at 10:52:22 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Mon, 5 Oct 2015 09:51:05 -0400 > Donald Stufft wrote: > > > > You can see these needs are different I think by looking at how what Nathaniel wants differs > from what me and Paul want. He wants something that will make the human side easier and > will support different tools, we want something that pip can consume more reasonably. > Trying to do too much with a singular format just means it sucks for all the uses cases instead > of being great for one use case. > > That doesn't seem to follow. You can have regular human-compatible > content and a few machine-compatible files besides (perhaps in a > dedicated subdirectory). I don't see how that "sucks?. Machine compatible needs things to be as tightly specified as possible, human compatible needs it to be as loosely specified as possible. In addition, the original proposal had things like "version" being dynamically computed because it might change based on something in the directory (like if the VCS gets another commit). For a machine format, we want all metadata to be static, for something designed for humans to interact with and much around with the code we want something that is more dynamic. If you have two different files one for machines and one for humans, what happens if they disagree? There should be a single source of truth in any one artifact for all of the information. We currently have the situation you describe, a PKG-INFO that is static metadata and a setup.py that is aimed towards humans... but nothing uses the PKG-INFO because the source of truth is the setup.py and the PKG-INFO might be lies. So we have a lot of effort and hacks going in to make something that is best designed for human interaction to be used for machines. > > > I also don't think it will be confusing. They'll associate the VCS thing (a source release) > as something focused on development for most everyone. Most people won't explicitly > make one and nobody will be uploading it to PyPI. > > Well, what is the point of standardizing the concept of source releases > if nobody produces them? > Well, a VCS commit will be a form of one, just not one that people explicitly distribute. I don't think it makes a sense to standard this idea of a "source release" as something different other than what we need to bootstrap a VCS based build as well. So we need to support it for VCSs anyways, and divorcing the standard from VCSs makes it more robust and allows the idea of a .tar.gz based "VCS" build anyways. I also didn't say nobody would produce them. I said most people would have no need for explicit source releases (particularly in a tarball form) and wouldn't produce them. If I thought *nobody* would produce them, then I wouldn't be willing to have them standardized. I think that they might be useful for some projects that have more advanced needs. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From fungi at yuggoth.org Mon Oct 5 17:14:39 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 5 Oct 2015 15:14:39 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <20151005153910.7f921992@fsol> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> Message-ID: <20151005151439.GM4731@yuggoth.org> On 2015-10-05 15:39:10 +0200 (+0200), Antoine Pitrou wrote: [...] > But why use two different formats for "source release" and "sdists"? > Currently sdists fit the assumptions for a source release, why > introduce some complexity and have the users deal with separate > concepts (with all the confusion that will inevitably ensue)? An sdist is an installable package which just happens to _look_ a lot like a source release tarball, but trying to pretend that downstream packagers will want to use it as such leads to a variety of pain points in the upstream/downstream relationship. For better or worse a lot of distros don't want generated files in upstream source code releases, since they need to confirm that they also ship the necessary tooling to regenerate any required files and that the generated files they ship match what their packaged tooling produces. While this similarity was probably seen as a "Good Thing [TM]" initially (hence standardizing on a .tar.gz extension), over time both the generated content of a typical sdist and the concern most distros have over shipping upstream-generated files has increased to the point where they really need to be viewed as separate and distinct release artifacts now. -- Jeremy Stanley From dholth at gmail.com Mon Oct 5 18:05:41 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 05 Oct 2015 16:05:41 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: sdists work well most of the time. Some problems 1 - hard to depend on something setup.py must import 2 - hard to use non-distutils build system due to the first problem and a poorly defined command line interface 3 - have to regenerate metadata to get dependencies Assume we are not ever going to be able to remove support for pip install git+https:// or the current sdist. So you need new-sdist for 3, but could add 1 & 2 within the current framework by using an elaborate generic setup.py or by adding a measurable amount of complexity to pip. Perhaps the underlying code for 1 & 2 is needed to implement features desired in new-sdist. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 5 18:08:07 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 5 Oct 2015 12:08:07 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On October 5, 2015 at 12:05:53 PM, Daniel Holth (dholth at gmail.com) wrote: > sdists work well most of the time. Some problems > > 1 - hard to depend on something setup.py must import > 2 - hard to use non-distutils build system due to the first problem and a > poorly defined command line interface > 3 - have to regenerate metadata to get dependencies > > Assume we are not ever going to be able to remove support for pip install > git+https:// or the current sdist. > > So you need new-sdist for 3, but could add 1 & 2 within the current > framework by using an elaborate generic setup.py or by adding a measurable > amount of complexity to pip. Perhaps the underlying code for 1 & 2 is > needed to implement features desired in new-sdist. > I'm not opposed to layering things ontop of the current sdist format in the interim to solve problems that currently exist until a better solution can be done. That's not what this PEP idea was though, it was creating a whole new sdist. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From guettliml at thomas-guettler.de Tue Oct 6 09:07:46 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Tue, 6 Oct 2015 09:07:46 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? Message-ID: <56137342.8090201@thomas-guettler.de> Hi, Where should I put tests when packaging python modules? I want a "cowpath", an "obvious way" Dear experts, please decide: inside the module like this answer: http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules XOR outside the module like this: https://github.com/pypa/sampleproject/tree/master/tests I think there is no need to hurry. Let's wait one week, and then check which one is preferred. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From glyph at twistedmatrix.com Tue Oct 6 09:20:36 2015 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 6 Oct 2015 00:20:36 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <56137342.8090201@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> Message-ID: <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> > On Oct 6, 2015, at 12:07 AM, Thomas G?ttler wrote: > > Hi, > > Where should I put tests when packaging python modules? > > I want a "cowpath", an "obvious way" > > Dear experts, please decide: > > inside the module like this answer: > > http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules > > XOR > > outside the module like this: > > https://github.com/pypa/sampleproject/tree/master/tests > > I think there is no need to hurry. Let's wait one week, > and then check which one is preferred. > > Regards, > Thomas G?ttler Inside the package. If you put your tests outside your package, then you can't install the tests for two packages simultaneously, because everyone's tests are just in the top-level package "tests". This tends to infest the whole package, since then tests import things from each other using 'from tests import ...'. This is recommended by the hitchhiker's guide, and seconded by . -glyph From ben+python at benfinney.id.au Tue Oct 6 09:38:33 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 06 Oct 2015 18:38:33 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> Message-ID: <85egh81m6e.fsf@benfinney.id.au> Thomas G?ttler writes: > Where should I put tests when packaging python modules? When packaging them? The same place they go when not packaging them :-) > I want a "cowpath", an "obvious way" For me, the obvious way is to have: > outside the module like this: > https://github.com/pypa/sampleproject/tree/master/tests and have ?setup.py? ensure the tests are distributed with the source package, but not installed. > I think there is no need to hurry. Let's wait one week, and then check > which one is preferred. More important than which is preferred, we should use the one which is best (regardless how popular it may be). So instead of just hearing votes, we should examine reasoned argument in favour of good options. -- \ ?Visitors are expected to complain at the office between the | `\ hours of 9 and 11 a.m. daily.? ?hotel, Athens | _o__) | Ben Finney From solipsis at pitrou.net Tue Oct 6 09:51:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 09:51:01 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> Message-ID: <20151006095101.031c8a56@fsol> On Tue, 6 Oct 2015 09:07:46 +0200 Thomas G?ttler wrote: > > Dear experts, please decide: > > inside the module like this answer: > > http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules They should be inside the module. That way, you can check an installed module is ok by running e.g. "python -m mypackage.tests". Any other choice makes testing installed modules more cumbersome. > outside the module like this: > > https://github.com/pypa/sampleproject/tree/master/tests There is no actual reason to do that except win a couple kilobytes if you are distributing your package on floppy disks for consumption on Z80-based machines with 64KB RAM. Even Python *itself* puts its test suite inside the standard library, not outside it (though some Linux distros may strip it away). Try "python -m test.regrtest" (again, this may fail if your distro decided to ship the test suite in a separate package). The PyP"A" should definitely fix its sample project to reflect good practices. Regards Antoine. From p.f.moore at gmail.com Tue Oct 6 10:17:22 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Oct 2015 09:17:22 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006095101.031c8a56@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On 6 October 2015 at 08:51, Antoine Pitrou wrote: > They should be inside the module. That way, you can check an installed > module is ok by running e.g. "python -m mypackage.tests". Any other > choice makes testing installed modules more cumbersome. One inconvenience with this is that if you use an external testing framework like nose or pytest, you either need to make your project depend on it, or you need to document that "python -m mypackage.tests" has additional dependencies that are not installed by default. With an external tests directory, the testing framework is just another "development requirement". It's only a minor point, conceded. > The PyP"A" should definitely fix its sample project to reflect good > practices. +1 on the sample project following recommended guidelines. When I originally wrote that project, the consensus (such as it was) was in favour of tests outside the installed project. But that may have simply reflected the group of people who responded at the time. It does however have the advantage that it's consistent with how other PyPA projects like pip and virtualenv structure their tests. The big problem is that I don't think there *is* a consensus on best practice. Both options have their supporters. It would also be very easy to take the view that the PyPA sample project should omit the test directory altogether, as it's a sample for the *packaging* guide, and development processes like testing are out of scope (that's why we don't include a documentation directory, or recommend sphinx, for example). Personally, I think if we go to that extreme, the sample project becomes useless as a "real world" template, but maybe people wanting a starter project template should be directed to projects such as cookiecutter instead. Paul From wes.turner at gmail.com Tue Oct 6 10:25:17 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Oct 2015 03:25:17 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Oct 6, 2015 3:17 AM, "Paul Moore" wrote: > > On 6 October 2015 at 08:51, Antoine Pitrou wrote: > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > One inconvenience with this is that if you use an external testing > framework like nose or pytest, you either need to make your project > depend on it, or you need to document that "python -m mypackage.tests" > has additional dependencies that are not installed by default. > > With an external tests directory, the testing framework is just > another "development requirement". > > It's only a minor point, conceded. otherwise try/except imports - [ ] @skipif in stdlib would be helpful > > > The PyP"A" should definitely fix its sample project to reflect good > > practices. > > +1 on the sample project following recommended guidelines. When I > originally wrote that project, the consensus (such as it was) was in > favour of tests outside the installed project. But that may have > simply reflected the group of people who responded at the time. It > does however have the advantage that it's consistent with how other > PyPA projects like pip and virtualenv structure their tests. many of these cookiecutter Python project templates also define e.g. tox.ini and .travis.yml: https://github.com/audreyr/cookiecutter-pypackage/blob/master/README.rst#similar-cookiecutter-templates > > The big problem is that I don't think there *is* a consensus on best > practice. Both options have their supporters. > > It would also be very easy to take the view that the PyPA sample > project should omit the test directory altogether, as it's a sample > for the *packaging* guide, and development processes like testing are > out of scope (that's why we don't include a documentation directory, > or recommend sphinx, for example). Personally, I think if we go to > that extreme, the sample project becomes useless as a "real world" > template, but maybe people wanting a starter project template should > be directed to projects such as cookiecutter instead. I think it wise to encourage TDD. https://github.com/audreyr/cookiecutter/blob/master/README.rst#python > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 10:44:15 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 10:44:15 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006104415.7fb554a2@fsol> On Tue, 6 Oct 2015 09:17:22 +0100 Paul Moore wrote: > On 6 October 2015 at 08:51, Antoine Pitrou wrote: > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > One inconvenience with this is that if you use an external testing > framework like nose or pytest, you either need to make your project > depend on it, or you need to document that "python -m mypackage.tests" > has additional dependencies that are not installed by default. > > With an external tests directory, the testing framework is just > another "development requirement". Doesn't / didn't setuptools have something called test_requires? > It would also be very easy to take the view that the PyPA sample > project should omit the test directory altogether, as it's a sample > for the *packaging* guide, and development processes like testing are > out of scope (that's why we don't include a documentation directory, > or recommend sphinx, for example). That sounds like the best course to me. Regards Antoine. From contact at ionelmc.ro Tue Oct 6 10:44:44 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 11:44:44 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On Tue, Oct 6, 2015 at 10:20 AM, Glyph Lefkowitz wrote: > If you put your tests outside your package, then you can't install the > tests for two packages simultaneously, because everyone's tests are just in > the top-level package "tests". This tends to infest the whole package, > since then tests import things from each other using 'from tests import > ...'. This is recommended by the hitchhiker's guide, and seconded by < > http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html > >. > ?I don't want to be harsh here but arguments would be way more interesting to discuss, as opposed to giving links to JPC's outdated packaging guide. He's an exceptional developer but many of the things inside that article are out of fashion (he disagrees about using console_scripts entrypoints?, doesn't understand what src is for and so on). I don't think anyone would ever intentionally put a "tests" package in site-packages, why would you mention that? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Tue Oct 6 10:30:00 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 11:30:00 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006095101.031c8a56@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > They should be inside the module. That way, you can check an installed > module is ok by running e.g. "python -m mypackage.tests". Any other > choice makes testing installed modules more cumbersome. > ?Does that really make sense? I haven't heard of any user actually running tests? that way. To be honest I haven't ever ran Python's own tests suite as part of a user installation. ?I've seen some projects that lump up lots of test data and crazy files in their packages tests and that created install issues? (Pelican is one example, pretty sure there are others). On the other hand, if the user really wants to run the tests he can just get the sources (that would naturally include everything)? Seems odd to suggest something is a best practice without giving any clue of how test dependencies would be managed. Just because CPython does it doesn't mean libraries should. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Tue Oct 6 11:18:03 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 06 Oct 2015 20:18:03 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <85612k1hkk.fsf@benfinney.id.au> Ionel Cristian M?rie? writes: > I've seen some projects that lump up lots of test data and crazy files > in their packages tests and that created install issues. > > On the other hand, if the user really wants to run the tests he can > just get the sources (that would naturally include everything)? Yes, this is a sensible approach: * The source package contains all the source files a developer would use to make further changes and test them. * The package for installation contains only those files useful run-time users, plus metadata (e.g. copyright information). I highly recommend it, and I would like the PyPA to also recommend the above approach. It does, though, require the acknowledgement of a separate *build* step in the development-and-release process. The build step is prior to the packaging step, and it generates run-time files from the collection of source files. That separation of a discrete build step is crucial for many good practices: generate documentation, integration testing, OS packaging, etc. -- \ ?The restriction of knowledge to an elite group destroys the | `\ spirit of society and leads to its intellectual | _o__) impoverishment.? ?Albert Einstein | Ben Finney From cournape at gmail.com Tue Oct 6 11:24:30 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 10:24:30 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Tue, Oct 6, 2015 at 9:30 AM, Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou > wrote: > >> They should be inside the module. That way, you can check an installed >> module is ok by running e.g. "python -m mypackage.tests". Any other >> choice makes testing installed modules more cumbersome. >> > > ?Does that really make sense? I haven't heard of any user actually running > tests? > that way. To be honest I haven't ever ran Python's own tests suite as part > of a user installation. > It makes a lot of sense for downstream packagers. Allowing testing installed packages is also the simplest way to enable testing on target machines which are different from the build machines. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Oct 6 11:51:44 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 6 Oct 2015 04:51:44 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Oct 6, 2015 4:24 AM, "David Cournapeau" wrote: > > > > On Tue, Oct 6, 2015 at 9:30 AM, Ionel Cristian M?rie? wrote: >> >> >> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: >>> >>> They should be inside the module. That way, you can check an installed >>> module is ok by running e.g. "python -m mypackage.tests". Any other >>> choice makes testing installed modules more cumbersome. >> >> >> ?Does that really make sense? I haven't heard of any user actually running tests? >> that way. To be honest I haven't ever ran Python's own tests suite as part >> of a user installation. > > > It makes a lot of sense for downstream packagers. Allowing testing installed packages is also the simplest way to enable testing on target machines which are different from the build machines. self-testable programs are really ideal (e.g POST power-on self test) relevant recent topical discussion of e.g CRC and an optional '-t' preemptive CLI parameter: https://github.com/audreyr/cookiecutter-pypackage/pull/52 > > David > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 11:54:25 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 05:54:25 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On October 6, 2015 at 3:21:04 AM, Glyph Lefkowitz (glyph at twistedmatrix.com) wrote: > > Inside the package. > > If you put your tests outside your package, then you can't install > the tests for two packages simultaneously, because everyone's > tests are just in the top-level package "tests". This tends to > infest the whole package, since then tests import things from > each other using 'from tests import ...'. This is recommended > by the hitchhiker's guide, and seconded by .? I dislike putting tests inside the package. The supposed benefit is that anyone can run the tests at anytime, but I don't find that actually true because it means (as someone else pointed out) that you either have to depend on all your test dependencies or that there is already an additional step to install them. If you're going to have to locate and install the test dependencies, then you might as well fetch the tarball with tests as well. Someone suggested setuptools test_requires, but that only functions when you have a setup.py available and you execute ``setup.py test``. It does not help you at all once the package is installed and the sdist is gone. I also don't think people actually run the tests when they are installed in any significant number, at least I've never once in my life done it or even had a desire to do it. Some projects have test suites which are significantly large too. The PyCA cryptography project for instance, has to ship it's vectors as an additional package to reduce the size of the final build product. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Tue Oct 6 12:18:30 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 11:18:30 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On Tue, Oct 6, 2015 at 10:54 AM, Donald Stufft wrote: > On October 6, 2015 at 3:21:04 AM, Glyph Lefkowitz (glyph at twistedmatrix.com) > wrote: > > > Inside the package. > > > > If you put your tests outside your package, then you can't install > > the tests for two packages simultaneously, because everyone's > > tests are just in the top-level package "tests". This tends to > > infest the whole package, since then tests import things from > > each other using 'from tests import ...'. This is recommended > > by the hitchhiker's guide, and seconded by . > > I dislike putting tests inside the package. > > The supposed benefit is that anyone can run the tests at anytime, but I > don't > find that actually true because it means (as someone else pointed out) > that you > either have to depend on all your test dependencies or that there is > already an > additional step to install them. If you're going to have to locate and > install > the test dependencies, then you might as well fetch the tarball with tests > as > well. > > Someone suggested setuptools test_requires, but that only functions when > you > have a setup.py available and you execute ``setup.py test``. It does not > help > you at all once the package is installed and the sdist is gone. > > I also don't think people actually run the tests when they are installed > in any > significant number, at least I've never once in my life done it or even > had a > desire to do it. > The significant number is not so relevant if you buy the argument that it is useful to downstream packagers: it may be a few users, but those are crucial. I also forgot to mention that the ability to test something without building is crucial when you want to distribute binaries. David > Some projects have test suites which are significantly large too. The PyCA > cryptography project for instance, has to ship it's vectors as an > additional > package to reduce the size of the final build product. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 12:21:49 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 06:21:49 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com) wrote: > > The significant number is not so relevant if you buy the argument > that it is useful to downstream packagers: it may be a few users, > but those are crucial. > > I also forgot to mention that the ability to test something without > building is crucial when you want to distribute binaries. Is it actually useful to them? None of the Linux downstreams I know of have ever mentioned a preference for it. As far as I know, the only preference they've ever expressed to me is that the tests are included in the sdist. FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, so it apparently doesn't find it useful either. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Tue Oct 6 12:33:07 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 11:33:07 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On Tue, Oct 6, 2015 at 11:21 AM, Donald Stufft wrote: > On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com) > wrote: > > > The significant number is not so relevant if you buy the argument > > that it is useful to downstream packagers: it may be a few users, > > but those are crucial. > > > > I also forgot to mention that the ability to test something without > > building is crucial when you want to distribute binaries. > > Is it actually useful to them? None of the Linux downstreams I know of have > ever mentioned a preference for it. As far as I know, the only preference > they've ever expressed to me is that the tests are included in the sdist. > It is at least useful to me, and I am packaging quite a few binaries. > > FreeBSD relies on ``python setup.py test`` as it's preferred test > invocation, > so it apparently doesn't find it useful either. > I would like to hear their rationale before guessing. It is hard for me to imagine they would not rather test the binaries rather than from sources. Something as simple as making sure you have not forgotten runtime dependencies becomes much easier this way. David > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 12:43:18 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 06:43:18 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: On October 6, 2015 at 6:33:10 AM, David Cournapeau (cournape at gmail.com) wrote: > On Tue, Oct 6, 2015 at 11:21 AM, Donald Stufft wrote: > > > On October 6, 2015 at 6:18:32 AM, David Cournapeau (cournape at gmail.com) > > wrote: > > > > The significant number is not so relevant if you buy the argument > > > that it is useful to downstream packagers: it may be a few users, > > > but those are crucial. > > > > > > I also forgot to mention that the ability to test something without > > > building is crucial when you want to distribute binaries. > > > > Is it actually useful to them? None of the Linux downstreams I know of have > > ever mentioned a preference for it. As far as I know, the only preference > > they've ever expressed to me is that the tests are included in the sdist. > > > > It is at least useful to me, and I am packaging quite a few binaries. > > > > > > FreeBSD relies on ``python setup.py test`` as it's preferred test > > invocation, > > so it apparently doesn't find it useful either. > > > > I would like to hear their rationale before guessing. It is hard for me to > imagine they would not rather test the binaries rather than from sources. > Something as simple as making sure you have not forgotten runtime > dependencies becomes much easier this way. > I'm able to test runtime dependencies just fine without needing to put my tests inside of the package. To the extent anyone can actually test runtime dependencies in a test framework without actually depending on your test tools at runtime [1]. I really don't think either way is "better" to be honest. I think any attempts that one way is better than the other relies on nebulous edge cases that don't really happen much in reality. You'll tell me that you've found it useful to run tests against an installed distribution without having to fetch the original tarball, I'll tell you that I've found it useful to run a newer version of the test suite against an older installed copy of the library. In practice, I think that you'll be able to manage finding the tarball and running those tests against your installed distribution while I'd be able to manage to copy just the tests I care about running out of the test suite and run them manually. IOW, I don't think it really matters and people should just do whatever they want. [1] By this I mean, you'll never detect if you're missing a runtime dependency ? ? on something that your test dependencies include as part of their ? ? dependencies, since installing your test tools to run your tests would then ? ? trigger this missing dependency to be installed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From contact at ionelmc.ro Tue Oct 6 12:47:51 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 13:47:51 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Tue, Oct 6, 2015 at 12:51 PM, Wes Turner wrote: > self-testable programs are really ideal (e.g POST power-on self test) > relevant recent topical discussion of e.g > CRC and an optional '-t' preemptive CLI parameter: > https://github.com/audreyr/cookiecutter-pypackage/pull/52 > ?I would be interesting ?to talk about what's worth including in a "self-test" feature. Most suites aren't suitable for including as whole. You don't want to include integration (functional) tests for sure :-) There's also the ever-unanswered question of how to deal with test dependencies. Some will think that it's ok to always install them, but then again, after seeing how `mock` depends on `pbr` - and how `pbr` just decides to alter your `setup.py sdist/bdist_*` output without being asked or invoked has made me very wary of this practice. How I have to make sure my python installs have certain versions of `pbr` or no `pbr` at all, every time I want to build a package :-( Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 13:00:18 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 13:00:18 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006130018.33a5760f@fsol> On Tue, 6 Oct 2015 11:30:00 +0300 Ionel Cristian M?rie? wrote: > On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > > > ?Does that really make sense? I haven't heard of any user actually running > tests? > that way. To be honest I haven't ever ran Python's own tests suite as part > of a user installation. There are several situations besides the "downstream packagers" use case mentioned somewhere else: * One of your users report a weird issue, you can ask them to run the test suite on their installation to check that nominal behaviour of the package is ok on their machine. If you don't ship the test suite, you have to ask them to do extra manual steps in order to do this verification, which can be cumbersome and delay proper response to the issue. * Your package requires non-Python data files for proper functioning, and you want to check the installation procedure puts them in the right place. The natural way to do that is to run the test suite on the installed package. Really, "ship the test suite" should be the norm and not shipping it should be the exception (if e.g. testing needs large data files). Regards Antoine. From donald at stufft.io Tue Oct 6 13:07:31 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 07:07:31 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006130018.33a5760f@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> Message-ID: On October 6, 2015 at 7:00:41 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Tue, 6 Oct 2015 11:30:00 +0300 > Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > > > > > They should be inside the module. That way, you can check an installed > > > module is ok by running e.g. "python -m mypackage.tests". Any other > > > choice makes testing installed modules more cumbersome. > > > > > > > ?Does that really make sense? I haven't heard of any user actually running > > tests > > that way. To be honest I haven't ever ran Python's own tests suite as part > > of a user installation. > > There are several situations besides the "downstream packagers" use case > mentioned somewhere else: > > * One of your users report a weird issue, you can ask them to run the > test suite on their installation to check that nominal behaviour of > the package is ok on their machine. If you don't ship the test suite, > you have to ask them to do extra manual steps in order to do this > verification, which can be cumbersome and delay proper response to > the issue. I've never, in my entire life, been asked to run someone's test suite to validate a bug nor have I asked someone to run a test suite to validate a bug. This is still ignoring the problems of test dependencies as well so you'll still need to ask them to install some number of dependencies, and I think it's fairly trivial to ask someone to download a tarball, untar it, and run two commands. > > * Your package requires non-Python data files for proper functioning, > and you want to check the installation procedure puts them in the > right place. The natural way to do that is to run the test suite on > the installed package. You're confusing "ships the test suite as part of the package" with "runs the test suite on the installed package". The two aren't really related, you can run the tests against an installed package trivially in either situation. > > Really, "ship the test suite" should be the norm and not shipping it > should be the exception (if e.g. testing needs large data files). > > Regards > > Antoine. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Tue Oct 6 14:13:25 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 14:13:25 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> Message-ID: <20151006141325.125514c7@fsol> On Tue, 6 Oct 2015 07:07:31 -0400 Donald Stufft wrote: > > I've never, in my entire life [...] Can I suggest your entire life is an anecdotal data point here? > This is still ignoring the problems of test dependencies Only if your tests have dependencies that runtime doesn't have. > as well so you'll > still need to ask them to install some number of dependencies, and I think it's > fairly trivial to ask someone to download a tarball, untar it, and run two > commands. Any number of things can be described as trivial depending on the skillset and patience of the user. When users report a bug, they are not expecting to be asked to download and "untar" stuff. Not every user is a programmer. > You're confusing "ships the test suite as part of the package" with "runs the > test suite on the installed package". The two aren't really related, you can > run the tests against an installed package trivially in either situation. It's not trivial, because if you aren't careful you'll be running them against the tarball / checkout instead (because of Python munging the PYTHONPATH behind your back, for example), and this can go unnoticed for a long time. By contrast, if you don't need a tarball / checkout to run them, you can guarantee they are run against the installed location. Regards Antoine. From contact at ionelmc.ro Tue Oct 6 14:34:38 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 15:34:38 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006141325.125514c7@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou wrote: > On Tue, 6 Oct 2015 07:07:31 -0400 > Donald Stufft wrote: > > > > I've never, in my entire life [...] > > Can I suggest your entire life is an anecdotal data point here? > ?Make that two anecdotal data points :-)? Any number of things can be described as trivial depending on the > skillset and patience of the user. When users report a bug, they are > not expecting to be asked to download and "untar" stuff. Not every > user is a programmer. > ? But seriously now, your arguments are also anecdotal. Lets not pretend we're objective here. That sort of attitude is disingenuous and will quickly devolve this discussion to mere ad hominems. ? > It's not trivial, because if you aren't careful you'll be running them > against the tarball / checkout instead (because of Python munging the > PYTHONPATH behind your back, for example), and this can go unnoticed for > a long time. > ?This is a flaw of the project layout really. If you don't isolate your sources from the import paths then you're probably testing the wrong way. In other words, you're probably not testing the installed code. Very few test runners change the current working directory by default [1], so it's better to just get a better project layout. pyca/cryptography is a good example.? ?[1] trial is the only one that I know of, and it's hardly popular for testing anything but projects that use Twisted?. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 14:51:10 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 14:51:10 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: <20151006145110.3e56ea96@fsol> On Tue, 6 Oct 2015 15:34:38 +0300 Ionel Cristian M?rie? wrote: > > Very few > test runners change the current working directory by default [1], so it's > better to just get a better project layout. pyca/cryptography > is a good example.? The "src" convention is actually terrible when working with Python code, since suddenly you can't experiment easily on a VCS checkout, you have to do extra steps and/or write helper scripts for it. The fact that few Python projects, including amongst the most popular projects, use that convention mean it's really not considered a good practice, nor convenient. Regards Antoine. From vivekmanchikanti at gmail.com Tue Oct 6 04:02:04 2015 From: vivekmanchikanti at gmail.com (vivek manchikanti) Date: Mon, 5 Oct 2015 21:02:04 -0500 Subject: [Distutils] Help regarding Python Theano on IDLE 3.5 Message-ID: Hi I'm a totally new to Python and I wanted to learn it. I have installed Python IDLE 3.5 successfully. Now I want to use Theano library in that IDE and I'm confused how to use that. Just have some questions like Do I need to install additional packages to use Theano on 3.5 ? Please kindly help. Thank you ------------------------------- Vivekanand Manchikanti -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 6 14:57:12 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 08:57:12 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006145110.3e56ea96@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On October 6, 2015 at 8:51:30 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Tue, 6 Oct 2015 15:34:38 +0300 > Ionel Cristian M?rie? wrote: > > > > Very few > > test runners change the current working directory by default [1], so it's > > better to just get a better project layout. pyca/cryptography > > is a good example. > > The "src" convention is actually terrible when working with Python > code, since suddenly you can't experiment easily on a VCS checkout, you > have to do extra steps and/or write helper scripts for it. Without doing it, you have very little assurances you?re actually testing against the installed project and not the project that's sitting in curdir. This is why pyca/cryptography does it, attempting to run the copy in . won't do anything but raise an exception since the .so won't be built. It doesn't really make experimenting in a VCS any harder, since all you need to do first is run ``pip install -e .`` and it will do a development install and add the src/ directory to sys.path. > > The fact that few Python projects, including amongst the most popular > projects, use that convention mean it's really not considered a good > practice, nor convenient. > > Of course, the same argument can be made for installing tests, since it's not very common. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tim at tim-smith.us Tue Oct 6 15:05:56 2015 From: tim at tim-smith.us (Tim Smith) Date: Tue, 6 Oct 2015 06:05:56 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? Message-ID: On Tue, Oct 6, 2015 at 5:35 AM, wrote: > On Tue, 6 Oct 2015 11:30:00 +0300 > Ionel Cristian M?rie? wrote: >> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: >> >> > They should be inside the module. That way, you can check an installed >> > module is ok by running e.g. "python -m mypackage.tests". Any other >> > choice makes testing installed modules more cumbersome. >> > >> >> ?Does that really make sense? I haven't heard of any user actually running >> tests that way. Numeric computing may be a special case because of its sensitivity to compiler/BLAS interactions but numpy and scipy support e.g. `numpy.test()` and Homebrew has received bug reports from people running them and getting unexpected results so I believe this is practiced and useful in some domains. Tim From contact at ionelmc.ro Tue Oct 6 15:00:12 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 16:00:12 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006145110.3e56ea96@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On Tue, Oct 6, 2015 at 3:51 PM, Antoine Pitrou wrote: > The "src" convention is actually terrible when working with Python > code, since suddenly you can't experiment easily on a VCS checkout, you > have to do extra steps and/or write helper scripts for it. > > The fact that few Python projects, including amongst the most popular > projects, use that convention mean it's really not considered a good > practice, nor convenient. > ?Convenience over correctness - I can understand that. Using virtualenvs (or tox) is not always something people want. People get attached to certain workflows, that's fine with me.? But lets not confuse that with what's right. Just because it's popular doesn't mean it anywhere close to correct. It means it works, once in a while people hit some pitfalls, suffer for it but continue the same way. The same people then complain about "terrible" packaging experience in Python. I think we should look at this more meticulously, not solely through the perspective of what's popular. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 6 15:07:29 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 15:07:29 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: <20151006150729.3e0c3f38@fsol> On Tue, 6 Oct 2015 08:57:12 -0400 Donald Stufft wrote: > > It doesn't really make experimenting in a VCS any harder, since all you need to > do first is run ``pip install -e .`` and it will do a development install and > add the src/ directory to sys.path. That means you're suddently polluting your Python install with a development package. So either you create a dedicated virtualenv (more command-line boilerplate, including each time you switch from a project another) or you risk messing with other package installs. Regards Antoine. From donald at stufft.io Tue Oct 6 15:33:03 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 09:33:03 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006150729.3e0c3f38@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> <20151006150729.3e0c3f38@fsol> Message-ID: On October 6, 2015 at 9:08:12 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > On Tue, 6 Oct 2015 08:57:12 -0400 > Donald Stufft wrote: > > > > It doesn't really make experimenting in a VCS any harder, since all you need to > > do first is run ``pip install -e .`` and it will do a development install and > > add the src/ directory to sys.path. > > That means you're suddently polluting your Python install with a > development package. So either you create a dedicated virtualenv (more > command-line boilerplate, including each time you switch from a project > another) or you risk messing with other package installs. > Unless your project has zero dependencies you?ll want to use a dedicated virtual environment anyways. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cournape at gmail.com Tue Oct 6 15:51:54 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Oct 2015 14:51:54 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On Tue, Oct 6, 2015 at 1:57 PM, Donald Stufft wrote: > > > On October 6, 2015 at 8:51:30 AM, Antoine Pitrou (solipsis at pitrou.net) > wrote: > > On Tue, 6 Oct 2015 15:34:38 +0300 > > Ionel Cristian M?rie? wrote: > > > > > > Very few > > > test runners change the current working directory by default [1], so > it's > > > better to just get a better project layout. pyca/cryptography > > > is a good example. > > > > The "src" convention is actually terrible when working with Python > > code, since suddenly you can't experiment easily on a VCS checkout, you > > have to do extra steps and/or write helper scripts for it. > > Without doing it, you have very little assurances you?re actually testing > against the installed project and not the project that's sitting in curdir. > This is why pyca/cryptography does it, attempting to run the copy in . > won't > do anything but raise an exception since the .so won't be built. > > It doesn't really make experimenting in a VCS any harder, since all you > need to > do first is run ``pip install -e .`` and it will do a development install > and > add the src/ directory to sys.path. > > > > > The fact that few Python projects, including amongst the most popular > > projects, use that convention mean it's really not considered a good > > practice, nor convenient. > > > > > > Of course, the same argument can be made for installing tests, since it's > not > very common. > So we can actually get some data here :) At Enthought, we support around 400 packages. More than half of them are python packages, and we can make the reasonable assumption than the vast majority of those packages are fairly popular. More precisely, if I install all our supported packages on Linux: - I count ~ 308 packages by making the assumption that one directory in site-packages is a package (wrong but decent approximation) - I count as `with tests` a package with at least one directory called test in say package With those assumptions, I count 46 % packages with tests installed. So it is not "not very common". Granted, we are biased toward scientific packages which to include tests in the package, but we also package popular web/etc... packages. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Oct 6 16:10:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Oct 2015 15:10:59 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On 6 October 2015 at 14:51, David Cournapeau wrote: >> Of course, the same argument can be made for installing tests, since it's >> not >> very common. > > > So we can actually get some data here :) > > At Enthought, we support around 400 packages. More than half of them are > python packages, and we can make the reasonable assumption than the vast > majority of those packages are fairly popular. > > More precisely, if I install all our supported packages on Linux: > - I count ~ 308 packages by making the assumption that one directory in > site-packages is a package (wrong but decent approximation) > - I count as `with tests` a package with at least one directory called test > in say package > > With those assumptions, I count 46 % packages with tests installed. So it is > not "not very common". Granted, we are biased toward scientific packages > which to include tests in the package, but we also package popular > web/etc... packages. Interesting. Prompted by this, I did a check on my Python 3.4 installation. 84 packages, of which 21 have a "tests" subdirectory containing __init__.py and 4 have a "test" subdirectory containing __init__.py. That's 25%, which is certainly higher than I would have guessed. Of course, the implication is that 75% (or 54% in David's case) use test suites *not* installed with the package :-) But I think it's fair to say that installing the tests with the package is a common enough model, and it clearly does have some benefits. To answer the OP's question, I stand by my original impression, which is that "it depends" :-) I don't want the sample project to *not* include tests, so as far as that is concerned, I'm going with "the status quo wins there" and suggesting there's no reason to change. Paul From njs at pobox.com Tue Oct 6 16:12:40 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 6 Oct 2015 07:12:40 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: Message-ID: On Oct 6, 2015 6:06 AM, "Tim Smith" wrote: > > On Tue, Oct 6, 2015 at 5:35 AM, wrote: > > On Tue, 6 Oct 2015 11:30:00 +0300 > > Ionel Cristian M?rie? wrote: > >> On Tue, Oct 6, 2015 at 10:51 AM, Antoine Pitrou wrote: > >> > >> > They should be inside the module. That way, you can check an installed > >> > module is ok by running e.g. "python -m mypackage.tests". Any other > >> > choice makes testing installed modules more cumbersome. > >> > > >> > >> ?Does that really make sense? I haven't heard of any user actually running > >> tests that way. > > Numeric computing may be a special case because of its sensitivity to > compiler/BLAS interactions but numpy and scipy support e.g. > `numpy.test()` and Homebrew has received bug reports from people > running them and getting unexpected results so I believe this is > practiced and useful in some domains. Yes, and numpy upstream gets bug / support requests on a totally regular basis (weekly-ish?) from end-users who have somehow managed to wedge up their install and either they noticed because they ran the installed test suite, or else they are reporting some bizarre results and we say "can you run python -m 'import numpy; numpy.test()'?". My impression is that there's a non-trivial subset of users who install numpy from binaries and then immediately run the test suite as a matter of course, and as a developer this is something that I'm happy to encourage (yay more test coverage). Adding another step where they have to download something extra would definitely discourage this. I can easily believe that there are other parts of the python world where this is uncommon, but if some people are saying "I've never seen this and it isn't useful" and others are saying "I see this all the time and it's very useful", then this isn't really anecdote versus contradictory anecdote. The conclusion is that we're talking about a thing that happens at least sometimes and is useful at least sometimes. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Oct 6 16:20:41 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 6 Oct 2015 07:20:41 -0700 Subject: [Distutils] Help regarding Python Theano on IDLE 3.5 In-Reply-To: References: Message-ID: On Oct 6, 2015 5:55 AM, "vivek manchikanti" wrote: > > Hi > > I'm a totally new to Python and I wanted to learn it. I have installed Python IDLE 3.5 successfully. > > Now I want to use Theano library in that IDE and I'm confused how to use that. > Just have some questions like > Do I need to install additional packages to use Theano on 3.5 ? > Please kindly help. Hi Vivek, This mailing list is for people talking about how to make it easier to build and install python packages; I doubt there's anyone here who knows much about theano. You'll probably have better luck finding someone who can help with your questions if you ask on stackoverflow or the theano-users group: http://www.deeplearning.net/software/theano/#how-to-seek-help Good luck! -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Oct 6 16:21:35 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 6 Oct 2015 14:21:35 +0000 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006142135.GS4731@yuggoth.org> On 2015-10-06 13:47:51 +0300 (+0300), Ionel Cristian M?rie? wrote: [...] > `pbr` just decides to alter your `setup.py sdist/bdist_*` output > without being asked or invoked [...] Assuming you're talking about https://launchpad.net/bugs/1483067 then it was fixed in 1.7.0. If you're still seeing it in later releases, a detailed bug report would be most appreciated since that's not at all the intent. -- Jeremy Stanley From solipsis at pitrou.net Tue Oct 6 16:24:21 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Oct 2015 16:24:21 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> <20151006150729.3e0c3f38@fsol> Message-ID: <20151006162421.5e6a847b@fsol> On Tue, 6 Oct 2015 09:33:03 -0400 Donald Stufft wrote: > On October 6, 2015 at 9:08:12 AM, Antoine Pitrou (solipsis at pitrou.net) wrote: > > On Tue, 6 Oct 2015 08:57:12 -0400 > > Donald Stufft wrote: > > > > > > It doesn't really make experimenting in a VCS any harder, since all you need to > > > do first is run ``pip install -e .`` and it will do a development install and > > > add the src/ directory to sys.path. > > > > That means you're suddently polluting your Python install with a > > development package. So either you create a dedicated virtualenv (more > > command-line boilerplate, including each time you switch from a project > > another) or you risk messing with other package installs. > > Unless your project has zero dependencies you?ll want to use a dedicated virtual environment anyways. Not necessarily, you can share that environment with other projects. Regards Antoine. From fungi at yuggoth.org Tue Oct 6 16:27:22 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 6 Oct 2015 14:27:22 +0000 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: <20151006142722.GT4731@yuggoth.org> On 2015-10-06 15:10:59 +0100 (+0100), Paul Moore wrote: [...] > That's 25%, which is certainly higher than I would have > guessed. Of course, the implication is that 75% (or 54% in David's > case) use test suites *not* installed with the package :-) [...] It seems rather optimistic to assume that 100% of projects have test suites. More accurately, 75% either have no tests _or_ keep them outside the package. ;) -- Jeremy Stanley From p.f.moore at gmail.com Tue Oct 6 16:39:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 6 Oct 2015 15:39:25 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006142722.GT4731@yuggoth.org> References: <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> <20151006142722.GT4731@yuggoth.org> Message-ID: On 6 October 2015 at 15:27, Jeremy Stanley wrote: > It seems rather optimistic to assume that 100% of projects have test > suites. More accurately, 75% either have no tests _or_ keep them > outside the package. ;) I'm having a happy day today - don't spoil it... :-) Paul From fred at fdrake.net Tue Oct 6 17:26:29 2015 From: fred at fdrake.net (Fred Drake) Date: Tue, 6 Oct 2015 11:26:29 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151006145110.3e56ea96@fsol> Message-ID: On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau wrote: > More precisely, if I install all our supported packages on Linux: > - I count ~ 308 packages by making the assumption that one directory in > site-packages is a package (wrong but decent approximation) > - I count as `with tests` a package with at least one directory called test > in say package > > With those assumptions, I count 46 % packages with tests installed. So it is > not "not very common". Granted, we are biased toward scientific packages > which to include tests in the package, but we also package popular > web/etc... packages. For what it's worth, the Zope community has long been including tests inside packages (regardless of whether that's right or wrong). How the dependencies issue has been addressed has varied over time and depending on the authors of each package; sometimes tests_require is used, other times there's a [test] extra (as setuptools is universally accepted in the Zope community), and other times the test dependencies are just made package dependencies. Since zc.buildout is widely used in the Zope community, getting the right dependencies for the right processes has never been a significant issue. (And yes, most Zope packages use a src/ directory. Not to isolate the tests and the implementation, but to isolate the code from other files that may be present in the package.) -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From erik.m.bray at gmail.com Tue Oct 6 17:33:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 6 Oct 2015 11:33:07 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 8:34 AM, Ionel Cristian M?rie? wrote: > On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou wrote: >> >> On Tue, 6 Oct 2015 07:07:31 -0400 >> Donald Stufft wrote: >> > >> > I've never, in my entire life [...] >> >> Can I suggest your entire life is an anecdotal data point here? > > > Make that two anecdotal data points :-) > >> Any number of things can be described as trivial depending on the >> skillset and patience of the user. When users report a bug, they are >> not expecting to be asked to download and "untar" stuff. Not every >> user is a programmer. > > But seriously now, your arguments are also anecdotal. Lets not pretend we're > objective here. That sort of attitude is disingenuous and will quickly > devolve this discussion to mere ad hominems. Okay, though, so maybe if there is nothing to offer here but anecdata then maybe we should stop acting like there's "one right way here". I have projects that install their test suite and test dependencies because it is frequently useful to ask users to run a self-test (and users themselves want to be able to do it for a variety of reasons). There are other projects where it doesn't make sense, and those don't have to install the tests (I still think in those cases that the tests should live in the package instead of outside it, but simply not installed). In any case, let's not get trapped into endless circular discussions about what is correct, period, and instead consider individual use cases--not dismissing individual projects' or peoples' experiences and needs--and discuss what the most appropriate action is for those uses cases. Python projects are not monolithic in their audiences (and that includes developer audiences and user audiences). Erik From contact at ionelmc.ro Tue Oct 6 18:04:54 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 6 Oct 2015 19:04:54 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 6:33 PM, Erik Bray wrote: > ?Okay, though, so maybe if there is nothing to offer here but anecdata > then maybe we should stop acting like there's "one right way here". I > have projects that install their test suite and test dependencies > because it is frequently useful to ask users to run a self-test (and > users themselves want to be able to do it for a variety of reasons). > ? > > There are other projects where it doesn't make sense, and those don't > have to install the tests (I still think in those cases that the tests > should live in the package instead of outside it, but simply not > installed). ?To be honest, I like the idea of providing the tests? in the installed package. Using the test suite as a debugging/diagnostic tool is obviously desirable. It's just that it's unpractical for general use. Another anecdote I know, but humour these two concerns: * How to deal with dependencies? ** Should we use extras? Installing test deps has the disadvantage of version conflicts. Unless we make some sort of "virtualenv-for-tests" tool? ** Should we vendor the deps? But isn't that maybe too hard to do (or plain wrong for other reasons)? ** Should we avoid having deps? That can be too limiting in some situations, unittest is very bare compared to pytest or nose. ?* What sort of tests should be included? Integration tests are special case, if they need external services, temporary storage or what not. Would we then need to have clear separation for different types of tests? I'm not saying tests inside package is bad at all. But to flaunt it around as a "best practice" requires at least some recommendations for the two concerns illustrated above, and probably better tooling. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Tue Oct 6 22:54:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 6 Oct 2015 16:54:07 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 12:04 PM, Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 6:33 PM, Erik Bray wrote: >> >> Okay, though, so maybe if there is nothing to offer here but anecdata >> then maybe we should stop acting like there's "one right way here". I >> have projects that install their test suite and test dependencies >> because it is frequently useful to ask users to run a self-test (and >> users themselves want to be able to do it for a variety of reasons). >> >> There are other projects where it doesn't make sense, and those don't >> have to install the tests (I still think in those cases that the tests >> should live in the package instead of outside it, but simply not >> installed). > > > To be honest, I like the idea of providing the tests in the installed > package. Using the test suite as a debugging/diagnostic tool is obviously > desirable. It's just that it's unpractical for general use. Another anecdote > I know, but humour these two concerns: Those are all fair questions, and I can't say I have great answers for all of them. But that's a fair point that if installable tests *are* recommended as a practice then they should be addressed. Before continuing on there are really three options being discussed here: 1) Tests totally outside the package. 2) Tests in the package but not installed (only in source dists / VCS checkout) 3) Tests in the package and some or all installed. Skimming back through the rest of the thread I don't see too much support for 1). The only argument against it is the need for specifying dependencies, etc., which really only impacts developers so long as the tests aren't *installed*, I think. But there's also the question of what kinds of tests we're talking about. I think unit tests should live in the .tests for a library. Other kinds of tests I don't have a strong opinion about. So in any case if we did recommend putting tests in a subpackage we could still do so without making a decree as to whether or not they should be installed. Maybe by default suggest that they not be installed, and only have them installable if the developer has a plan for addressing the questions below. I think each of these questions may have different answers depending on the needs of different projects. > * How to deal with dependencies? What dependencies exactly? Dependencies just needed for running the tests (as opposed to running the code being tested? Some of my code has optional dependencies, and the tests for that code are only run if those optional dependencies are satisfied, but that's not what we're talking about right?) > ** Should we use extras? Installing test deps has the disadvantage of > version conflicts. Unless we make some sort of "virtualenv-for-tests" tool? I'm rather fond of mktmpenv in virtualenv-wrapper, but I admit that may be out of scope here... Not to point to using easy_install as best practice, but something resembling the way setup_requires could actually work here--dependencies are downloaded into a temp dir and installed there as eggs; added to sys.path. No need for something as heavy-weight as a virtualenv. This still has some potential for VersionConflict errors if not handled carefully though. But I might try out something like this and see where I get with it, because I think it would be useful. > ** Should we vendor the deps? But isn't that maybe too hard to do (or plain > wrong for other reasons)? > ** Should we avoid having deps? That can be too limiting in some situations, > unittest is very bare compared to pytest or nose. py.test comes with a command to generate a bundled copy of py.test for vendoring: https://pytest.org/latest/goodpractises.html#deprecated-create-a-pytest-standalone-script (for reasons I've lost track of it's apparently deprecated now though?) I've used this with quite a bit of success to support installed tests. It allows me to ship a version of py.test that works with my tests along with the package, and there is no issue with version conflicts or anything. In principle the same trick could be used to bundle other dependencies. In practice it's a little more complicated because downstream packagers don't like this, but it's easy to remove the bundled py.test and use the one provided by the system instead (as in on Debian). In that case we do have to work with the downstream packagers to make sure all the tests are working for them. I don't think this will be be much of an issue for the average pure-Python package. > * What sort of tests should be included? Integration tests are special case, > if they need external services, temporary storage or what not. Would we then > need to have clear separation for different types of tests? I try to separate out tests like this and disable them by default. I think the answer to that question is going to vary a lot on a case by case basis, but it is a good question to pose to anyone considering having installed tests. > I'm not saying tests inside package is bad at all. But to flaunt it around > as a "best practice" requires at least some recommendations for the two > concerns illustrated above, and probably better tooling. Fair point, I agree! Best, Erik From barry at python.org Tue Oct 6 23:19:22 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:19:22 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: <20151006171922.0a53de51@anarchist.wooz.org> On Oct 06, 2015, at 05:54 AM, Donald Stufft wrote: >I dislike putting tests inside the package. I'm a big fan of putting the tests inside the package. I've often looked at a package's tests to get a better understanding of something that was unclear for the documentation, or didn't work the way I expected. Having the tests there in the installed package makes it easier to refer to. I also find that with tox+nose2 (my preferred one-two punch for testing), it makes it quite easy to find and run the full test suite or individual tests based on a regexp pattern. I also like the symmetry of having a docs/ directory for doctests and a tests/ directory for unittests. For complex packages with lots of subpackages, I have lots of tests/ directories, so that the unitests are near to the code they test. This way the source tree gets organized for free without additional complexity in an outside-the-package tests tree. YMMV, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Oct 6 23:21:27 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:21:27 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: <20151006172127.5dd9867a@anarchist.wooz.org> On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, >so it apparently doesn't find it useful either. Oh how I wish there was a standard way to *declare* how to run the test suite, such that all our automated tools (or the humans :) didn't have to guess. At least in Debuntu though, we can pretty much make any of the usual ways work during package build. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Oct 6 23:31:28 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:31:28 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> Message-ID: <20151006173128.7d712b62@anarchist.wooz.org> On Oct 06, 2015, at 11:33 AM, David Cournapeau wrote: >I would like to hear their rationale before guessing. It is hard for me to >imagine they would not rather test the binaries rather than from sources. >Something as simple as making sure you have not forgotten runtime >dependencies becomes much easier this way. In Debian we can do both. It's usually good practice to run the package's test suite during the build process, on the unbuilt source tree. That doesn't work for all packages though (tox comes to mind as a recent example), so we *also* have a way to run the test suite on a built-and-installed version of the Debian binary package. I usually try to at least do an import test in this phase, so for some like tox, I'll do a more extensive test. In Ubuntu, failing the built-and-installed test (a.k.a. autopkgtest or DEP-8) will prevent a package from getting promoted from the -proposed channel to the release channel, which usually shields end users from seeing broken packages. Debian doesn't have this gateway in place yet. There is a way to run those on a local test build, so that's pretty nice. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Tue Oct 6 23:40:35 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:40:35 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006174035.7557d23d@anarchist.wooz.org> On Oct 06, 2015, at 09:51 AM, Antoine Pitrou wrote: >The PyP"A" should definitely fix its sample project to reflect good >practices. Here's my own sample project. There are actually two git branches, one with an extension module and one with pure Python. https://gitlab.com/warsaw/stupid Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From donald at stufft.io Tue Oct 6 23:41:55 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 17:41:55 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006171922.0a53de51@anarchist.wooz.org> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> Message-ID: On October 6, 2015 at 5:20:03 PM, Barry Warsaw (barry at python.org) wrote: > On Oct 06, 2015, at 05:54 AM, Donald Stufft wrote: > > >I dislike putting tests inside the package. > > I'm a big fan of putting the tests inside the package. I've often looked at a > package's tests to get a better understanding of something that was unclear > for the documentation, or didn't work the way I expected. Having the tests > there in the installed package makes it easier to refer to. I also find that > with tox+nose2 (my preferred one-two punch for testing), it makes it quite > easy to find and run the full test suite or individual tests based on a regexp > pattern. I also like the symmetry of having a docs/ directory for doctests > and a tests/ directory for unittests. > > For complex packages with lots of subpackages, I have lots of tests/ > directories, so that the unitests are near to the code they test. This way > the source tree gets organized for free without additional complexity in an > outside-the-package tests tree. > I?m not sure I understand what you?re advocating here, it sounds like you want your tests at something like mycoolproject/tests so that they are importable from mycoolproject.tests? but then you talk about symmetry with docs/ and tests/ which sounds more like you have top level directories for tests/ docs/ and then mycoolproject/. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From barry at python.org Tue Oct 6 23:42:11 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:42:11 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006104415.7fb554a2@fsol> Message-ID: <20151006174211.78ae9bc2@anarchist.wooz.org> On Oct 06, 2015, at 10:44 AM, Antoine Pitrou wrote: >Doesn't / didn't setuptools have something called test_requires? Since I pretty much use tox everywhere these days, I've taken to defining test requirements in my tox.ini rather than my setup.py. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From contact at ionelmc.ro Tue Oct 6 23:47:31 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 00:47:31 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: On Tue, Oct 6, 2015 at 11:54 PM, Erik Bray wrote: > Skimming back through the rest of the thread I don't see too much > support for 1). The only argument against it is the need for > specifying dependencies, etc., which really only impacts developers so > long as the tests aren't *installed*, I think. But there's also the > question of what kinds of tests we're talking about. I think unit > tests should live in the .tests for a library. Other > kinds of tests I don't have a strong opinion about. > ?I think there's some confusion here. The pain point with "inside" tests is exactly the dependencies. And by dependencies I mean test dependencies, we're talking about tests here :-)? If you install two packages with "inside" tests, then how do you deal with the version conflict of test dependencies? That's the big elephant in the room everyone is ignoring :) If you have to bend over backwards (to install the test dependencies) in order to run the installed tests then what's the point of installing them at all? It's far more safer to ask the user to just checkout the sources and run the tests from there. Why completely mess up user's site-packages just because you want to have this weird `python -mfoobar.tests` workflow? I like the idea, I really do. But it's not for everyone. I strongly feel that only projects that don't have any test dependencies should install the tests, or provide the tests inside the package. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Oct 6 23:46:24 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 17:46:24 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <85612k1hkk.fsf@benfinney.id.au> Message-ID: <20151006174624.62f2bf9f@anarchist.wooz.org> On Oct 06, 2015, at 08:18 PM, Ben Finney wrote: >Yes, this is a sensible approach: > >* The source package contains all the source files a developer would use > to make further changes and test them. > >* The package for installation contains only those files useful run-time > users, plus metadata (e.g. copyright information). I generally don't both stripping out in-package tests when building binary packages for Debian. First because it's more work for (IMHO) dubious value and second because I think installed test files are actually useful sometimes. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From ben+python at benfinney.id.au Tue Oct 6 23:51:46 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 07 Oct 2015 08:51:46 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> Message-ID: <85mvvvzmvh.fsf@benfinney.id.au> Barry Warsaw writes: > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > >FreeBSD relies on ``python setup.py test`` as it's preferred test > >invocation > > Oh how I wish there was a standard way to *declare* how to run the > test suite, such that all our automated tools (or the humans :) didn't > have to guess. I think the above describes the standard way of declaring the test runner: The ?setup.py test? command. Now, I lament that more Python projects don't *conform to* that standard, but at least it exists. -- \ ?I have never made but one prayer to God, a very short one: ?O | `\ Lord, make my enemies ridiculous!? And God granted it.? | _o__) ?Voltaire | Ben Finney From ben+python at benfinney.id.au Tue Oct 6 23:54:37 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 07 Oct 2015 08:54:37 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> Message-ID: <85h9m3zmqq.fsf@benfinney.id.au> Barry Warsaw writes: > I'm a big fan of putting the tests inside the package. I've often > looked at a package's tests to get a better understanding of something > that was unclear for the documentation, or didn't work the way I > expected. Having the tests there in the installed package makes it > easier to refer to. That doesn't follow, or I'm not understanding you. If you have the tests in the source package, as is being advocated, you have the tests available for reference. So the *relative location* of the tests, within that source tree, doesn't argue for what you're saying. Are you arguing the separate point of whether tests should be *installed* with the package? -- \ ?We now have access to so much information that we can find | `\ support for any prejudice or opinion.? ?David Suzuki, 2008-06-27 | _o__) | Ben Finney From solipsis at pitrou.net Wed Oct 7 00:04:01 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2015 00:04:01 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: <20151007000401.48fb497c@fsol> On Wed, 7 Oct 2015 00:47:31 +0300 Ionel Cristian M?rie? wrote: > On Tue, Oct 6, 2015 at 11:54 PM, Erik Bray wrote: > > > Skimming back through the rest of the thread I don't see too much > > support for 1). The only argument against it is the need for > > specifying dependencies, etc., which really only impacts developers so > > long as the tests aren't *installed*, I think. But there's also the > > question of what kinds of tests we're talking about. I think unit > > tests should live in the .tests for a library. Other > > kinds of tests I don't have a strong opinion about. > > > > ?I think there's some confusion here. The pain point with "inside" tests is > exactly the dependencies. Is it your personal experience or some theoretical argument you're making? > If you install two packages with "inside" tests, then how do you deal with > the version conflict of test dependencies? Well, how do you deal with the version conflict of non-test dependencies? How are tests supposed to be a problem here, while they usually have so few dependencies of their own? > If you have to bend over backwards (to install the test dependencies) While some packages may have non-trivial test dependencies, usual practice is for test suites to require the exact same dependencies as the rest of the package, + perhaps a test runner. Since we're talking about good practice for the average package, it's not very useful to point out that 0.1% of PyPI packages may have excruciatingly annoying test dependencies. > Why completely mess up user's site-packages > just because you want to have this weird `python -mfoobar.tests` workflow? Did you have such an experience or are you making this up for the sake of the argument? And just because you are not used to a "workflow" doesn't make it "weird" in any case. Python itself uses such a workflow ("python -m test"). Regards Antoine. From barry at python.org Wed Oct 7 00:08:37 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 18:08:37 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: <20151006180837.5233d549@anarchist.wooz.org> On Oct 07, 2015, at 08:51 AM, Ben Finney wrote: >I think the above describes the standard way of declaring the test >runner: The ?setup.py test? command. > >Now, I lament that more Python projects don't *conform to* that >standard, but at least it exists. It's *a* standard but not *the* standard, just from pure observation. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From contact at ionelmc.ro Wed Oct 7 00:08:40 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 01:08:40 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <85mvvvzmvh.fsf@benfinney.id.au> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney wrote: > I think the above describes the standard way of declaring the test > runner: The ?setup.py test? command. > > Now, I lament that more Python projects don't *conform to* that > standard, but at least it exists. > ?There's a very simple answer to that: easy_install (that's what `setup.py test` will use to install deps). It has several design issue wrt how packages are installed and how dependencies are managed. Lets not use `setup.py test`. It's either bad or useless. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Oct 7 00:16:15 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 18:16:15 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> Message-ID: <20151006181615.53f69fdd@anarchist.wooz.org> On Oct 06, 2015, at 05:41 PM, Donald Stufft wrote: >I?m not sure I understand what you?re advocating here, it sounds like you >want your tests at something like mycoolproject/tests so that they are >importable from mycoolproject.tests? but then you talk about symmetry with >docs/ and tests/ which sounds more like you have top level directories for >tests/ docs/ and then mycoolproject/. Ah, sorry for being unclear. I put tests in myproj/tests so yes they are importable via myproj.tests. I also put __init__.py's in my docs/ directory so the directory, but not the individual .rst doc files are also importable via myproj.docs. I do this because of the handy nose2 plugin I cargo cult around that allows me to run individual tests or doctests with a command line switch. If I have subpackages, the pattern repeats, so e.g. myproj/subpkg1/tests -> import myproj.subpkg1.tests myproj/subpkg2/docs -> import myproj.subpkg2.docs Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From carl at oddbird.net Wed Oct 7 00:16:41 2015 From: carl at oddbird.net (Carl Meyer) Date: Tue, 6 Oct 2015 16:16:41 -0600 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151007000401.48fb497c@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> Message-ID: <56144849.3070107@oddbird.net> On 10/06/2015 04:04 PM, Antoine Pitrou wrote: [snip] > ...How are tests supposed to be a problem here, while they > usually have so few dependencies of their own? > >> If you have to bend over backwards (to install the test dependencies) > > While some packages may have non-trivial test dependencies, usual > practice is for test suites to require the exact same dependencies as > the rest of the package, + perhaps a test runner. > > Since we're talking about good practice for the average package, it's > not very useful to point out that 0.1% of PyPI packages may have > excruciatingly annoying test dependencies. I think this discussion could probably do with fewer unsupported assertions about what is "usual" -- it's clear that experiences in different parts of the community vary widely. Speaking personally and anecdotally, I maintain 15 or so projects on PyPI, and every single one of them has at least three or four test-only dependencies; not just a test runner, but also testing utilities of one kind or another (e.g. the mock backport for Python 2). So my personal percentage of "packages with more than one test-only dependency" is not 0.1%, it's 100%. I don't have any idea what the real percentage is on PyPI, and wouldn't hazard a guess. I'm fairly sure you don't know either. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 00:19:34 2015 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Oct 2015 18:19:34 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> <85h9m3zmqq.fsf@benfinney.id.au> Message-ID: <20151006181934.410b0395@anarchist.wooz.org> On Oct 07, 2015, at 08:54 AM, Ben Finney wrote: >Barry Warsaw writes: > >> I'm a big fan of putting the tests inside the package. I've often >> looked at a package's tests to get a better understanding of something >> that was unclear for the documentation, or didn't work the way I >> expected. Having the tests there in the installed package makes it >> easier to refer to. > >That doesn't follow, or I'm not understanding you. > >If you have the tests in the source package, as is being advocated, you >have the tests available for reference. So the *relative location* of >the tests, within that source tree, doesn't argue for what you're >saying. Since I'm not sure I follow that, I'll answer the question you asked: >Are you arguing the separate point of whether tests should be >*installed* with the package? Yes. We've had this conversation before in the context of Debian package sponsorship. I know and respect that you disagree. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From solipsis at pitrou.net Wed Oct 7 00:34:14 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2015 00:34:14 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> <56144849.3070107@oddbird.net> Message-ID: <20151007003414.36cfa9d7@fsol> On Tue, 6 Oct 2015 16:16:41 -0600 Carl Meyer wrote: > On 10/06/2015 04:04 PM, Antoine Pitrou wrote: > [snip] > > ...How are tests supposed to be a problem here, while they > > usually have so few dependencies of their own? > > > >> If you have to bend over backwards (to install the test dependencies) > > > > While some packages may have non-trivial test dependencies, usual > > practice is for test suites to require the exact same dependencies as > > the rest of the package, + perhaps a test runner. > > > > Since we're talking about good practice for the average package, it's > > not very useful to point out that 0.1% of PyPI packages may have > > excruciatingly annoying test dependencies. > > I think this discussion could probably do with fewer unsupported > assertions about what is "usual" -- it's clear that experiences in > different parts of the community vary widely. > > Speaking personally and anecdotally, I maintain 15 or so projects on > PyPI, and every single one of them has at least three or four test-only > dependencies; not just a test runner, but also testing utilities of one > kind or another (e.g. the mock backport for Python 2). They're still trivial dependencies, though. Usually small or medium-sized pure Python packages with a rather stable API (especially stdlib backports, of course). I don't see how they could cause the kind of mess the OP claimed they would. So I'd still like to know what "bend over backwards" is supposed to mean here. Regards Antoine. From contact at ionelmc.ro Wed Oct 7 00:44:34 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 01:44:34 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151007000401.48fb497c@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> Message-ID: On Wed, Oct 7, 2015 at 1:04 AM, Antoine Pitrou wrote: > > ?I think there's some confusion here. The pain point with "inside" tests > is > > exactly the dependencies. > > Is it your personal experience or some theoretical argument you're > making? > I've published about 27 packages that have tests on PyPI. Out of those only 5 could run purely on stdlib unittest.? That leaves me with 22 packages that need test tools like pytest/nose and assorted plugins. > > If you install two packages with "inside" tests, then how do you deal > with > > the version conflict of test dependencies? > > Well, how do you deal with the version conflict of non-test > dependencies? How are tests supposed to be a problem here, while they > usually have so few dependencies of their own? > ?It's double the trouble to find compatible releases. Current tooling don't resolve conflicts automatically.? Maybe it's something handled better in Conda for all I know but I don't use that. > > If you have to bend over backwards (to install the test dependencies) > > While some packages may have non-trivial test dependencies, usual > practice is for test suites to require the exact same dependencies as > the rest of the package, + perhaps a test runner. > Can't relate to that - definitely not `usual practice` from my perspective.? Since we're talking about good practice for the average package, it's > not very useful to point out that 0.1% of PyPI packages may have > excruciatingly annoying test dependencies. > ?Already feeling guilty ... I hope you're finally happy now :-) ? > > Why completely mess up user's site-packages > > just because you want to have this weird `python -mfoobar.tests` > workflow? > > Did you have such an experience or are you making this up for the sake > of the argument? > ?I got burnt by the pbr issue [1] once (cause mock has it as a run-time dependency). I don't normally use `mock` but in the circle of hell I live in someone else depended on it.? ?I don't look forward to that happening again, and I don't want to litter my site-packages with useless test stuff. I already have too much stuff in there. > And just because you are not used to a "workflow" doesn't make it > "weird" in any case. Python itself uses such a workflow ("python -m > test"). ?It's weird in the sense that you have to do all these gymnastics to get the test dependencies right? ?before running that. As I previously stated, I like the idea of `python -mfoobar.test` - just that dependencies and scope make it weird and impractival for *general* use. ?[1] https://bugs.launchpad.net/pbr/+bug/1483067? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Wed Oct 7 00:46:35 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 07 Oct 2015 09:46:35 +1100 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> <85h9m3zmqq.fsf@benfinney.id.au> <20151006181934.410b0395@anarchist.wooz.org> Message-ID: <857fmzzkc4.fsf@benfinney.id.au> Barry Warsaw writes: > On Oct 07, 2015, at 08:54 AM, Ben Finney wrote: > > >Barry Warsaw writes: > > > >> I'm a big fan of putting the tests inside the package. I've often > >> looked at a package's tests to get a better understanding of something > >> that was unclear for the documentation, or didn't work the way I > >> expected. Having the tests there in the installed package makes it > >> easier to refer to. > > > >That doesn't follow, or I'm not understanding you. > >[?] > > > >Are you arguing the separate point of whether tests should be > >*installed* with the package? > > Yes. We've had this conversation before in the context of Debian package > sponsorship. I know and respect that you disagree. Okay. That's quite an orthogonal dimension, though, to the *relative location* of tests within the source tree. So ?I'm a big fan of putting tests inside the [Python] package [directory]? can't be motivated by ?Having the tests there in the installed package?. The two aren't related, AFAICT. -- \ ?There was a point to this story, but it has temporarily | `\ escaped the chronicler's mind.? ?Douglas Adams | _o__) | Ben Finney From solipsis at pitrou.net Wed Oct 7 00:55:22 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 7 Oct 2015 00:55:22 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <20151007000401.48fb497c@fsol> Message-ID: <20151007005522.6e10aa0f@fsol> On Wed, 7 Oct 2015 01:44:34 +0300 Ionel Cristian M?rie? wrote: > > That leaves me with 22 packages that need test tools like pytest/nose and > assorted plugins. > [...] > > ?It's double the trouble to find compatible releases. Hmm, are you saying py.test / nose and assorted plugins break APIs often? I would be a bit surprised, but I don't use them nowadays. Modern unittest is quite capable. > > And just because you are not used to a "workflow" doesn't make it > > "weird" in any case. Python itself uses such a workflow ("python -m > > test"). > > ?It's weird in the sense that you have to do all these gymnastics to get > the test dependencies right? Well, "getting the dependencies right" must be done, whether you run the tests from their installed location or from the source tree. In both cases, you can probably get the package management system to automate package downloads and installs. Rehards Antoine. From robertc at robertcollins.net Wed Oct 7 01:23:13 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 7 Oct 2015 12:23:13 +1300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On 6 October 2015 at 23:47, Ionel Cristian M?rie? wrote: > > On Tue, Oct 6, 2015 at 12:51 PM, Wes Turner wrote: >> >> self-testable programs are really ideal (e.g POST power-on self test) >> relevant recent topical discussion of e.g >> CRC and an optional '-t' preemptive CLI parameter: >> https://github.com/audreyr/cookiecutter-pypackage/pull/52 > > > I would be interesting to talk about what's worth including in a "self-test" > feature. > > Most suites aren't suitable for including as whole. You don't want to > include integration (functional) tests for sure :-) > > There's also the ever-unanswered question of how to deal with test > dependencies. Some will think that it's ok to always install them, but then > again, after seeing how `mock` depends on `pbr` - and how `pbr` just decides > to alter your `setup.py sdist/bdist_*` output without being asked or invoked > has made me very wary of this practice. How I have to make sure my python > installs have certain versions of `pbr` or no `pbr` at all, every time I > want to build a package :-( Hangon, there's clearly a *huge* gap in understanding here. pbr does *not* modify *anyones* setup.py output unless its enabled. ***setuptools*** enables all its plugins unconditionally (because entrypoints, yay), and then pbr has to explicitly *opt-out* of doing anything. Which *we do*. There was a bug where a new thing added didn't have this opt-out, and its since been fixed. There's a separate related bug that the opt-out checking code isn't quite robust enough, and thats causing some havoc, but there's a patch up to fix it and as soon as we have a reliable test we'll be landing it and cutting a release. If there was a way to plug into setuptools where pbr code wasn't called on random packages, and we didn't have to manually opt-out, why that would be brilliant. [Note: I'm not saying setuptools is buggy - it chose this interface, and thats fine, but the consequence is bugs like this that we have to fix]. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 7 01:32:43 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 7 Oct 2015 12:32:43 +1300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <56137342.8090201@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> Message-ID: On 6 October 2015 at 20:07, Thomas G?ttler wrote: > Hi, > > Where should I put tests when packaging python modules? > > I want a "cowpath", an "obvious way" > > Dear experts, please decide: > > inside the module like this answer: > > http://stackoverflow.com/questions/5341006/where-should-i-put-tests-when-packaging-python-modules > > XOR > > outside the module like this: > > https://github.com/pypa/sampleproject/tree/master/tests > > I think there is no need to hurry. Let's wait one week, > and then check which one is preferred. My preference is to have the tests clearly namespaced to the project. That can be done one of three ways: ./projectpackage/[zero-or-more-dirs-deep]/tests ./projectname_tests/ ./tests with a runner configuration to add appropriate metadata The namespacing lets me gather up test output from multiple projects and not have it confused (nor have to artificially separate out what tests ran in what test process). This is mainly useful when taking a big-data mindset to your tests, so not relevant so much to 'heres my 1k loc project, enjoy world' scenarios. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From contact at ionelmc.ro Wed Oct 7 01:46:28 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 02:46:28 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins wrote: > > Hangon, there's clearly a *huge* gap in understanding here. > > pbr does *not* modify *anyones* setup.py output unless its enabled. ?Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr doing weird stuff to distributions by abusing said entrypoints. For reference: https://bugs.launchpad.net/pbr/+bug/1483067 ? There's nothing special about pbr here. It's not like it's the first package doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like pdbpp, but you don't put that in production. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 7 01:52:34 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 6 Oct 2015 19:52:34 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: On October 6, 2015 at 7:47:16 PM, Ionel Cristian M?rie? (contact at ionelmc.ro) wrote: > > ?Unless it's >=1.7.0. You can't blame setuptools having entrypoints > for pbr doing weird stuff to distributions by abusing said entry points.? Relax. pbr had a bug, it was acknowledged as a bug and subsequently fixed. The design of setuptools entry points made it an easy bug to have happen than if setuptools entry points were designed differently. Using the entry points given to it by setuptools is hardly abusing anything. ? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From guettliml at thomas-guettler.de Wed Oct 7 07:12:22 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 7 Oct 2015 07:12:22 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: <5614A9B6.7030506@thomas-guettler.de> Am 07.10.2015 um 00:08 schrieb Ionel Cristian M?rie?: > On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney > wrote: > >> I think the above describes the standard way of declaring the test >> runner: The ?setup.py test? command. >> >> Now, I lament that more Python projects don't *conform to* that >> standard, but at least it exists. >> > > ?There's a very simple answer to that: easy_install (that's what `setup.py > test` will use to install deps). It has several design issue wrt how > packages are installed and how dependencies are managed. > > Lets not use `setup.py test`. It's either bad or useless. Sorry, I am not an expert in the area "packaging". I don't understand what you say. I thought "easy_install" is a very old and deprecated method. Why not use `setup.py test`? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Wed Oct 7 07:38:58 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 7 Oct 2015 07:38:58 +0200 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> Message-ID: <5614AFF2.1090604@thomas-guettler.de> Am 06.10.2015 um 17:33 schrieb Erik Bray: > On Tue, Oct 6, 2015 at 8:34 AM, Ionel Cristian M?rie? > wrote: >> On Tue, Oct 6, 2015 at 3:13 PM, Antoine Pitrou wrote: >>> >>> On Tue, 6 Oct 2015 07:07:31 -0400 >>> Donald Stufft wrote: >>>> >>>> I've never, in my entire life [...] >>> >>> Can I suggest your entire life is an anecdotal data point here? >> >> >> Make that two anecdotal data points :-) >> >>> Any number of things can be described as trivial depending on the >>> skillset and patience of the user. When users report a bug, they are >>> not expecting to be asked to download and "untar" stuff. Not every >>> user is a programmer. >> >> But seriously now, your arguments are also anecdotal. Lets not pretend we're >> objective here. That sort of attitude is disingenuous and will quickly >> devolve this discussion to mere ad hominems. > > Okay, though, so maybe if there is nothing to offer here but anecdata > then maybe we should stop acting like there's "one right way here". I > have projects that install their test suite and test dependencies > because it is frequently useful to ask users to run a self-test (and > users themselves want to be able to do it for a variety of reasons). > > There are other projects where it doesn't make sense, and those don't > have to install the tests (I still think in those cases that the tests > should live in the package instead of outside it, but simply not > installed). > > In any case, let's not get trapped into endless circular discussions > about what is correct, period, and instead consider individual use > cases--not dismissing individual projects' or peoples' experiences and > needs--and discuss what the most appropriate action is for those uses > cases. Python projects are not monolithic in their audiences (and > that includes developer audiences and user audiences). Yes, there is not generic "one right way here". Yes, let's consider individual use cases. My use case are the docs for new comers: - https://github.com/pypa/sampleproject - https://packaging.python.org/en/latest/distributing/ That's why started the thread. New comers don't have the experience you have. New comers want to get their coding done. They need a simple advice: "If unsure, then do X" And for this question, I want a "one right way here". Up to now it looks like there is no consensus. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From marius at gedmin.as Wed Oct 7 07:35:08 2015 From: marius at gedmin.as (Marius Gedminas) Date: Wed, 7 Oct 2015 08:35:08 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006172127.5dd9867a@anarchist.wooz.org> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> Message-ID: <20151007053508.GA29553@platonas> On Tue, Oct 06, 2015 at 05:21:27PM -0400, Barry Warsaw wrote: > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, > >so it apparently doesn't find it useful either. > > Oh how I wish there was a standard way to *declare* how to run the test suite, > such that all our automated tools (or the humans :) didn't have to guess. I have hopes for 'tox.ini' becoming the standard way to test a Python project. Marius Gedminas -- "Actually, the Singularity seems rather useful in the entire work avoidance field. "I _could_ write up that report now but if I put it off, I may well become a weakly godlike entity, at which point not only will I be able to type faster but my comments will be more on-target." - James Nicoll -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 173 bytes Desc: Digital signature URL: From chris.barker at noaa.gov Wed Oct 7 08:11:47 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 6 Oct 2015 23:11:47 -0700 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: <5614AFF2.1090604@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <5614AFF2.1090604@thomas-guettler.de> Message-ID: On Tue, Oct 6, 2015 at 10:38 PM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > Yes, there is not generic "one right way here". > > Yes, let's consider individual use cases. > > My use case are the docs for new comers: > > - https://github.com/pypa/sampleproject > - https://packaging.python.org/en/latest/distributing/ > > That's why started the thread. > unfortunately, that isn't a use-case -- every newcomer has a different use case. I was happy to see this thread, because I thought maybe I"d learn what i should teach my students - new to python. But alas - there clearly really is no consensus. What i've told newbies in the past is somethig like: """ if you want your user to be able to install you package, and then run something like: import my_package my_package.test() then put your tests inside the package. If you are fine with only being able to run the tests from the source tree -- then put your tests outside the package. """ but really, newbies have no idea how to make this decsion. Maybe we could come up with a decision tree for this -- some guidance for knowing what to do, when? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 7 13:11:43 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Oct 2015 06:11:43 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151007053508.GA29553@platonas> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <20151007053508.GA29553@platonas> Message-ID: On Oct 7, 2015 12:44 AM, "Marius Gedminas" wrote: > > On Tue, Oct 06, 2015 at 05:21:27PM -0400, Barry Warsaw wrote: > > On Oct 06, 2015, at 06:21 AM, Donald Stufft wrote: > > > > >FreeBSD relies on ``python setup.py test`` as it's preferred test invocation, > > >so it apparently doesn't find it useful either. > > > > Oh how I wish there was a standard way to *declare* how to run the test suite, > > such that all our automated tools (or the humans :) didn't have to guess. make test > > I have hopes for 'tox.ini' becoming the standard way to test a Python > project. * https://tox.readthedocs.org/en/latest/config.html * https://github.com/docker/docker-registry/blob/master/tox.ini #flake8 * dox = docker + tox | PyPI: https://pypi.python.org/pypi/dox | Src: https://git.openstack.org/cgit/stackforge/dox/tree/dox.yml * docker-compose.yml | Docs: https://docs.docker.com/compose/ | Docs: https://github.com/docker/compose/blob/master/docs/yml.md * https://github.com/kelseyhightower/kubernetes-docker-files/blob/master/docker-compose.yml * https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/pods.md#alternatives-considered * https://github.com/docker/docker/issues/8781 ( pods ( containers ) ) * http://docs.buildbot.net/latest/tutorial/docker.html * http://docs.buildbot.net/current/tutorial/docker.html#building-and-running-buildbot tox.ini often is not sufficient: * [Makefile: make test/tox] * setup.py * tox.ini * docker/platform-ver/Dockerfile * [dox.yml] * [docker-compose.yml] * [CI config] * http://docs.buildbot.net/current/manual/configuration.html * jenkins-kubernetes, jenkins-mesos > > Marius Gedminas > -- > "Actually, the Singularity seems rather useful in the entire work avoidance > field. "I _could_ write up that report now but if I put it off, I may well > become a weakly godlike entity, at which point not only will I be able to > type faster but my comments will be more on-target." - James Nicoll > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 7 13:58:06 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 14:58:06 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <5614A9B6.7030506@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 8:12 AM, Thomas G?ttler wrote: > I thought "easy_install" is a very old and deprecated method. > ?Indeed it is. That why people put all sorts of custom "test" commands in their setup.py to work around the deficiencies of the "test? ?" command setuptools provides.? So we end up with lots of variations of "how to use pytest to run tests via `setup.py test`", "how to use pip to install deps, instead of what `setup.py test` normally does" and so on. If you're gonna implement a test runner in your setup.py you might as well use a supported and well maintained tool: tox. > Why not use `setup.py test`? > ?Because: 1. There's Tox?, which does exactly that, and more. It's maintained. It gets features. 2. The "test" command will install the "test_requires" dependencies as eggs. You will end up with multiple versions of the same eggs right in your source checkout. 3. The "test" command will install the "test_requires" dependencies with easy_install. That means wheels cannot be used. 4. Because the builtin "test" command is so bare people tend to implement a custom one. Everyone does something slightly different, and slightly buggy. 5. There's no established tooling that relies on `setup.py test`. There isn't even a test result protocol like TAP [1] for it. Why use something so limited and outdated if there's no practical advantage? [1] https://testanything.org/ Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 7 14:18:25 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 08:18:25 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On October 7, 2015 at 7:58:55 AM, Ionel Cristian M?rie? (contact at ionelmc.ro) wrote: > On Wed, Oct 7, 2015 at 8:12 AM, Thomas G?ttler > > wrote: > > Why not use `setup.py test`? > > > > Because: > > 1. There's Tox?, which does exactly that, and more. It's maintained. It > gets features. tox and setup.py test are not really equivalent.?There?s no way (to my knowledge) to test the item outside of a virtual environment. This is important for?downstreams who want to test that the package build and the tests successfully are executed in their environment, not within some virtual environment. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Wed Oct 7 14:20:36 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 7 Oct 2015 07:20:36 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Oct 7, 2015 6:58 AM, "Ionel Cristian M?rie?" wrote: > > > On Wed, Oct 7, 2015 at 8:12 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: >> >> I thought "easy_install" is a very old and deprecated method. > > > ?Indeed it is. That why people put all sorts of custom "test" commands in their setup.py to work around the deficiencies of the "test? > ?" command setuptools provides.? So we end up with lots of variations of "how to use pytest to run tests via `setup.py test`", "how to use pip to install deps, instead of what `setup.py test` normally does" and so on. > > If you're gonna implement a test runner in your setup.py you might as well use a supported and well maintained tool: tox. > >> >> Why not use `setup.py test`? > > > ?Because: > > 1. There's Tox?, which does exactly that, and more. It's maintained. It gets features. Tox rocks. * detox can run concurrent processes: https://pypi.python.org/pypi/detox/ * TIL timeit.default_timer measures **wall time** by default and not CPU time: concurrent test timings are likely different from linear tests run on an machine with load > 2. The "test" command will install the "test_requires" dependencies as eggs. You will end up with multiple versions of the same eggs right in your source checkout. * is there no way around this? * is this required / spec'd / fixable? > 3. The "test" command will install the "test_requires" dependencies with easy_install. That means wheels cannot be used. would it be possible to add this to wheel? as if, after package deployment, in-situ tests are no longer relevant. (I think it wise to encourage TDD here) > 4. Because the builtin "test" command is so bare people tend to implement a custom one. Everyone does something slightly different, and slightly buggy. * README.rst test invocation examples (all, subset, one) * Makefile (make test; [vim] :make) * python setup.py nosetests http://nose.readthedocs.org/en/latest/api/commands.html * python setup.py [test] https://pytest.org/latest/goodpractises.html#integrating-with-setuptools-python-setup-py-test > 5. There's no established tooling that relies on `setup.py test`. There isn't even a test result protocol like TAP [1] for it. Why use something so limited and outdated if there's no practical advantage? * xUnit XML: https://westurner.org/wiki/awesome-python-testing#xunit-xml ``` xUnit XML? https://en.wikipedia.org/wiki/XUnit https://nose.readthedocs.org/en/latest/plugins/xunit.html http://nosexunit.sourceforge.net/ https://pytest.org/latest/usage.html#creating-junitxml-format-files https://github.com/xmlrunner/unittest-xml-reporting https://github.com/zandev/shunit2/compare/master...jeremycarroll:master ``` * TAP protocol > > [1] https://testanything.org/ > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 7 15:42:05 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 16:42:05 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 3:18 PM, Donald Stufft wrote: > tox and setup.py test are not really equivalent. There?s no way (to my > knowledge) to test the item outside of a virtual environment. This is > important for downstreams who want to test that the package build and the > tests successfully are executed in their environment, not within some > virtual environment. ?Hmmmm ... you're right. But making Tox not use virtualenvs is not impossible - much alike? to how Detox is working, we could have a "Tax" (just made that up) that just skips making any virtualenv. It's a matter of making two subclasses and a console_scripts entrypoint (I think). I think it's a good name: ``use Tax instead of Tox if you wanna "tax" your global site-packages`` :-) We only need someone to do it. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 7 15:49:52 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 16:49:52 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 3:20 PM, Wes Turner wrote: > > 2. The "test" command will install the "test_requires" dependencies as > eggs. You will end up with multiple versions of the same eggs right in your > source checkout. > > * is there no way around this? > * is this required / spec'd / fixable? > ?It's not that bad now, recent setuptools put the eggs in a ".eggs" dir - so it's not as messy as before. ? > > 3. The "test" command will install the "test_requires" dependencies with > easy_install. That means wheels cannot be used. > > would it be possible to add this to wheel? > It's up to the maintainers of wheel/setuptools to figure this one out (or not) I think. Either way, you should search through the distutils-sig archives for clues/intentions, eg: https://mail.python.org/pipermail/distutils-sig/2014-December/thread.html#25482 ? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Oct 7 17:13:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 7 Oct 2015 11:13:07 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Tue, Oct 6, 2015 at 6:08 PM, Ionel Cristian M?rie? wrote: > > On Wed, Oct 7, 2015 at 12:51 AM, Ben Finney > wrote: >> >> I think the above describes the standard way of declaring the test >> runner: The ?setup.py test? command. >> >> Now, I lament that more Python projects don't *conform to* that >> standard, but at least it exists. > > > There's a very simple answer to that: easy_install (that's what `setup.py > test` will use to install deps). It has several design issue wrt how > packages are installed and how dependencies are managed. > > Lets not use `setup.py test`. It's either bad or useless. Says who? Many of the projects I'm involved in use `setup.py test` exclusively and for good reason--they all have C and/or Cython extension modules that need to be built for the tests to even run. Only setup.py knows about those extension modules and how to find and build them. Using `setup.py test` ensures that everything required to run the package (including runtime dependencies) is built and ready, and then the tests can start. Without it, we would have to tell developers to go through a build process first and then make sure they're running the tests on the built code. `setup.py test` makes it a no-brainer. For pure Python packages I think it's less important and can usually rely on "just run 'nose', or 'py.test'" (or "tox" but that's true regardless of how the tests are invoked outside of tox). Best, Erik From contact at ionelmc.ro Wed Oct 7 17:31:26 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 18:31:26 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 6:13 PM, Erik Bray wrote: > > Lets not use `setup.py test`. It's either bad or useless. > > Says who? Many of the projects I'm involved in use `setup.py test` > exclusively and for good reason--they all have C and/or Cython > extension modules that need to be built for the tests to even run. > Only setup.py knows about those extension modules and how to find and > build them. Using `setup.py test` ensures that everything required to > run the package (including runtime dependencies) is built and ready, ?Well ok, then it's not useless. :-) For pure Python packages I think it's less important and can usually > rely on "just run 'nose', or 'py.test'" (or "tox" but that's true > regardless of how the tests are invoked outside of tox). > That implies you would be testing code that you didn't install. That allows preventable mistakes, like publishing releases on PyPI that don't actually work, or do not even install at all (because you didn't test that). `setup.py test` doesn't really allow you to fully test that part, but Tox does. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Oct 7 17:32:07 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 7 Oct 2015 11:32:07 -0400 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) Message-ID: On Tue, Oct 6, 2015 at 7:46 PM, Ionel Cristian M?rie? wrote: > > On Wed, Oct 7, 2015 at 2:23 AM, Robert Collins > wrote: >> >> >> Hangon, there's clearly a *huge* gap in understanding here. >> >> pbr does *not* modify *anyones* setup.py output unless its enabled. > > > Unless it's >=1.7.0. You can't blame setuptools having entrypoints for pbr > doing weird stuff to distributions by abusing said entrypoints. > > For reference: https://bugs.launchpad.net/pbr/+bug/1483067 > > There's nothing special about pbr here. It's not like it's the first package > doing dangerous stuff (distribute, suprocess.run, pdbpp). I really like > pdbpp, but you don't put that in production. Starting a sub-thread since issues with pbr are orthogonal to the original disucssion. But one point I'd like to raise about this is that when I originally designed d2to1, on which a chunk of pbr is based, it was *explicitly* designed to never be installed in site-packages (with the exception of downstream packaging systems which can do what they want and are more controlled). This is exactly because I knew different packages might have dependencies on different versions of d2to1 as features are added, and that if some version is installed in site-packages it can lead to VersionConflict issues (this is in part exacerbated by a bug/misfeature in setuptools--I fixed that bug a while ago but the fix had to be rolled back due to a regression [1]). So TL;DR unless you know what you're doing, d2to1 should *never* be "installed"--it was only meant to be used with setup_requires, where the appropriate d2to1 used in building/installing a package is only temporarily enabled on sys.path via a temporary egg install. If some project is making it a *runtime* requirement that's a mistake. I don't know what features pbr has grown that might make someone want it to be a runtime dependency (the only entry-points I noticed were for adding egg-info writers but that should only be needed at build-time too...), but maybe something like that should be split off as a separate module or something... Best, Erik [1] https://bitbucket.org/tarek/distribute/pull-requests/20/fixes-and-adds-a-regression-test-for-323/diff From erik.m.bray at gmail.com Wed Oct 7 17:37:45 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 7 Oct 2015 11:37:45 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 11:31 AM, Ionel Cristian M?rie? wrote: > > On Wed, Oct 7, 2015 at 6:13 PM, Erik Bray wrote: >> >> > Lets not use `setup.py test`. It's either bad or useless. >> >> Says who? Many of the projects I'm involved in use `setup.py test` >> exclusively and for good reason--they all have C and/or Cython >> extension modules that need to be built for the tests to even run. >> Only setup.py knows about those extension modules and how to find and >> build them. Using `setup.py test` ensures that everything required to >> run the package (including runtime dependencies) is built and ready, > > > Well ok, then it's not useless. :-) > >> For pure Python packages I think it's less important and can usually >> rely on "just run 'nose', or 'py.test'" (or "tox" but that's true >> regardless of how the tests are invoked outside of tox). > > > That implies you would be testing code that you didn't install. That allows > preventable mistakes, like publishing releases on PyPI that don't actually > work, or do not even install at all (because you didn't test that). > `setup.py test` doesn't really allow you to fully test that part, but Tox > does. Which, incidentally, is a great reason for installable tests :) Running in the source tree is great for development. But when preparing a release it's great to be able to create an sdist, install that into a virtualenv, and run `package.test()` or `python -m package.tests` or whatever. Occasionally catches problems with the source dist if nothing else. Best, Erik From contact at ionelmc.ro Wed Oct 7 18:29:26 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 19:29:26 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> Message-ID: On Wed, Oct 7, 2015 at 6:37 PM, Erik Bray wrote: > Which, incidentally, is a great reason for installable tests :) > Not really. Doesn't matter where you have the tests. It matters where you have the code being tested. Tests being installed is a mere consequence of the location of tests. ? > Running in the source tree is great for development. But when > preparing a release it's great to be able to create an sdist, install > that into a virtualenv, and run `package.test()` or `python -m > package.tests` or whatever. Occasionally catches problems with the > source dist if nothing else. > ?? ?As I said, I like the idea. It's just that it's not feasible right now. Lets go over the issues again: * Tests too bulky (pyca/cryptography) * Tests can't be installed at all: ? https://github.com/getpelican/pelican/issues/1409? * Not clear how to install test dependencies. tests_require? extras? no deps? What about version conflicts and way too many deps being installed. Dependencies are like cars, they are very useful but too many of them create problems. * Real problems like standardized test output or run protocol are not solved at all. Little benefit of doing it like this if you can't build good CI tools around this. * Workflows are under-specified. User are not guided to make quality releases on PyPI. ?Maybe we should have a PEP that would specify/propose some concrete solutions to all those? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 7 19:27:28 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 10:27:28 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Mon, Oct 5, 2015 at 6:51 AM, Donald Stufft wrote: [...] > I also don't think it will be confusing. They'll associate the VCS thing (a source release) as something focused on development for most everyone. Most people won't explicitly make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with manually producing binary wheels or producing source releases other than what's checked into vcs will be something that I suspect only advanced users will do. Of course people will make source releases, and should be able to upload them to PyPI. The end goal is that *pip* will not use source releases, but PyPI is not just there for pip. If it was, it wouldn't even show package descriptions :-). There are projects on PyPI right now, today, that have no way to generate sdists and will never have any need for "source wheels" (because they don't use distutils and they build "none-any" wheels directly from their source). It should still be possible for them to upload source releases for all the other reasons that having source releases is useful: they form a permanent record of the whole project state (including potentially docs, tests, working notes, etc. that don't make it into the wheels), human users may well want to download those archives, Debian may prefer to use that as their orig.tar.gz, etc. etc. And on the other end of the complexity scale, there are projects like numpy where it's not clear to me whether they'll ever be able to support "source wheels", and even if they do they'll still need source releases to support user configuration at build time. -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Wed Oct 7 19:51:44 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 13:51:44 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 7, 2015 at 1:27:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Mon, Oct 5, 2015 at 6:51 AM, Donald Stufft wrote: > [...] > > I also don't think it will be confusing. They'll associate the VCS thing (a source release) > as something focused on development for most everyone. Most people won't explicitly > make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces > a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with > manually producing binary wheels or producing source releases other than what's checked > into vcs will be something that I suspect only advanced users will do. > > Of course people will make source releases, and should be able to > upload them to PyPI. The end goal is that *pip* will not use source > releases, but PyPI is not just there for pip. If it was, it wouldn't > even show package descriptions :-). > > There are projects on PyPI right now, today, that have no way to > generate sdists and will never have any need for "source wheels" > (because they don't use distutils and they build "none-any" wheels > directly from their source). It should still be possible for them to > upload source releases for all the other reasons that having source > releases is useful: they form a permanent record of the whole project > state (including potentially docs, tests, working notes, etc. that > don't make it into the wheels), human users may well want to download > those archives, Debian may prefer to use that as their orig.tar.gz, > etc. etc. > > And on the other end of the complexity scale, there are projects like > numpy where it's not clear to me whether they'll ever be able to > support "source wheels", and even if they do they'll still need source > releases to support user configuration at build time. We must have different ideas of what a source release vs source wheel would look like, because I'm having a hard time squaring what you've said here with what it looks like in my head. In my head, source releases (outside of the VCS use case) will be rare and only for very complex packages that are doing very complex things. Source wheels will be something that will be semi mandatory to being a well behaved citizen (for Debian and such to download) and binary wheels will be something that you'll want to have but aren't required. I don't see any reason why source wheels wouldn't include docs, tests, and other misc files. I picture building a binary wheel directly being something similar to using fpm to build binary .deb packages directly, totally possible but unadvised. Having talked to folks who deal with Debian/Fedora packages, they won't accept a binary wheel as the input source and (given how I explained it to them) they are excited about the concept of source wheels and moving away from dynamic metadata and towards static metadata. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Wed Oct 7 20:14:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 19:14:20 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 18:27, Nathaniel Smith wrote: > There are projects on PyPI right now, today, that have no way to > generate sdists and will never have any need for "source wheels" I think I'm as confused by what you're saying here as Donald is. Could you give a few examples of such projects? I'd like to go & take a look at them and try to understand what they are doing that is so incompatible with what Donald and I are thinking of as a"source wheel". Paul From donald at stufft.io Wed Oct 7 20:23:02 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 14:23:02 -0400 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On October 7, 2015 at 11:32:32 AM, Erik Bray (erik.m.bray at gmail.com) wrote: > > I don't know what features pbr has grown that might make someone > want > it to be a runtime dependency (the only entry-points I noticed > were > for adding egg-info writers but that should only be needed at > build-time too...), but maybe something like that should be > split off > as a separate module or something? It has runtime utilities for dealing with versions. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From oscar.j.benjamin at gmail.com Wed Oct 7 20:30:46 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 07 Oct 2015 18:30:46 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, 7 Oct 2015 18:51 Donald Stufft wrote: On October 7, 2015 at 1:27:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Mon, Oct 5, 2015 at 6:51 AM, Donald Stufft wrote: > [...] > > I also don't think it will be confusing. They'll associate the VCS thing (a source release) > as something focused on development for most everyone. Most people won't explicitly > make one and nobody will be uploading it to PyPI. The end goal in my mind is someone produces > a source wheel and uploads that to PyPI and PyPI takes it from there. Mucking around with > manually producing binary wheels or producing source releases other than what's checked > into vcs will be something that I suspect only advanced users will do. > > Of course people will make source releases, and should be able to > upload them to PyPI. The end goal is that *pip* will not use source > releases, but PyPI is not just there for pip. If it was, it wouldn't > even show package descriptions :-). > > There are projects on PyPI right now, today, that have no way to > generate sdists and will never have any need for "source wheels" > (because they don't use distutils and they build "none-any" wheels > directly from their source). It should still be possible for them to > upload source releases for all the other reasons that having source > releases is useful: they form a permanent record of the whole project > state (including potentially docs, tests, working notes, etc. that > don't make it into the wheels), human users may well want to download > those archives, Debian may prefer to use that as their orig.tar.gz, > etc. etc. > > And on the other end of the complexity scale, there are projects like > numpy where it's not clear to me whether they'll ever be able to > support "source wheels", and even if they do they'll still need source > releases to support user configuration at build time. We must have different ideas of what a source release vs source wheel would look like, because I'm having a hard time squaring what you've said here with what it looks like in my head. In my head, source releases (outside of the VCS use case) will be rare and only for very complex packages that are doing very complex things. Source wheels will be something that will be semi mandatory to being a well behaved citizen (for Debian and such to download) and binary wheels will be something that you'll want to have but aren't required. I don't see any reason why source wheels wouldn't include docs, tests, and other misc files. I picture building a binary wheel directly being something similar to using fpm to build binary .deb packages directly, totally possible but unadvised. Having talked to folks who deal with Debian/Fedora packages, they won't accept a binary wheel as the input source and (given how I explained it to them) they are excited about the concept of source wheels and moving away from dynamic metadata and towards static metadata. Your idea of an sdist as something that has fully static build/runtime dependency metadata and a one to one correspondence with binary wheels is not a usable format when releasing the code for e.g. numpy 1.10. It's fine to say that pip/PyPI should work with the source in some other distribution format and numpy could produce that but it means that the standard tarball release needs to be supported some how separately. Numpy should be able to use PyPI in order to host the tarball even if pip ignores the file. If numpy released only source wheels then there would be more than one source wheel for each release corresponding to e.g. the different ways that numpy is linked. There still needs to be a way to release a single file representing the code for the release as a whole. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 7 20:42:00 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 14:42:00 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 7, 2015 at 2:31:03 PM, Oscar Benjamin (oscar.j.benjamin at gmail.com) wrote: > > > Your idea of an sdist as something that has fully static build/runtime > dependency metadata and a one to one correspondence with binary > wheels is not a usable format when releasing the code for e.g. > numpy 1.10. It's fine to say that pip/PyPI should work with the > source in some other distribution format and numpy could produce > that but it means that the standard tarball release needs to be > supported some how separately. Numpy should be able to use PyPI > in order to host the tarball even if pip ignores the file. > > > If numpy released only source wheels then there would be more > than one source wheel for each release corresponding to e.g. > the different ways that numpy is linked. There still needs to > be a way to release a single file representing the code for the > release as a whole. > Can you expand on this please? I've never used numpy for anything serious and I'm trying to figure out why and what parts of what I'm thinking of wouldn't work for it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From barry at python.org Wed Oct 7 21:04:26 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 15:04:26 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: <20151007150426.436ae7fd@anarchist.wooz.org> On Oct 07, 2015, at 08:18 AM, Donald Stufft wrote: >tox and setup.py test are not really equivalent.?There?s no way (to my >knowledge) to test the item outside of a virtual environment. This is >important for?downstreams who want to test that the package build and the >tests successfully are executed in their environment, not within some virtual >environment. I usually do not use tox to test a package when building it for Debian. It's pretty easy to extract the actual command used to run the test suit from the tox.ini and that's what I put in the debian/rules file. It can make things build a little more reliably, but also eliminates a build dependency on tox. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 21:13:14 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 15:13:14 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <20151007053508.GA29553@platonas> Message-ID: <20151007151314.35bf4c3e@anarchist.wooz.org> On Oct 07, 2015, at 08:35 AM, Marius Gedminas wrote: >I have hopes for 'tox.ini' becoming the standard way to test a Python >project. As do I, modulo outliers of course. I'd like to see 90% of PyPI packages have a tox.ini. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From barry at python.org Wed Oct 7 21:14:58 2015 From: barry at python.org (Barry Warsaw) Date: Wed, 7 Oct 2015 15:14:58 -0400 Subject: [Distutils] Where should I put tests when packaging python modules? References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006171922.0a53de51@anarchist.wooz.org> <85h9m3zmqq.fsf@benfinney.id.au> <20151006181934.410b0395@anarchist.wooz.org> <857fmzzkc4.fsf@benfinney.id.au> Message-ID: <20151007151458.61cf001b@anarchist.wooz.org> On Oct 07, 2015, at 09:46 AM, Ben Finney wrote: >So ?I'm a big fan of putting tests inside the [Python] package >[directory]? can't be motivated by ?Having the tests there in the >installed package?. The two aren't related, AFAICT. It makes it easier for sure. When the tests are inside the package, nothing special has to be done; you just install the package and the tests subdirectories come along for the ride. If the tests are outside the package then you first have to figure out where they're going to go when they're installed, and then do something special to get them there. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Wed Oct 7 21:36:12 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 07 Oct 2015 19:36:12 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, 7 Oct 2015 19:42 Donald Stufft wrote: On October 7, 2015 at 2:31:03 PM, Oscar Benjamin (oscar.j.benjamin at gmail.com) wrote: > > > Your idea of an sdist as something that has fully static build/runtime > dependency metadata and a one to one correspondence with binary > wheels is not a usable format when releasing the code for e.g. > numpy 1.10. It's fine to say that pip/PyPI should work with the > source in some other distribution format and numpy could produce > that but it means that the standard tarball release needs to be > supported some how separately. Numpy should be able to use PyPI > in order to host the tarball even if pip ignores the file. > > > If numpy released only source wheels then there would be more > than one source wheel for each release corresponding to e.g. > the different ways that numpy is linked. There still needs to > be a way to release a single file representing the code for the > release as a whole. > Can you expand on this please? I've never used numpy for anything serious and I'm trying to figure out why and what parts of what I'm thinking of wouldn't work for it. Currently I can take the code from the numpy release and compile it in different incompatible ways. For example I could make a wheel that bundles a BLAS library. Or I could make a wheel that expects to use a system BLAS library that should be installed separately somehow or I could build a wheel against pyopenblas and make a wheel that depends on pyopenblas. Or I could link a BLAS library statically into numpy. A numpy release supports being compiled and linked in many different ways and will continue to do so regardless of any decisions made by PYPA. What that means is that there is not a one to one correspondence between a numpy release and a binary wheel. If there must be a one to one correspondence between a source wheel and a binary wheel then it follows that there cannot be a one to one correspondence between the source release and a source wheel. Of course numpy could say that they will only upload one particular source wheel and binary wheel to PyPI but people need to be able to use the source release in many different ways. So only releasing a source wheel that maps one to one to a particular way of compiling numpy is not an acceptable way for numpy to release its code. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 7 22:28:42 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 21:28:42 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 20:36, Oscar Benjamin wrote: > Currently I can take the code from the numpy release and compile it in > different incompatible ways. For example I could make a wheel that bundles a > BLAS library. Or I could make a wheel that expects to use a system BLAS > library that should be installed separately somehow or I could build a wheel > against pyopenblas and make a wheel that depends on pyopenblas. Or I could > link a BLAS library statically into numpy. > > A numpy release supports being compiled and linked in many different ways > and will continue to do so regardless of any decisions made by PYPA. What > that means is that there is not a one to one correspondence between a numpy > release and a binary wheel. If there must be a one to one correspondence > between a source wheel and a binary wheel then it follows that there cannot > be a one to one correspondence between the source release and a source > wheel. > > Of course numpy could say that they will only upload one particular source > wheel and binary wheel to PyPI but people need to be able to use the source > release in many different ways. So only releasing a source wheel that maps > one to one to a particular way of compiling numpy is not an acceptable way > for numpy to release its code. The disconnect here seems to be that I view all of those wheels as being numpy 1.9.X wheels (or whatever). They differ in terms of compatibility details, but they are all wheels for the same project/version. So there's no problem with them all being built from the same source wheel. I also have no problem with it being possible to configure the build differently from a single source wheel, to generate all those wheels. The configuration isn't metadata, it's "just" settings for the build. Of course, there *is* an unsolved issue here, which is how we manage compatibility for wheels at the level needed for numpy. But I thought the discussion on that was ongoing? I'm concerned that this proposal is actually about bypassing that discussion, and instead trying to treat incompatibly linked wheels as "different" in terms of project metadata, which I think is the wrong way of handling things. I note that Christoph Gohlke's numpy builds are tagged with a "+mkl" local version modifier - that's presumably intended to mark the fact that they are built with an incompatible runtime - but that's a misuse of local versions (and I've found it causes niggling issues with how pip recognises upgrades, etc). So, in summary: Your points above don't seem to me to in any way preclude having a single numpy source wheel, and a number of (mutually incompatible, but the same in terms of project and version) binary wheels. Paul From contact at ionelmc.ro Wed Oct 7 22:34:19 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 7 Oct 2015 23:34:19 +0300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <909A7AAE-C2FA-463D-9EA8-F68809C079CE@twistedmatrix.com> <20151006172127.5dd9867a@anarchist.wooz.org> <85mvvvzmvh.fsf@benfinney.id.au> <5614A9B6.7030506@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 4:42 PM, Ionel Cristian M?rie? wrote: > On Wed, Oct 7, 2015 at 3:18 PM, Donald Stufft wrote: > >> tox and setup.py test are not really equivalent. There?s no way (to my >> knowledge) to test the item outside of a virtual environment. This is >> important for downstreams who want to test that the package build and the >> tests successfully are executed in their environment, not within some >> virtual environment. > > > ?Hmmmm ... you're right. But making Tox not use virtualenvs is not > impossible - much alike? to how Detox is working, we could have a "Tax" > (just made that up) that just skips making any virtualenv. It's a matter of > making two subclasses and a console_scripts entrypoint (I think). I think > it's a good name: ``use Tax instead of Tox if you wanna "tax" your global > site-packages`` :-) > ?Just for kicks, I verified this, it's not hard at all: https://pypi.python.org/pypi/tax Barry may want to look at it, in case he has too many tox.ini files to copy-paste from :-) Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 7 23:28:50 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 14:28:50 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, Oct 7, 2015 at 1:28 PM, Paul Moore wrote: > The disconnect here seems to be that I view all of those wheels as > being numpy 1.9.X wheels (or whatever). They differ in terms of > compatibility details, but they are all wheels for the same > project/version. So there's no problem with them all being built from > the same source wheel. I also have no problem with it being possible > to configure the build differently from a single source wheel, to > generate all those wheels. The configuration isn't metadata, it's > "just" settings for the build. But the different builds for the different configurations end up with different metadata. If I'm understanding right, the whole point of "source wheels" is that they have all the static metadata that pip needs in order to make decisions, and this has to match the resulting wheels -- right? The way I'm imagining it is that there are multiple levels of metadata staticness: package name, author, description, ... static in: VCS checkouts, source releases, source wheels, wheels package version static in: source releases, source wheels, wheels package dependencies static in: source wheels, wheels environment tag static in: wheels > Of course, there *is* an unsolved issue here, which is how we manage > compatibility for wheels at the level needed for numpy. But I thought > the discussion on that was ongoing? I'm concerned that this proposal > is actually about bypassing that discussion, and instead trying to > treat incompatibly linked wheels as "different" in terms of project > metadata, which I think is the wrong way of handling things. I note > that Christoph Gohlke's numpy builds are tagged with a "+mkl" local > version modifier - that's presumably intended to mark the fact that > they are built with an incompatible runtime - but that's a misuse of > local versions (and I've found it causes niggling issues with how pip > recognises upgrades, etc). Yeah, that's not a good long term solution -- it needs to be moved into the metadata (probably by creating an MKL wheel and then making the numpy wheel depend on it). That's exactly the problem :-) > So, in summary: Your points above don't seem to me to in any way > preclude having a single numpy source wheel, and a number of (mutually > incompatible, but the same in terms of project and version) binary > wheels. Maybe I have misunderstood: does it actually help pip at all to have static access to name and version, but not to anything else? I've been assuming not, but I don't think anyone's pointed to any examples yet of the problems that pip is encountering due to the lack of static metadata -- would this actually be enough to solve them? -n -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Wed Oct 7 23:41:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 22:41:56 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 22:28, Nathaniel Smith wrote: > Maybe I have misunderstood: does it actually help pip at all to have > static access to name and version, but not to anything else? I've been > assuming not, but I don't think anyone's pointed to any examples yet > of the problems that pip is encountering due to the lack of static > metadata -- would this actually be enough to solve them? The principle I am working on is that *all* metadata in a source wheel should be statically available - that's not just for pip, but for all other consumers, including distro packagers. What's not set in stone is precisely what (subsets of) metadata are appropriate for source wheels as opposed to (binary) wheels. So I'd counter your question with the converse - what metadata specifically are you unwilling to include statically in source wheels? My feeling is that there isn't anything you'd be unwilling to include that I'd consider as "source wheel metadata". Possibly the nearest we'd have to an issue is over allowing the build process to *add* dependencies to a binary wheel (e.g. a some builds depend on currently-hypothetical MKL wheel, which provides needed DLLs). I don't in principle object to that, but I'd like to see a fleshed out proposal on how wheels containing just DLLs (as opposed to Python packages) would work in practice - until we have a mechanism for building/distributing such wheels, I think it's premature to worry about specifying dependencies. But whatever comes out of this, the Metadata 2.0 spec should ultimately be updated to note which metadata is mandated in source wheels, and which in binary wheels only. Paul From njs at pobox.com Wed Oct 7 23:53:46 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 14:53:46 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, Oct 7, 2015 at 11:14 AM, Paul Moore wrote: > On 7 October 2015 at 18:27, Nathaniel Smith wrote: >> There are projects on PyPI right now, today, that have no way to >> generate sdists and will never have any need for "source wheels" > > I think I'm as confused by what you're saying here as Donald is. Could > you give a few examples of such projects? I'd like to go & take a look > at them and try to understand what they are doing that is so > incompatible with what Donald and I are thinking of as a"source > wheel". An example would be flit itself: https://github.com/takluyver/flit https://pypi.python.org/pypi/flit It's not that you couldn't support a "source wheel" here, it's just that forcing them to go checkout -> source wheel -> wheel would be adding pointless hassle while accomplishing nothing useful. pip would never actually touch the source wheel, and for the remaining use cases for source distribution, a classic "source release" that's basically a tarball of a VCS checkout + static version number would be more familiar and useful. -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Wed Oct 7 23:58:45 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 7 Oct 2015 17:58:45 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > Yeah, that's not a good long term solution -- it needs to be moved > into the metadata (probably by creating an MKL wheel and then > making > the numpy wheel depend on it). That's exactly the problem :-) >? Are you available on IRC or for a video call or something? I feel like there's something foundational from both sides that we're each missing here and it'd be easier to just hash it out in real time rather than lobbying random emails coming from places of confusion (at least on my side). I'm not sure if Paul (or anyone else!) would want to jump in on it too, though I feel like probably if it's me and you then the two "sides" will probably be reasonably well represented so if more folks don't want to join that's probably OK too, particularly since we wouldn't be making any actual decisions there :D ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Thu Oct 8 00:25:36 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 7 Oct 2015 23:25:36 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 22:53, Nathaniel Smith wrote: >> I think I'm as confused by what you're saying here as Donald is. Could >> you give a few examples of such projects? I'd like to go & take a look >> at them and try to understand what they are doing that is so >> incompatible with what Donald and I are thinking of as a"source >> wheel". > > An example would be flit itself: > https://github.com/takluyver/flit > https://pypi.python.org/pypi/flit > > It's not that you couldn't support a "source wheel" here, it's just > that forcing them to go checkout -> source wheel -> wheel would be > adding pointless hassle while accomplishing nothing useful. pip would > never actually touch the source wheel, and for the remaining use cases > for source distribution, a classic "source release" that's basically a > tarball of a VCS checkout + static version number would be more > familiar and useful. I'm not sure I follow. If you have a binary wheel of flit, "pip install flit" won't need a source wheel, certainly (that's just as true for flit as for something complex like numpy). But distro packages would still want a source wheel to build their packages. If you mean that flit itself wouldn't use a source wheel, then while that may well be true, it's hardly relevant - whether flit chooses to use a source wheel is its own choice. But I'd hope flit *could* use a source wheel, as otherwise I couldn't use it to build wheels for other projects which want to use it and distribute source wheels. Should any such exist - this is pretty hypothetical at this point, and so not likely to be very productive. I am inclined to think that we're basically in agreement, we're just confused over terminology, and/or worrying about hypothetical cases. Would it help if I said that the *only* distinction between "source release" and source wheel that I care about is that in a source wheel the metadata must be static? We can discuss what metadata precisely, and we can thrash out other differences that might make more use of the fact that conceptually a "source release" is for humans to work with whereas a source wheel is for tools to consume, but those are details. I'm not clear if you think I have some more complicated picture than that, but really I don't [1]. Paul [1] I'd like a source wheel to have a defined format, but even that's not a killer. A zipfile with 2 directories "metadata" containing machine readable static metadata, and "source" with the complete contents of a source release, would do me. Of course when you build, if the metadata the build produces doesn't match the static data, that's a bug in the project packaging and we'd want to guard against it (it's the main reason the static data in the current sdist format is useless, that we can't rely on it :-() We can thrash this sort of stuff out, though. From robertc at robertcollins.net Thu Oct 8 00:39:11 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 8 Oct 2015 11:39:11 +1300 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On 8 October 2015 at 04:32, Erik Bray wrote: > Starting a sub-thread since issues with pbr are orthogonal to the > original disucssion. > > But one point I'd like to raise about this is that when I originally > designed d2to1, on which a chunk of pbr is based, it was *explicitly* > designed to never be installed in site-packages (with the exception of > downstream packaging systems which can do what they want and are more > controlled). This is exactly because I knew different packages might > have dependencies on different versions of d2to1 as features are > added, and that if some version is installed in site-packages it can > lead to VersionConflict issues (this is in part exacerbated by a > bug/misfeature in setuptools--I fixed that bug a while ago but the fix > had to be rolled back due to a regression [1]). So yes - that principle makes a lot of sense. There are two factors for pbr. a) as Donald mentions, there's a runtime API too - but we could indeed split that out into a separate package, if it would help things. b) more importantly, in OpenStack infra we can't use easy-install - its inconsistency with pip and lack of updated handling for wheels, HTTPS, separate configuration - consistently cause headaches every single time it happens. So many years ago we put in place pre-installation of all known build-time dependencies - we just install them statically, because we find the effort required to make them be compatible is less than the headaches from easy-install. As such pbr has a hard API requirement: thou shalt be backwards compatible. Clearly b) can hit VersionConflicts if pbr (and any other build dependencies like setuptools itself) are out of date, but that is easily handled for automated environments (pip install -U pip setuptools wheel pbr && echo YAY), and it errors cleanly enough for hand-use by folk that its a decent enough tradeoff in our experience. > I don't know what features pbr has grown that might make someone want > it to be a runtime dependency (the only entry-points I noticed were > for adding egg-info writers but that should only be needed at > build-time too...), but maybe something like that should be split off > as a separate module or something... Its all about easy-install. This is why I put a proof of concept static-setup-requires thing together for pip (and its on our teams roadmap to deliver a production version of it via patches to all the pypa projects, we're just not at that point yet - the resolver is first, and we have to finish rolling out constraints within OpenStack before that gets large timeslices). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Thu Oct 8 06:47:38 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 7 Oct 2015 21:47:38 -0700 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On Wed, Oct 7, 2015 at 3:39 PM, Robert Collins wrote: [...] > Its all about easy-install. This is why I put a proof of concept > static-setup-requires thing together for pip (and its on our teams > roadmap to deliver a production version of it via patches to all the > pypa projects, we're just not at that point yet - the resolver is > first, and we have to finish rolling out constraints within OpenStack > before that gets large timeslices). I remembered Nick saying something about this at PyCon, but I couldn't find anything when I looked -- could you point me to the PoC? -n -- Nathaniel J. Smith -- http://vorpus.org From oscar.j.benjamin at gmail.com Thu Oct 8 12:18:22 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 11:18:22 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 7 October 2015 at 22:41, Paul Moore wrote: > On 7 October 2015 at 22:28, Nathaniel Smith wrote: >> Maybe I have misunderstood: does it actually help pip at all to have >> static access to name and version, but not to anything else? I've been >> assuming not, but I don't think anyone's pointed to any examples yet >> of the problems that pip is encountering due to the lack of static >> metadata -- would this actually be enough to solve them? > > The principle I am working on is that *all* metadata in a source wheel > should be statically available - that's not just for pip, but for all > other consumers, including distro packagers. What's not set in stone > is precisely what (subsets of) metadata are appropriate for source > wheels as opposed to (binary) wheels. A concrete example would be whether or not the numpy source wheel depends on pyopenblas. Depending on how numpy is built the binary wheel may or may not depend on pyopenblas. It doesn't make any sense to say that the numpy source release depends on pyopenblas so what should be the dependencies of the source wheel? One possibility which I think is what Nathaniel is getting at is that there is a source release and then that could be used to generate different possible source wheels each of which would correspond to a particular configuration of numpy. Each source wheel would correspond to one binary wheel and have all static metadata but there still needs to be a separate source release that is used to generate the different source wheels. The step that turns a source wheel into a binary wheel would be analogous to the ./configure step in a typical makefile project. ./configure is used to specify the options corresponding to all the different ways of compiling and installing the project. After running ./configure the command "make" is unparametrised and performs the actual compilation: this step is analogous to converting a source wheel to a binary wheel. I think this satisfies all of the requirements for static metadata and one-to-one correspondence of source wheels and binary wheels. If numpy followed this then I imagine that there would be a single source wheel on PyPI corresponding to the one configuration that would be used consistently there. However numpy still needs to separately release the code in a form that is also usable in all of the many other contexts that it is already used. IOW they will need to continue to issue source releases in more or less the same form as today. It makes sense for PyPI to host the source release archives on the project page even if pip will simply ignore them. -- Oscar From p.f.moore at gmail.com Thu Oct 8 13:46:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Oct 2015 12:46:30 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 11:18, Oscar Benjamin wrote: > On 7 October 2015 at 22:41, Paul Moore wrote: >> On 7 October 2015 at 22:28, Nathaniel Smith wrote: >>> Maybe I have misunderstood: does it actually help pip at all to have >>> static access to name and version, but not to anything else? I've been >>> assuming not, but I don't think anyone's pointed to any examples yet >>> of the problems that pip is encountering due to the lack of static >>> metadata -- would this actually be enough to solve them? >> >> The principle I am working on is that *all* metadata in a source wheel >> should be statically available - that's not just for pip, but for all >> other consumers, including distro packagers. What's not set in stone >> is precisely what (subsets of) metadata are appropriate for source >> wheels as opposed to (binary) wheels. > > A concrete example would be whether or not the numpy source wheel > depends on pyopenblas. Depending on how numpy is built the binary > wheel may or may not depend on pyopenblas. It doesn't make any sense > to say that the numpy source release depends on pyopenblas so what > should be the dependencies of the source wheel? Well, I said this previously but I don't have any objections to the idea that binary wheels have additional dependencies - so the source wheel doesn't depend on pyopenblas but the binary does. But as I understand it, this is currently theoretical - there isn't yet any pyopenblas validate these speculations against? I say this not because I think the approach is invalid, but because I think there are probably a lot of untested questions that need answering. Let's expand the scenario a bit. The user (presumably) still just says "python -m pip install numpy". What happens then? 1. Assume there's a binary wheel that's compatible with the user's platform. 1a. If there are multiple compatible binary wheels, pip chooses the "most compatible" so we're safe to assume there's only one. [1] 2. Looking at the dependencies, say it depends on pyopenblas. So pip needs to install pyopenblas. 2a. If there's a compatible wheel for pyopenblas, pip installs that too. 2b. If there's no compatible pyopenblas wheel, pip falls back to a source wheel, builds it, and uses that. If the build fails, the whole numpy install fails. 3. If there's no compatible numpy binary wheel, pip gets the source wheel and builds it. There's no user interaction possible here [2], so the build uses whatever defaults the numpy build process identifies as "most appropriate" for the user's platform. This may be simply a lowest common denominator, or it may do some form of introspection of the user's system to get the best possible build. Either way, a wheel is generated that's known to work on the user's system, so there should be no additional dependencies injected at this point, and pip will use that wheel directly. The only constraint here is that a binary numpy wheel built with the default options on a given machine from a numpy source wheel cannot have extra dependencies that aren't known to be already satisfied by the user's system, because by the time pip generates a wheel from the source wheel, it's finished doing dependency resolution so any new dependencies won't get checked. I don't see it as a problem for any hypothetical new build system to conform to this constraint - by default a built wheel must work on the system it's built on. All it means is that to build binaries with additional dependencies must be done manually, supplying options describing your intent. [1] Dependencies are *not* considered as part of the compatibility matching, so it's correct that this step happens before the dependency checks. Maybe you're assuming that if there are two wheels, one depending on pyopenblas and one not, then if the user doesn't have pyopenblas installed the wheel that doesn't depend on it will be used? But that's not how pip works. [2] When pip runs installs, it does so non-interactively. Whatever command pip uses to build a wheel ("python setup.py bdist_wheel" at the moment) must run without user interaction and produce a wheel that is compatible with the user's environment. So unless I'm mistaken about what you're saying, I don't see any issue here. Unless you're saying that you're not willing to work under some of the constraints I describe above - but in that case, you need pip's compatibility matching, dependency resolution, or automated wheel build processes to change. That's fine but to move the discussion forwards, we'd then need to understand (and agree with) whatever changes you need in pip. At the moment, I'm not aware that anyone has asked for substantive changes to pip's behaviour in these areas as part of this proposal. > One possibility which I think is what Nathaniel is getting at is that > there is a source release and then that could be used to generate > different possible source wheels each of which would correspond to a > particular configuration of numpy. Each source wheel would correspond > to one binary wheel and have all static metadata but there still needs > to be a separate source release that is used to generate the different > source wheels. That's possible, but what would these multiple source wheels be called? They couldn't all be called "numpy" as how would the user say which one they wanted? Pip can't decide. They can't be called numpy and distinguished by versions, as then how would you decide whether "numpy with openblas" is "newer" or "older" than "numpy with MKL"? That's the issue with Christoph Gohlke's current means of versioning his MKL builds. So you're looking at multiple PyPI projects, one for each "flavour" of numpy. Or you're looking at changes to how PyPI and pip define a "project". Neither of those options sound particularly straightforward to me. > The step that turns a source wheel into a binary wheel would be > analogous to the ./configure step in a typical makefile project. > ./configure is used to specify the options corresponding to all the > different ways of compiling and installing the project. After running > ./configure the command "make" is unparametrised and performs the > actual compilation: this step is analogous to converting a source > wheel to a binary wheel. But the Python (PyPI/pip) model is different from the autoconf "typical makefile project" model. There's no configure step. If you're proposing that we add one, then that's a pretty major change in structure and would have some fairly wide-ranging impacts (on PyPI and pip, and also on 3rd party projects like bandersnatch and devpi). I don't think we're even close to understanding how we'd manage such a change. > I think this satisfies all of the requirements for static metadata and > one-to-one correspondence of source wheels and binary wheels. If numpy > followed this then I imagine that there would be a single source wheel > on PyPI corresponding to the one configuration that would be used > consistently there. However numpy still needs to separately release > the code in a form that is also usable in all of the many other > contexts that it is already used. IOW they will need to continue to > issue source releases in more or less the same form as today. It makes > sense for PyPI to host the source release archives on the project page > even if pip will simply ignore them. So you're talking about numpy only supporting one configuration via PyPI, and expecting any other configurations to be made available only via other channels? I guess you could do that, but I hope you won't. It feels to me like giving up before we've properly tried to understand the issues. Paul From contact at ionelmc.ro Thu Oct 8 14:05:14 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 8 Oct 2015 15:05:14 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Thu, Oct 8, 2015 at 1:18 PM, Oscar Benjamin wrote: > I think this satisfies all of the requirements for static metadata and > one-to-one correspondence of source wheels and binary wheels. If numpy > followed this then I imagine that there would be a single source wheel > on PyPI corresponding to the one configuration that would be used > consistently there. However numpy still needs to separately release > the code in a form that is also usable in all of the many other > contexts that it is already used. > ?Can't that configuration just be the build defaults?? There would be a single source but with some preset build configuration. People with different needs can just override those. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Oct 8 14:39:11 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 13:39:11 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 12:46, Paul Moore wrote: > On 8 October 2015 at 11:18, Oscar Benjamin wrote: >> >> A concrete example would be whether or not the numpy source wheel >> depends on pyopenblas. Depending on how numpy is built the binary >> wheel may or may not depend on pyopenblas. It doesn't make any sense >> to say that the numpy source release depends on pyopenblas so what >> should be the dependencies of the source wheel? > > Well, I said this previously but I don't have any objections to the > idea that binary wheels have additional dependencies - so the source > wheel doesn't depend on pyopenblas but the binary does. Okay, I guess I'm confused by what you mean when you say that a source wheel (or sdist) should have a "one-to-one" correspondence with a binary wheel. > But as I understand it, this is currently theoretical - there isn't > yet any pyopenblas validate these speculations against? I don't think pyopenblas is ready yet but it is being developed with specifically this scenario in mind. > So unless I'm mistaken about what you're saying, I don't see any issue > here. Unless you're saying that you're not willing to work under some > of the constraints I describe above As an aside : I'm not a contributor to numpy. I just use it a lot and teach people how to use it (which is where the packaging problems come in). > - but in that case, you need pip's > compatibility matching, dependency resolution, or automated wheel > build processes to change. That's fine but to move the discussion > forwards, we'd then need to understand (and agree with) whatever > changes you need in pip. At the moment, I'm not aware that anyone has > asked for substantive changes to pip's behaviour in these areas as > part of this proposal. I don't think anyone is suggesting significant changes to pip's dependency resolution. Compatibility matching does need improvement IMO. Also the automated build process does need to be changed - specifically we need build-requires so that third party build tools can work. I didn't think improving the build process was controversial... >> I think this satisfies all of the requirements for static metadata and >> one-to-one correspondence of source wheels and binary wheels. If numpy >> followed this then I imagine that there would be a single source wheel >> on PyPI corresponding to the one configuration that would be used >> consistently there. However numpy still needs to separately release >> the code in a form that is also usable in all of the many other >> contexts that it is already used. IOW they will need to continue to >> issue source releases in more or less the same form as today. It makes >> sense for PyPI to host the source release archives on the project page >> even if pip will simply ignore them. > > So you're talking about numpy only supporting one configuration via > PyPI, and expecting any other configurations to be made available only > via other channels? I guess you could do that, but I hope you won't. > It feels to me like giving up before we've properly tried to > understand the issues. Okay so again I'm not a numpy dev. Numpy already supports being used in lots of setups that are not via pypi. Apart from Cristoph's builds you have all kinds of people building on all kinds of OSes and linking with different BLAS libraries in different ways. Some people will compile numpy statically with CPython. If you follow the discussions about numpy development it's clear that the numpy devs don't know all of the ways that numpy is built and used. Clearly pip/PyPI cannot be used to statically link numpy with CPython or for all of the different (often non-redistributable) BLAS libraries so numpy will support some setups that are not possible through pip. That's fine, I don't see the problem with that. At the moment an sdist is the same thing is a source release. If you propose to change it so that projects should upload source wheels and then make source wheels something tightly defined (e.g. a zip file containing exactly two directories setup to build for one particular configuration) then there needs to be a separate way to simply release the code in traditional format as is done now. -- Oscar From oscar.j.benjamin at gmail.com Thu Oct 8 14:47:28 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 13:47:28 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 13:05, Ionel Cristian M?rie? wrote: > On Thu, Oct 8, 2015 at 1:18 PM, Oscar Benjamin > wrote: >> >> I think this satisfies all of the requirements for static metadata and >> one-to-one correspondence of source wheels and binary wheels. If numpy >> followed this then I imagine that there would be a single source wheel >> on PyPI corresponding to the one configuration that would be used >> consistently there. However numpy still needs to separately release >> the code in a form that is also usable in all of the many other >> contexts that it is already used. > > Can't that configuration just be the build defaults? There would be a single > source but with some preset build configuration. People with different needs > can just override those. Yeah, I guess so. Maybe I'm just not understanding what the "one-to-one" correspondence is supposed to mean. Earlier in the thread it was said to be important because of wheel caching etc. but if it's possible to configure different builds then it's not really one-to-one. -- Oscar From donald at stufft.io Thu Oct 8 15:01:59 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Oct 2015 09:01:59 -0400 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On October 8, 2015 at 8:48:16 AM, Oscar Benjamin (oscar.j.benjamin at gmail.com) wrote: > On 8 October 2015 at 13:05, Ionel Cristian M?rie? wrote: > > On Thu, Oct 8, 2015 at 1:18 PM, Oscar Benjamin > > wrote: > >> > >> I think this satisfies all of the requirements for static metadata and > >> one-to-one correspondence of source wheels and binary wheels. If numpy > >> followed this then I imagine that there would be a single source wheel > >> on PyPI corresponding to the one configuration that would be used > >> consistently there. However numpy still needs to separately release > >> the code in a form that is also usable in all of the many other > >> contexts that it is already used. > > > > Can't that configuration just be the build defaults? There would be a single > > source but with some preset build configuration. People with different needs > > can just override those. > > Yeah, I guess so. Maybe I'm just not understanding what the > "one-to-one" correspondence is supposed to mean. Earlier in the thread > it was said to be important because of wheel caching etc. but if it's > possible to configure different builds then it's not really > one-to-one. > One of the features in the original PEP was the ability to produce multiple different Wheels from the same source release much like how Debian does. e.g. numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions that would differ from the name/version of the original sdist thing that was being proposed. That won't work with our packaging toolchain, the idea that their a singular name/version for one sdist (and then for the wheels that produces) is pretty heavily baked into the entire toolchain. That?s what I meant by wheels and sdists being 1:1. As far as static metadata goes, I think that one of my earlier messages tried to get across the idea that if there is a good reason for something to be dynamic then we can possibly do that, but that the current PEP went too far and made (well kept) *everything* dynamic. My point was that we should assume static for all metadata and then make exceptions for the data that we can't assume that for, but each case should be properly documented with motivations? or *why* that can't be static. That will give everyone else the ability to see the use case, and figure out if that?s a use case we want to support, if we like how the PEP is supporting it, or if there is possibly some other feature we could add instead that would still support that, while offering the static-ness that we desire. Or if not, at least we'll have it documented as to *why* it needs to be dynamic. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Thu Oct 8 15:20:41 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 8 Oct 2015 14:20:41 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 13:39, Oscar Benjamin wrote: > On 8 October 2015 at 12:46, Paul Moore wrote: >> On 8 October 2015 at 11:18, Oscar Benjamin wrote: >>> >>> A concrete example would be whether or not the numpy source wheel >>> depends on pyopenblas. Depending on how numpy is built the binary >>> wheel may or may not depend on pyopenblas. It doesn't make any sense >>> to say that the numpy source release depends on pyopenblas so what >>> should be the dependencies of the source wheel? >> >> Well, I said this previously but I don't have any objections to the >> idea that binary wheels have additional dependencies - so the source >> wheel doesn't depend on pyopenblas but the binary does. > > Okay, I guess I'm confused by what you mean when you say that a source > wheel (or sdist) should have a "one-to-one" correspondence with a > binary wheel. OK, let me try to clarify: 1. The identifying data is name and version. There should only ever be *one* source wheel with a given name/version. 2. It's possible to have multiple binary wheels derived from a source wheel, but they must all have the same name/version as the source wheel, and only one will ever be considered as the "right" one for a given system. (I.e., compatibility matching takes place and a single best match is selected). The key things here is that binary wheels that differ only in compatibility tags are considered "the same" in this sense (think equivalence class, if you have a mathematical background). Things that may have muddied the water: 1. When I say "all metadata in a source wheel must be static". Technically, tools (pip) don't care about anything other than name and version. The rest is mostly for things like publishing on PyPI and distro packaging, which is not a "core" use. 2. The whole pyopenblas question, which as I understand it is part of the effort to work around the limitations of the compatibility tag mechanism by making binary library dependencies into package dependencies. That approach looks promising, but it moves some of the "what is the appropriate wheel for this environment" question away from compatibility tags and into dependency checking. Which is a non-trivial change to the model. I'd like to see the "binary dependency as package dependency" approach explored properly, but it's completely orthogonal to the question of how source wheels should work. You'd have the same issues with the current sdist format. I'd suggest that the work around this area be separated out from the revamped source distribution discussion. It could probably even be prototyped without getting involved with numpy - how about creating a test project that linked either statically or dynamically to a dummy external library, and then sorting out the wheel issues with that (project testproj depends on project testlib if it's built to link dynamically, and doesn't if it's built statically)? You could do that without anyone needing to understand about BLAS, or numpy, and get much better informed feedback from the wider distutils-sig than you currently get (because you keep needing to explain numpy build processes every time anyone asks a question). For the purposes of this discussion, however, I'd suggest that we assume that building a binary wheel from a source wheel *doesn't* add any new dependencies. Apart from the pyopenblas case, I don't know of any other case where that would happen. (If making that restriction makes the whole discussion trivial, then that's sort of my point - I don't think there *is* a big debate to be had outside of the pyopenblas question). Paul From contact at ionelmc.ro Thu Oct 8 15:34:08 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 8 Oct 2015 16:34:08 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Thu, Oct 8, 2015 at 4:01 PM, Donald Stufft wrote: > One of the features in the original PEP was the ability to produce multiple > different Wheels from the same source release much like how Debian does. > e.g. > numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and > numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions > that > would differ from the name/version of the original sdist thing that was > being > proposed. > ?Sorry if this sounds obtuse but isn't that useless overspecialization? They can just publish `numpy-mlk` and `numpy-?thatblas` or whatever on PyPI, and that will even work better when it comes to dependencies. I mean, if you build something for `numpy-mkl` then it wouldn't work on a `numpy-otherblas` anyway right? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Thu Oct 8 15:51:23 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 8 Oct 2015 14:51:23 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 8 October 2015 at 14:34, Ionel Cristian M?rie? wrote: > > On Thu, Oct 8, 2015 at 4:01 PM, Donald Stufft wrote: >> >> One of the features in the original PEP was the ability to produce >> multiple >> different Wheels from the same source release much like how Debian does. >> e.g. >> numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and >> numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions >> that >> would differ from the name/version of the original sdist thing that was >> being >> proposed. > > > Sorry if this sounds obtuse but isn't that useless overspecialization? They > can just publish `numpy-mlk` and `numpy-thatblas` or whatever on PyPI, and > that will even work better when it comes to dependencies. > I mean, if you > build something for `numpy-mkl` then it wouldn't work on a `numpy-otherblas` > anyway right? It depends. If you're using numpy from pure Python code the difference between mkl and otherblas is probably irrelevant. So in most cases you'd want to be able to depend just on "numpy" but in some cases you'd need to be more specific. Perhaps you could solve that with "provides"... Really though it's probably best to keep the set of binaries on PyPI internally consistent and not try to represent everything. My point earlier was that regardless of what goes on PyPI as the official numpy wheel there will be many people using the numpy code in other ways. If pip is not the only consumer of a source release then it's not really reasonable to dictate (and redesign in a less human-friendly way) its layout purely for pip's benefit. -- Oscar From contact at ionelmc.ro Thu Oct 8 16:16:07 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 8 Oct 2015 17:16:07 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Thu, Oct 8, 2015 at 4:51 PM, Oscar Benjamin wrote: > It depends. If you're using numpy from pure Python code the difference > between mkl and otherblas is probably irrelevant. So in most cases > you'd want to be able to depend just on "numpy" but in some cases > you'd need to be more specific. Perhaps you could solve that with > "provides"... > > Really though it's probably best to keep the set of binaries on PyPI > internally consistent and not try to represent everything. My point > earlier was that regardless of what goes on PyPI as the official numpy > wheel there will be many people using the numpy code in other ways. If > pip is not the only consumer of a source release then it's not really > reasonable to dictate (and redesign in a less human-friendly way) its > layout purely for pip's benefit. > ?Yes indeed. But then shouldn't we talk about proper dependency resolution, compatible releases, meta packages and stuff like that? Unless I completely misunderstood the discussion here (quite probable :-) then this whole multiple source distributions idea is more like a workaround. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Oct 8 16:43:44 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Oct 2015 09:43:44 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 9:23 AM, "Ionel Cristian M?rie?" wrote: > > > On Thu, Oct 8, 2015 at 4:51 PM, Oscar Benjamin wrote: >> >> It depends. If you're using numpy from pure Python code the difference >> between mkl and otherblas is probably irrelevant. So in most cases >> you'd want to be able to depend just on "numpy" but in some cases >> you'd need to be more specific. Perhaps you could solve that with >> "provides"... >> >> Really though it's probably best to keep the set of binaries on PyPI >> internally consistent and not try to represent everything. My point >> earlier was that regardless of what goes on PyPI as the official numpy >> wheel there will be many people using the numpy code in other ways. If >> pip is not the only consumer of a source release then it's not really >> reasonable to dictate (and redesign in a less human-friendly way) its >> layout purely for pip's benefit. > > > ?Yes indeed. But then shouldn't we talk about proper dependency resolution, compatible releases, meta packages and stuff like that? Unless I completely misunderstood the discussion here (quite probable :-) then this whole multiple source distributions idea is more like a workaround. so, because install_requires and extras_require are computed at [egg-info] time, an sdist['s metadata] is/can/maybe technically different on different platforms, no? because of things like if sys.platform in [...]: INSTALL_REQUIRES.extend([...]) > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Oct 8 16:54:43 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Oct 2015 09:54:43 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 8:34 AM, "Ionel Cristian M?rie?" wrote: > > > On Thu, Oct 8, 2015 at 4:01 PM, Donald Stufft wrote: >> >> One of the features in the original PEP was the ability to produce multiple >> different Wheels from the same source release much like how Debian does. e.g. >> numpy-1.0.newsdistthing could produce numpy-pyopenblas-12.6.whl and >> numpy-mkl-7.8.whl, etc etc where there would be a bunch of names/versions that >> would differ from the name/version of the original sdist thing that was being >> proposed. > > > ?Sorry if this sounds obtuse but isn't that useless overspecialization? They can just publish `numpy-mlk` and `numpy-?thatblas` or whatever on PyPI, and that will even work better when it comes to dependencies. I mean, if you build something for `numpy-mkl` then it wouldn't work on a `numpy-otherblas` anyway right? from a reproducibility standpoint, when I run e.g. pip freeze, how (without e.g. numpy-mkl, numpy-blas3k) do I reinstall the same package? * { } URI params, post-#frag=ment encoding? * {+1} prefix-[suffix] how do I install the same version of the package that [you had / this Jupyter notebook has (version_information, watermark)] on [your / their] machine? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 8 17:14:33 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Oct 2015 08:14:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > > Yeah, that's not a good long term solution -- it needs to be moved > > into the metadata (probably by creating an MKL wheel and then > > making > > the numpy wheel depend on it). That's exactly the problem :-) > > > > Are you available on IRC or for a video call or something? I feel like there's > something foundational from both sides that we're each missing here and it'd be > easier to just hash it out in real time rather than lobbying random emails > coming from places of confusion (at least on my side). > > I'm not sure if Paul (or anyone else!) would want to jump in on it too, though > I feel like probably if it's me and you then the two "sides" will probably be > reasonably well represented so if more folks don't want to join that's probably > OK too, particularly since we wouldn't be making any actual decisions there :D This does sound like it would be a good idea -- couldn't hurt, anyway :-). I'll contact you offlist. If anyone else wants to join in, email me... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Oct 8 17:50:42 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 8 Oct 2015 08:50:42 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: > > But the different builds for the different configurations end up with > different metadata. If I'm understanding right, the whole point of > "source wheels" is that they have all the static metadata that pip > needs in order to make decisions, and this has to match the resulting > wheels -- right? > I think we're largely talking about variances in "external" non-python system dependencies (and their build settings). PEP426 currently doesn't cover this in the core metadata, so as it stands, any 2.0 sdist couldn't exhaust these build variances in it's core metadata. There has been some discussion on how to represent external dependencies. In brief, I think the going idea is that it would be through extensions ( https://www.python.org/dev/peps/pep-0426/#metadata-extensions), not in the core python metadata, and the various groups (distro folks, science folks, etc..) would implement these themselves to fulfill their needs... Assuming they did implement such an extension, it would exist in the sdist, and for cases like numpy likely support some notion of "build options", and hence allow for a 1 to many mapping between sdist and binary wheels. Marcus > The way I'm imagining it is that there are multiple levels of metadata > staticness: > > package name, author, description, ... > static in: VCS checkouts, source releases, source wheels, wheels > package version > static in: source releases, source wheels, wheels > package dependencies > static in: source wheels, wheels > environment tag > static in: wheels > > > Of course, there *is* an unsolved issue here, which is how we manage > > compatibility for wheels at the level needed for numpy. But I thought > > the discussion on that was ongoing? I'm concerned that this proposal > > is actually about bypassing that discussion, and instead trying to > > treat incompatibly linked wheels as "different" in terms of project > > metadata, which I think is the wrong way of handling things. I note > > that Christoph Gohlke's numpy builds are tagged with a "+mkl" local > > version modifier - that's presumably intended to mark the fact that > > they are built with an incompatible runtime - but that's a misuse of > > local versions (and I've found it causes niggling issues with how pip > > recognises upgrades, etc). > > Yeah, that's not a good long term solution -- it needs to be moved > into the metadata (probably by creating an MKL wheel and then making > the numpy wheel depend on it). That's exactly the problem :-) > > > So, in summary: Your points above don't seem to me to in any way > > preclude having a single numpy source wheel, and a number of (mutually > > incompatible, but the same in terms of project and version) binary > > wheels. > > Maybe I have misunderstood: does it actually help pip at all to have > static access to name and version, but not to anything else? I've been > assuming not, but I don't think anyone's pointed to any examples yet > of the problems that pip is encountering due to the lack of static > metadata -- would this actually be enough to solve them? > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 8 18:33:47 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Oct 2015 09:33:47 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: > > On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > > > Yeah, that's not a good long term solution -- it needs to be moved > > > into the metadata (probably by creating an MKL wheel and then > > > making > > > the numpy wheel depend on it). That's exactly the problem :-) > > > > > > > Are you available on IRC or for a video call or something? I feel like there's > > something foundational from both sides that we're each missing here and it'd be > > easier to just hash it out in real time rather than lobbying random emails > > coming from places of confusion (at least on my side). > > > > I'm not sure if Paul (or anyone else!) would want to jump in on it too, though > > I feel like probably if it's me and you then the two "sides" will probably be > > reasonably well represented so if more folks don't want to join that's probably > > OK too, particularly since we wouldn't be making any actual decisions there :D > > This does sound like it would be a good idea -- couldn't hurt, anyway :-). I'll contact you offlist. If anyone else wants to join in, email me... Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / 2 pm New York / 7 pm London. Since there's been at least some interest, we'll do it as a Google hangout and send a link around in case anyone wants to listen in, and of course summarize back to the list. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 8 21:58:38 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 8 Oct 2015 12:58:38 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Oct 8, 2015 09:33, "Nathaniel Smith" wrote: > > On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: > > > > On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > > > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: > > > > > Yeah, that's not a good long term solution -- it needs to be moved > > > > into the metadata (probably by creating an MKL wheel and then > > > > making > > > > the numpy wheel depend on it). That's exactly the problem :-) > > > > > > > > > > Are you available on IRC or for a video call or something? I feel like there's > > > something foundational from both sides that we're each missing here and it'd be > > > easier to just hash it out in real time rather than lobbying random emails > > > coming from places of confusion (at least on my side). > > > > > > I'm not sure if Paul (or anyone else!) would want to jump in on it too, though > > > I feel like probably if it's me and you then the two "sides" will probably be > > > reasonably well represented so if more folks don't want to join that's probably > > > OK too, particularly since we wouldn't be making any actual decisions there :D > > > > This does sound like it would be a good idea -- couldn't hurt, anyway :-). I'll contact you offlist. If anyone else wants to join in, email me... > > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / 2 pm New York / 7 pm London. Since there's been at least some interest, we'll do it as a Google hangout and send a link around in case anyone wants to listen in, and of course summarize back to the list. Correction: happening *Monday Oct 12* at 11 am California / 2 pm New York / 7 pm London. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Thu Oct 8 22:05:16 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 8 Oct 2015 22:05:16 +0200 Subject: [Distutils] sampleproject: use tox? Message-ID: <5616CC7C.4060601@thomas-guettler.de> This is a follow up to the thread "Where should I put tests when packaging python modules?" I have never used tox up to now. But reading the mails of the thread, it seems that tox is the current state of the art. This leads me to the conclusion that the sample project should use tox. Any reasons not to do it? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From chris.jerdonek at gmail.com Thu Oct 8 22:40:36 2015 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 8 Oct 2015 13:40:36 -0700 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <5616CC7C.4060601@thomas-guettler.de> References: <5616CC7C.4060601@thomas-guettler.de> Message-ID: On Thursday, October 8, 2015, Thomas G?ttler wrote: > This is a follow up to the thread "Where should I put tests when packaging > python modules?" > > I have never used tox up to now. But reading the mails of the thread, it > seems > that tox is the current state of the art. > > This leads me to the conclusion that the sample project should use tox. > > Any reasons not to do it? In my experience, tox is most useful when developing something that needs to work (and be tested) against multiple Python versions, environments, etc. For projects where the environment is fixed (e.g. an internal application), this isn't necessarily needed. So tox may be an unnecessary addition in those cases. --Chris > > Regards, > Thomas G?ttler > > > > > -- > http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Oct 8 23:33:06 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 8 Oct 2015 23:33:06 +0200 Subject: [Distutils] sampleproject: use tox? References: <5616CC7C.4060601@thomas-guettler.de> Message-ID: <20151008233306.279700b6@fsol> On Thu, 8 Oct 2015 22:05:16 +0200 Thomas G?ttler wrote: > This is a follow up to the thread "Where should I put tests when packaging python modules?" > > I have never used tox up to now. But reading the mails of the thread, it seems > that tox is the current state of the art. I don't see why that would be the case. tox is certainly "the state of the art" in certain circles, but many projects don't mandate it at all or even don't use it at all and rely on other mechanisms (e.g. CI with a rich configuration matrix). Regards Antoine. From robertc at robertcollins.net Fri Oct 9 06:29:58 2015 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 9 Oct 2015 17:29:58 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: Toss me an invite too - I'm very interested in this, for all that I havent' kibbitzed on the thread yet :) On 9 October 2015 at 05:33, Nathaniel Smith wrote: > On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: >> >> On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: >> > >> > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) wrote: >> > > > Yeah, that's not a good long term solution -- it needs to be moved >> > > into the metadata (probably by creating an MKL wheel and then >> > > making >> > > the numpy wheel depend on it). That's exactly the problem :-) >> > > >> > >> > Are you available on IRC or for a video call or something? I feel like >> > there's >> > something foundational from both sides that we're each missing here and >> > it'd be >> > easier to just hash it out in real time rather than lobbying random >> > emails >> > coming from places of confusion (at least on my side). >> > >> > I'm not sure if Paul (or anyone else!) would want to jump in on it too, >> > though >> > I feel like probably if it's me and you then the two "sides" will >> > probably be >> > reasonably well represented so if more folks don't want to join that's >> > probably >> > OK too, particularly since we wouldn't be making any actual decisions >> > there :D >> >> This does sound like it would be a good idea -- couldn't hurt, anyway :-). >> I'll contact you offlist. If anyone else wants to join in, email me... > > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / 2 pm > New York / 7 pm London. Since there's been at least some interest, we'll do > it as a Google hangout and send a link around in case anyone wants to listen > in, and of course summarize back to the list. > > -n > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Robert Collins Distinguished Technologist HP Converged Cloud From opensource at ronnypfannschmidt.de Fri Oct 9 08:57:32 2015 From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Fri, 09 Oct 2015 08:57:32 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: I'd also like to join Am 9. Oktober 2015 06:29:58 MESZ, schrieb Robert Collins : >Toss me an invite too - I'm very interested in this, for all that I >havent' kibbitzed on the thread yet :) > >On 9 October 2015 at 05:33, Nathaniel Smith wrote: >> On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: >>> >>> On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: >>> > >>> > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) >wrote: >>> > > > Yeah, that's not a good long term solution -- it needs to be >moved >>> > > into the metadata (probably by creating an MKL wheel and then >>> > > making >>> > > the numpy wheel depend on it). That's exactly the problem :-) >>> > > >>> > >>> > Are you available on IRC or for a video call or something? I feel >like >>> > there's >>> > something foundational from both sides that we're each missing >here and >>> > it'd be >>> > easier to just hash it out in real time rather than lobbying >random >>> > emails >>> > coming from places of confusion (at least on my side). >>> > >>> > I'm not sure if Paul (or anyone else!) would want to jump in on it >too, >>> > though >>> > I feel like probably if it's me and you then the two "sides" will >>> > probably be >>> > reasonably well represented so if more folks don't want to join >that's >>> > probably >>> > OK too, particularly since we wouldn't be making any actual >decisions >>> > there :D >>> >>> This does sound like it would be a good idea -- couldn't hurt, >anyway :-). >>> I'll contact you offlist. If anyone else wants to join in, email >me... >> >> Looks like this is happening tomorrow (Fri Oct 9) at 11 am California >/ 2 pm >> New York / 7 pm London. Since there's been at least some interest, >we'll do >> it as a Google hangout and send a link around in case anyone wants to >listen >> in, and of course summarize back to the list. >> >> -n >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> MFG Ronny From guettliml at thomas-guettler.de Fri Oct 9 11:32:03 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 9 Oct 2015 11:32:03 +0200 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <20151008233306.279700b6@fsol> References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> Message-ID: <56178993.7070802@thomas-guettler.de> Am 08.10.2015 um 23:33 schrieb Antoine Pitrou: > On Thu, 8 Oct 2015 22:05:16 +0200 > Thomas G?ttler wrote: >> This is a follow up to the thread "Where should I put tests when packaging python modules?" >> >> I have never used tox up to now. But reading the mails of the thread, it seems >> that tox is the current state of the art. > > I don't see why that would be the case. tox is certainly "the state of > the art" in certain circles, but many projects don't mandate it at all > or even don't use it at all and rely on other mechanisms (e.g. CI with > a rich configuration matrix). My question was: Any reasons not to use tox in the sample project? I read your answer like this: Many projects don't mandate it at all or even don't use it at all and rely on other mechanisms Yes, it is correct that many projects is a different way. I search a simple guideline for new comers to set up a python project including testing. "Since many projects don't mandate it at all ..." I don't know if this a good argument against it. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From p.f.moore at gmail.com Fri Oct 9 11:43:52 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 10:43:52 +0100 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <56178993.7070802@thomas-guettler.de> References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On 9 October 2015 at 10:32, Thomas G?ttler wrote: > My question was: Any reasons not to use tox in the sample project? 1. Because the sample project is intended to explain how to *package* a project, not how to test it, or document it, or any of a number of other non-packaging things. 2. Because tox is not universally agreed as best practice, and the sample project is intended to demonstrate agreed best practice. > I search a simple guideline for new comers to set up > a python project including testing. The sample project is not intended for that purpose. It's a good starting point for something like that, but that's not its purpose. Paul PS I sympathise with your wish for a "do this to set up your first project" beginner's guide. I've wanted one of those myself for a long time now. But I don't believe such a thing exists, unfortunately. From contact at ionelmc.ro Fri Oct 9 12:11:18 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Fri, 9 Oct 2015 13:11:18 +0300 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore wrote: > > The sample project is not intended for that purpose. It's a good > starting point for something like that, but that's not its purpose. > > Paul > > PS I sympathise with your wish for a "do this to set up your first > project" beginner's guide. I've wanted one of those myself for a long > time now. But I don't believe such a thing exists, unfortunately. ?Should we at least give some pointers as to what people should look at for a fully fledged project? Like a lists with examples/templates covering all the variants. Eg: "PyPA's sample projects only covers the packaging aspect of a project. For more exhaustive examples take a look at cookiecutter templates, barry's sample, ogrisel's appveyor demo, this and that etc etc" How about something like that? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 9 12:23:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 11:23:25 +0100 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: Well, https://packaging.python.org/en/latest/distributing/ already says """ The section does not aim to cover best practices for Python project development as a whole. For example, it does not provide guidance or tool recommendations for version control, documentation, or testing """ It may be good to add a footnote with your "For more exhaustive examples..." comment - but it would need links, and good luck on getting any sort of agreement on what constitutes good examples :-) Paul On 9 October 2015 at 11:11, Ionel Cristian M?rie? wrote: > > On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore wrote: >> >> >> The sample project is not intended for that purpose. It's a good >> starting point for something like that, but that's not its purpose. >> >> Paul >> >> PS I sympathise with your wish for a "do this to set up your first >> project" beginner's guide. I've wanted one of those myself for a long >> time now. But I don't believe such a thing exists, unfortunately. > > > Should we at least give some pointers as to what people should look at for a > fully fledged project? Like a lists with examples/templates covering all the > variants. Eg: > > "PyPA's sample projects only covers the packaging aspect of a project. For > more exhaustive examples take a look at cookiecutter templates, barry's > sample, ogrisel's appveyor demo, this and that etc etc" > > How about something like that? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro From dw+distutils-sig at hmmz.org Wed Oct 7 01:14:24 2015 From: dw+distutils-sig at hmmz.org (David Wilson) Date: Tue, 6 Oct 2015 23:14:24 +0000 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006095101.031c8a56@fsol> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> Message-ID: <20151006231424.GA30512@k3> On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: > They should be inside the module. That way, you can check an installed > module is ok by running e.g. "python -m mypackage.tests". Any other > choice makes testing installed modules more cumbersome. As Donald mentioned, this doesn't work in the general case since many packages ship quite substantial test data around that often doesn't end up installed, and in other cases since the package requires significant fixture setup or external resources (e.g. running SQLAlchemy tests without a working database server would be meaningless). The option of always shipping test data as a standard part of a package in a vein attempt to always ensure it can be tested (which is not always likely given the SQLAlchemy example above) strikes me as incredibly wasteful, not from some oh-precious-bytes standpoint, but from the perspective of distributing a Python application of any size where the effects of always shipping half-configured test suites has increased the resulting distribution size potentially by 3 or 4x. https://github.com/bennoleslie/pexif is the first hit on Google for a module I thought would need some test data. It's actually quite minimally tested, yet already the tests + data are 3.6x the size of the module itself. I appreciate arguments for inlining tests alongside a package in order to allow reuse of the suite's functionality by consuming applications' test suites, but as above, in the general case this simply isn't something that will always work and can be relied on by default. Is there perhaps a third option that was absent from the original post? e.g. organizing tests in a separate, optional, potentially pip-installable package. > > > outside the module like this: > > > > https://github.com/pypa/sampleproject/tree/master/tests > > There is no actual reason to do that except win a couple kilobytes if > you are distributing your package on floppy disks for consumption on > Z80-based machines with 64KB RAM. > > Even Python *itself* puts its test suite inside the standard library, > not outside it (though some Linux distros may strip it away). > Try "python -m test.regrtest" (again, this may fail if your distro > decided to ship the test suite in a separate package). > > The PyP"A" should definitely fix its sample project to reflect good > practices. > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From chris.barker at noaa.gov Fri Oct 9 18:05:31 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 09:05:31 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: How many people can join a hangout? we may be bumping up against that limit :-) -CHB On Thu, Oct 8, 2015 at 12:58 PM, Nathaniel Smith wrote: > On Oct 8, 2015 09:33, "Nathaniel Smith" wrote: > > > > On Oct 8, 2015 8:14 AM, "Nathaniel Smith" wrote: > > > > > > On Oct 7, 2015 2:58 PM, "Donald Stufft" wrote: > > > > > > > > On October 7, 2015 at 5:28:54 PM, Nathaniel Smith (njs at pobox.com) > wrote: > > > > > > Yeah, that's not a good long term solution -- it needs to be > moved > > > > > into the metadata (probably by creating an MKL wheel and then > > > > > making > > > > > the numpy wheel depend on it). That's exactly the problem :-) > > > > > > > > > > > > > Are you available on IRC or for a video call or something? I feel > like there's > > > > something foundational from both sides that we're each missing here > and it'd be > > > > easier to just hash it out in real time rather than lobbying random > emails > > > > coming from places of confusion (at least on my side). > > > > > > > > I'm not sure if Paul (or anyone else!) would want to jump in on it > too, though > > > > I feel like probably if it's me and you then the two "sides" will > probably be > > > > reasonably well represented so if more folks don't want to join > that's probably > > > > OK too, particularly since we wouldn't be making any actual > decisions there :D > > > > > > This does sound like it would be a good idea -- couldn't hurt, anyway > :-). I'll contact you offlist. If anyone else wants to join in, email me... > > > > Looks like this is happening tomorrow (Fri Oct 9) at 11 am California / > 2 pm New York / 7 pm London. Since there's been at least some interest, > we'll do it as a Google hangout and send a link around in case anyone wants > to listen in, and of course summarize back to the list. > > Correction: happening *Monday Oct 12* at 11 am California / 2 pm New York > / 7 pm London. > > -n > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Fri Oct 9 18:16:13 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Fri, 9 Oct 2015 19:16:13 +0300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Fri, Oct 9, 2015 at 7:05 PM, Chris Barker wrote: > How many people can join a hangout? we may be bumping up against that > limit :-) ?AFAIK there's no limit on the number of people that can listen in. Also, Hangouts can record video (the "On Air" thing).? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 19:04:00 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 10:04:00 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: I"ve been following this thread and gotten a bit lost. But I do care, because I'm a heavy numpy user, and also because I was involved for years in building pacakges for OS-X, and currently need distribute some of my own stuff that has semi-ugly C lib dependencies (Not as ugly as BLAS.,though :-) ) In the last couple years I gave up on PyPi and pip (and went with Anaconda and conda)-- that ecosystem simply doesn't currently support my use-cases. But it would be nice to go back, and it looks like there are some idea on the table that will make that possible. But could someone clarify a thing or two for me?? 1) what in the world is a "source wheel"? And how is it different than an sdist (other than maybe in a different file format. 2) Is it indeed "OK" with the current PEPs and tools for different binary wheels to have different dependencies? This would be the example of, for instance the Matplotlib binary wheel for Windows depends on a py_zlib, whereas the binary wheel for OS-X relies on the the system lib, and therefor does not have that dependency? (and has anyone worked out the linking issues so that that would all work with virtualenv and friends...) if (2) then it seems the issue is what to do with the same package on the same platform having potentially different dependencies -- i.e. numpy w/ mkl and numpy w/some_other_blas. In that case, I think that this could completely explode into m**n possible wheels if we try to accommodate it in a fully flexible manner -- so making it a name-thing (like was proposed here), makes sense to me -- numpy_mkl and numpy_openblas are, as far as pip is concerned, different packages. I think this is OK, as we probably only want some small subset of possible build up on PyPi anyway (only one?). But it does get a bit tricky if you want to put up a package that depends on "pure" numpy -- i.e. it doesn't care which BLAS the numpy it uses, but it DOES need numpy. This could this be Accomidated byt have an "or" option for dependencies: numpy_mkl>=1.9 or numpy_openblas >= 1.9 However, that would mean that the author of that wheel would need to know all of the wheels that might be available up front - less than ideal. Which seems to point more to having an optional "binary_build" component to the name. Not sure what syntax is available, but the idea would be that: "numpy|mkl" and "numpy|openblas" would both match "numpy" which, of course would require changes to the whole stack... and off the top of my head, I'm wondering whether having one "binary_build' flag would be enough, or if we'd find that we wanted n options, and n*m combinations, and this would all blow up. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Oct 9 19:18:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 18:18:19 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 9 October 2015 at 18:04, Chris Barker wrote: > 1) what in the world is a "source wheel"? And how is it different than an > sdist (other than maybe in a different file format. A "source wheel" is the proposed name for a to-be-defined replacement for sdists. For now, you can think of "source wheel" and "sdist" as the same. > 2) Is it indeed "OK" with the current PEPs and tools for different binary > wheels to have different dependencies? This would be the example of, for > instance the Matplotlib binary wheel for Windows depends on a py_zlib, > whereas the binary wheel for OS-X relies on the the system lib, and therefor > does not have that dependency? > (and has anyone worked out the linking issues so that that would all work > with virtualenv and friends...) It's not *currently* OK for different binary wheels to have different dependencies. At least I don't think it is. It's basically not something that as far as I'm aware anyone has ever considered an option up till now, and so it's quite likely that there are assumptions baked into the tools that would break if different builds of (a given version of) a package had different dependencies. One of the proposed approaches to binary dependencies (the one I've been referring to as "the pyopenblas approach") is based on the idea that different wheels could have different dependency metadata. I've tried to enumerate the questions that need to be looked at if we were to go down that route, but to my knowledge, no-one has yet either tested how well things would work if this happened, or audited the code for problematic assumptions. So it's a possibility for the future, but like many things it depends on someone doing the work to make it happen (and as far as I know it's mostly the numpy community who have the need, so I assume someone in that community would need to move this forward). Paul From p.f.moore at gmail.com Fri Oct 9 19:28:09 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Oct 2015 18:28:09 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 9 October 2015 at 18:04, Chris Barker wrote: > Which seems to point more to having an optional "binary_build" component to > the name. Not sure what syntax is available, but the idea would be that: > > "numpy|mkl" > > and > > "numpy|openblas" > > would both match "numpy" > > which, of course would require changes to the whole stack... This sounds more like it's an extension of the wheel "compatibility tag" approach. That's another option I can imagine being worth looking at. Basically, wheels currently encode in their names the Python version, ABI and architecture(s) they work on. That's sufficient for simple uses, but for more complicated scenarios you need more (such as, "this wheel is only valid if library FOO is present). It's not impossible to extend the compatibility tag mechanism to include such things, but there are a number of issues that would need to be thrashed out (spot the common theme here? :-)) Specifically, it's horribly cumbersome to encode everything in the filename, so some more scalable mechanism is needed. Also, you need a notion of compatibility order (is wheel X "more compatible" with your system than wheel Y?). And of course, compatibility is about "does it work with what's there?" which doesn't allow for the possibility of downloading and installing a dependency (that's what dependency resolution does, compatibility checking is at a later stage in the process). So again this is a possible solution, but needs someone to work on the details. ... or thinking again, maybe you mean having multiple packages (numpy_mkl, numpy_openblas, ...) all of which satisfy a "numpy" requirement? That's definitely viable, the Metadata 2.0 spec allows for one package "providing" an implementation of another (https://www.python.org/dev/peps/pep-0426/#provides). But that part of Metadata 2.0 isn't implemented yet - it needs some people with time to work on it again, and it will probably be implemented alongside "sdist 2.0", which is what we've been calling "when we get round to reworking/rethinking the sdist format". Paul Paul From guettliml at thomas-guettler.de Fri Oct 9 19:44:23 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Fri, 9 Oct 2015 19:44:23 +0200 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006231424.GA30512@k3> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> Message-ID: <5617FCF7.9000504@thomas-guettler.de> Am 07.10.2015 um 01:14 schrieb David Wilson: > On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: > >> They should be inside the module. That way, you can check an installed >> module is ok by running e.g. "python -m mypackage.tests". Any other >> choice makes testing installed modules more cumbersome. > > As Donald mentioned, this doesn't work in the general case since many > packages ship quite substantial test data around that often doesn't end > up installed, and in other cases since the package requires significant > fixture setup or external resources (e.g. running SQLAlchemy tests > without a working database server would be meaningless). Should work with a temporary sqlite db. > The option of always shipping test data as a standard part of a package > in a vein attempt to always ensure it can be tested (which is not always > likely given the SQLAlchemy example above) strikes me as incredibly > wasteful, not from some oh-precious-bytes standpoint, but from the > perspective of distributing a Python application of any size where the > effects of always shipping half-configured test suites has increased the > resulting distribution size potentially by 3 or 4x. > > https://github.com/bennoleslie/pexif is the first hit on Google for a > module I thought would need some test data. It's actually quite > minimally tested, yet already the tests + data are 3.6x the size of the > module itself. > > > I appreciate arguments for inlining tests alongside a package in order > to allow reuse of the suite's functionality by consuming applications' > test suites, but as above, in the general case this simply isn't > something that will always work and can be relied on by default. > > > Is there perhaps a third option that was absent from the original post? > e.g. organizing tests in a separate, optional, potentially > pip-installable package. Yes, this third way is plausible. I guess there is even a fourth way. The question remains: If a new comer asks you "How to package my python code and its tests?", there should be one default answer which works for 80% of all cases. I think the confusion gets worse by creating new public accessible repos which explain "Hey that's my way to package stupid simple python code". Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From brett at python.org Fri Oct 9 19:51:45 2015 From: brett at python.org (Brett Cannon) Date: Fri, 09 Oct 2015 17:51:45 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: Listening in to an On Air session is unbounded, but direct participants are capped at 10. On Fri, 9 Oct 2015 at 09:17 Ionel Cristian M?rie? wrote: > > On Fri, Oct 9, 2015 at 7:05 PM, Chris Barker > wrote: > >> How many people can join a hangout? we may be bumping up against that >> limit :-) > > > ?AFAIK there's no limit on the number of people that can listen in. Also, > Hangouts can record video (the "On Air" thing).? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Oct 9 20:00:57 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 9 Oct 2015 12:00:57 -0600 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: <561800D9.8050503@oddbird.net> On 10/09/2015 11:18 AM, Paul Moore wrote: > On 9 October 2015 at 18:04, Chris Barker wrote: >> 1) what in the world is a "source wheel"? And how is it different than an >> sdist (other than maybe in a different file format. > > A "source wheel" is the proposed name for a to-be-defined replacement > for sdists. For now, you can think of "source wheel" and "sdist" as > the same. > >> 2) Is it indeed "OK" with the current PEPs and tools for different binary >> wheels to have different dependencies? This would be the example of, for >> instance the Matplotlib binary wheel for Windows depends on a py_zlib, >> whereas the binary wheel for OS-X relies on the the system lib, and therefor >> does not have that dependency? >> (and has anyone worked out the linking issues so that that would all work >> with virtualenv and friends...) > > It's not *currently* OK for different binary wheels to have different > dependencies. At least I don't think it is. It's basically not > something that as far as I'm aware anyone has ever considered an > option up till now, and so it's quite likely that there are > assumptions baked into the tools that would break if different builds > of (a given version of) a package had different dependencies. AFAIK this is actually just fine currently, it's just not considered ideal for a hopeful future static-metadata world. Today, in the all-metadata-is-dynamic world that we actually live in, the tools aren't able to make any assumptions at all, and just have to download whatever best-match wheel or sdist they find for a given package/version requirement, unpack it, and see what it says about its own metadata (including dependencies). So different binary wheels of the same package at the same version having different dependencies works just fine. It's not like any of the tooling would actually know the difference, since they will only deal with one of those wheels at a given time. But of course this dynamic-metadata world prevents the tools from doing all kinds of useful things (like actually building dependency graphs in advance so they can do proper dependency conflict resolution), which is why we want to move towards static metadata. And it's in that transition that issues like "are different wheels for the same project at the same version allowed to have different dependencies" become relevant. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Fri Oct 9 20:28:28 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 09 Oct 2015 18:28:28 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <561800D9.8050503@oddbird.net> References: <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> <561800D9.8050503@oddbird.net> Message-ID: On Fri, 9 Oct 2015 19:01 Carl Meyer wrote: On 10/09/2015 11:18 AM, Paul Moore wrote: > On 9 October 2015 at 18:04, Chris Barker wrote: >> 1) what in the world is a "source wheel"? And how is it different than an >> sdist (other than maybe in a different file format. > > A "source wheel" is the proposed name for a to-be-defined replacement > for sdists. For now, you can think of "source wheel" and "sdist" as > the same. > >> 2) Is it indeed "OK" with the current PEPs and tools for different binary >> wheels to have different dependencies? This would be the example of, for >> instance the Matplotlib binary wheel for Windows depends on a py_zlib, >> whereas the binary wheel for OS-X relies on the the system lib, and therefor >> does not have that dependency? >> (and has anyone worked out the linking issues so that that would all work >> with virtualenv and friends...) > > It's not *currently* OK for different binary wheels to have different > dependencies. At least I don't think it is. It's basically not > something that as far as I'm aware anyone has ever considered an > option up till now, and so it's quite likely that there are > assumptions baked into the tools that would break if different builds > of (a given version of) a package had different dependencies. AFAIK this is actually just fine currently, it's just not considered ideal for a hopeful future static-metadata world. Why would it need dynamic metadata for the windows matplotlib wheel to have different metadata from the OSX matplotlib wheel? The platform Windows/OSX is static and each wheel declares its own dependencies statically but differently. Am I missing something? -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 20:29:53 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:29:53 -0700 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <5617FCF7.9000504@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <5617FCF7.9000504@thomas-guettler.de> Message-ID: On Fri, Oct 9, 2015 at 10:44 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > The question remains: If a new comer asks you "How to package my > python code and its tests?", there should be one default answer > which works for 80% of all cases. > Should be, maybe -- but clearly there is no consensus as to what the "one" answer shoudl be. But I _think_ there are two answers: 1) inside the package, so it can be installed and the tests run on the installed version. 2) outside the package, so that potentially large or complex test requirements are not required for installation. So the intro docs can lay out those two options, with a bit of text that helps a newbie make a decision. from above, -- I think that EITHER option would work fine for 80% of cases. which makes me think -- why not pick one as the default while clearly documenting the other. And despite the fact that I have always used option (2) in my work, I think, if there is going to be one in the simple example, it should be (1) -- it's a fine way to get started, and users can move their tests outside of the package later if they start getting big. As a package grows to that extent, it will likely need other structural changes anyway. -Chris > I think the confusion gets worse by creating new public accessible > repos which explain "Hey that's my way to package stupid simple > python code". > > Regards, > Thomas G?ttler > > > > > > -- > http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Oct 9 20:35:19 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 9 Oct 2015 12:35:19 -0600 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <561800D9.8050503@oddbird.net> Message-ID: <561808E7.2020505@oddbird.net> On 10/09/2015 12:28 PM, Oscar Benjamin wrote: > Why would it need dynamic metadata for the windows matplotlib wheel to > have different metadata from the OSX matplotlib wheel? The platform > Windows/OSX is static and each wheel declares its own dependencies > statically but differently. Am I missing something? I didn't say that required dynamic metadata (wheel metadata is already static). I just said that it works fine currently, and that it becomes an open question with the move towards static metadata in both source and binary releases, because we have to answer questions like "what information beyond just package/version makes up a complete node in a dependency graph." Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From oscar.j.benjamin at gmail.com Fri Oct 9 20:44:49 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 09 Oct 2015 18:44:49 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: <561808E7.2020505@oddbird.net> References: <561800D9.8050503@oddbird.net> <561808E7.2020505@oddbird.net> Message-ID: On Fri, 9 Oct 2015 19:35 Carl Meyer wrote: On 10/09/2015 12:28 PM, Oscar Benjamin wrote: > Why would it need dynamic metadata for the windows matplotlib wheel to > have different metadata from the OSX matplotlib wheel? The platform > Windows/OSX is static and each wheel declares its own dependencies > statically but differently. Am I missing something? I didn't say that required dynamic metadata (wheel metadata is already static). I just said that it works fine currently, and that it becomes an open question with the move towards static metadata in both source and binary releases, because we have to answer questions like "what information beyond just package/version makes up a complete node in a dependency graph." Assuming it's tied to the operating system it doesn't matter surely. When pip runs on Windows it can ignore dependencies that apply to other platforms so I don't see how this case makes it more complex. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at oddbird.net Fri Oct 9 20:48:05 2015 From: carl at oddbird.net (Carl Meyer) Date: Fri, 9 Oct 2015 12:48:05 -0600 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <561800D9.8050503@oddbird.net> <561808E7.2020505@oddbird.net> Message-ID: <56180BE5.7080708@oddbird.net> On 10/09/2015 12:44 PM, Oscar Benjamin wrote: > > On Fri, 9 Oct 2015 19:35 Carl Meyer > wrote: > > On 10/09/2015 12:28 PM, Oscar Benjamin wrote: > > Why would it need dynamic metadata for the windows matplotlib wheel to > > have different metadata from the OSX matplotlib wheel? The platform > > Windows/OSX is static and each wheel declares its own dependencies > > statically but differently. Am I missing something? > > I didn't say that required dynamic metadata (wheel metadata is already > static). I just said that it works fine currently, and that it becomes > an open question with the move towards static metadata in both source > and binary releases, because we have to answer questions like "what > information beyond just package/version makes up a complete node in a > dependency graph." > > > > Assuming it's tied to the operating system it doesn't matter surely. > When pip runs on Windows it can ignore dependencies that apply to other > platforms so I don't see how this case makes it more complex. Sure. "Assuming it's tied to the operating system" is an assumption that can't be made here, though, if I understand the examples that have already been given repeatedly regarding numpy and BLAS-linking. If "OS" were actually sufficient to distinguish all cases, then the existing wheel platform tags would already be an adequate solution to this problem. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From chris.barker at noaa.gov Fri Oct 9 20:47:28 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:47:28 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Fri, Oct 9, 2015 at 10:28 AM, Paul Moore wrote: > ... or thinking again, maybe you mean having multiple packages > (numpy_mkl, numpy_openblas, ...) all of which satisfy a "numpy" > requirement? yes, that is EXACTLY what I meant. the idea here is that if you build a package that requires both numpy with a particular BLAS, then you'd do: eg: numpy|mkl >= 1.9 but if you build one that only required numpy, and doesn't care which BLAS it's using, then you simply do: numpy >= 1.9 So pip, when asked to install said package, would look and see if any package called numpy|* was installed (the ther right version). if so, it would move along. If not, then it would go look on PyPi for "numpy" -- and this is where it gets tricky -- which numpy| should it use? At this stage, it wouldn't matter, anyone would do. But say it installed numpy|mkl. Then next the user goes to install a package that depends on numpy|openblas -- now pip goes and looks, and find numpy|mkl, but not numpy|openblas -- so it need to go install numpy|openblas. which overwrites numpy|mkl, which is still OK. Until the user goes to install somethign that DOES depend on numpy|mkl. No we are stuck. But is this any different that two packages that depend on two different specific versions of the same package? But all this is making me think that the way this could be handled is by numpy NOT building a BLAS into the package at all. But rather, having the package depend on another package that provides the BLAS, so: numpy on PyPi would depend on this theoretical py_openblas any package that depends on only numpy would be easy. any package that depends on openblas would then depend on py_openblas. So the question is: if I compile my third party package against numpy and mkl, it would depend on numpy and py_mkl. If I went to installed this on a system that had a numpy that depends on openblas, pip would install py_mkl (having already installed numpy and py_openblas). And I'd have a numpy extension calling into a different BLAS than numpy itself is calling into -- would that cause any problems? Python would be linked to two libs with the same names -- would that cause conflict? I'm way out of my depth here! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 20:49:33 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:49:33 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> <561800D9.8050503@oddbird.net> Message-ID: On Fri, Oct 9, 2015 at 11:28 AM, Oscar Benjamin wrote: > Why would it need dynamic metadata for the windows matplotlib wheel to > have different metadata from the OSX matplotlib wheel? The platform > Windows/OSX is static and each wheel declares its own dependencies > statically but differently. Am I missing something? > I think the metadata can be static for the binary wheels, yes. but the dependencies would be different for the two wheels. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Oct 9 20:53:08 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 9 Oct 2015 11:53:08 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <561800D9.8050503@oddbird.net> <561808E7.2020505@oddbird.net> Message-ID: On Fri, Oct 9, 2015 at 11:44 AM, Oscar Benjamin wrote: > Assuming it's tied to the operating system it doesn't matter surely. When > pip runs on Windows it can ignore dependencies that apply to other > platforms so I don't see how this case makes it more complex. > does pip currently support platfrom-specific dependencies? That would solve part of the problem. but yes, there are dependencies that are a function of how the wheel was built, beyond what platform it is on. So those could be static in binary wheel, but not static in the sdist (or source wheel, or...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Fri Oct 9 21:20:44 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 9 Oct 2015 15:20:44 -0400 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <5614AFF2.1090604@thomas-guettler.de> Message-ID: On Wed, Oct 7, 2015 at 2:11 AM, Chris Barker wrote: > On Tue, Oct 6, 2015 at 10:38 PM, Thomas G?ttler > wrote: >> >> Yes, there is not generic "one right way here". >> >> Yes, let's consider individual use cases. >> >> My use case are the docs for new comers: >> >> - https://github.com/pypa/sampleproject >> - https://packaging.python.org/en/latest/distributing/ >> >> That's why started the thread. > > > unfortunately, that isn't a use-case -- every newcomer has a different use > case. Indeed--I've helped newcomers whose very first attempt at packaging Python code includes Cython code for simulations. I think even for newcomers what we should be providing is not a "this is the way to do it" because then they get confused when that way doesn't work for them. Better, in the long term (and I'm happy to contribute to such an effort if it will help) is to provide a sort of Choose Your Own Adventure story. It can't all go on one page because that would be a mess, but a sort of "If you need to do this, read this. If you need to do this, read this. Now if you need to include some data files that are installed in your package read on, because there's really only one right way to do that. But now you have some options if you want to include tests: ..." > I was happy to see this thread, because I thought maybe I"d learn what i > should teach my students - new to python. > > But alas - there clearly really is no consensus. > > What i've told newbies in the past is somethig like: > > """ > if you want your user to be able to install you package, and then run > something like: > > import my_package > my_package.test() > > then put your tests inside the package. > > If you are fine with only being able to run the tests from the source tree > -- then put your tests outside the package. > """ > > but really, newbies have no idea how to make this decsion. > > Maybe we could come up with a decision tree for this -- some guidance for > knowing what to do, when? Exactly. I think it could even be fun :) How could we get started to add something like this to the packaging docs? Erik From mmericke at gmail.com Fri Oct 9 17:05:54 2015 From: mmericke at gmail.com (Michael Merickel) Date: Fri, 9 Oct 2015 10:05:54 -0500 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <20151006231424.GA30512@k3> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> Message-ID: At the pylons project we've had a history of keeping our tests inside the packages. However, keeping them outside has proven to be nicer in some projects as well. 1) It reduces the size of the binary wheels that do not need to package the tests. 2) It still allows you to run the tests on an arbitrary installation if you want to by pulling down the repo and running them against the version installed in the virtualenv. Distributing the tests to every single installation is definitely not a requirement in order to be able to run them for people who want to run them, and decoupling them can really help with that. Projects that are shipping tests inside the package may have them removed upstream and then it is very difficult to run the suite against some binary distribution. On Tue, Oct 6, 2015 at 6:14 PM, David Wilson wrote: > On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: > > > They should be inside the module. That way, you can check an installed > > module is ok by running e.g. "python -m mypackage.tests". Any other > > choice makes testing installed modules more cumbersome. > > As Donald mentioned, this doesn't work in the general case since many > packages ship quite substantial test data around that often doesn't end > up installed, and in other cases since the package requires significant > fixture setup or external resources (e.g. running SQLAlchemy tests > without a working database server would be meaningless). > > The option of always shipping test data as a standard part of a package > in a vein attempt to always ensure it can be tested (which is not always > likely given the SQLAlchemy example above) strikes me as incredibly > wasteful, not from some oh-precious-bytes standpoint, but from the > perspective of distributing a Python application of any size where the > effects of always shipping half-configured test suites has increased the > resulting distribution size potentially by 3 or 4x. > > https://github.com/bennoleslie/pexif is the first hit on Google for a > module I thought would need some test data. It's actually quite > minimally tested, yet already the tests + data are 3.6x the size of the > module itself. > > > I appreciate arguments for inlining tests alongside a package in order > to allow reuse of the suite's functionality by consuming applications' > test suites, but as above, in the general case this simply isn't > something that will always work and can be relied on by default. > > > Is there perhaps a third option that was absent from the original post? > e.g. organizing tests in a separate, optional, potentially > pip-installable package. > > > > > > > outside the module like this: > > > > > > https://github.com/pypa/sampleproject/tree/master/tests > > > > There is no actual reason to do that except win a couple kilobytes if > > you are distributing your package on floppy disks for consumption on > > Z80-based machines with 64KB RAM. > > > > Even Python *itself* puts its test suite inside the standard library, > > not outside it (though some Linux distros may strip it away). > > Try "python -m test.regrtest" (again, this may fail if your distro > > decided to ship the test suite in a separate package). > > > > The PyP"A" should definitely fix its sample project to reflect good > > practices. > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuaxo2 at yahoo.com Sat Oct 10 08:48:59 2015 From: stuaxo2 at yahoo.com (Stuart Axon) Date: Sat, 10 Oct 2015 06:48:59 +0000 (UTC) Subject: [Distutils] Uploading stdeb built debs ? References: <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> Hi,? ? I built a package of using stdeb, any idea if these are acceptable to debian + ubuntu, and where to start / who to contact to get a package included in those distros ??S++ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wichert at wiggy.net Sat Oct 10 10:47:07 2015 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 10 Oct 2015 09:47:07 +0100 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> Message-ID: <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> Also having test code in the package can be very painful if you use tools like venusian which scan and try to import all Python files. Sent from my iPhone > On 09 Oct 2015, at 16:05, Michael Merickel wrote: > > At the pylons project we've had a history of keeping our tests inside the packages. However, keeping them outside has proven to be nicer in some projects as well. 1) It reduces the size of the binary wheels that do not need to package the tests. 2) It still allows you to run the tests on an arbitrary installation if you want to by pulling down the repo and running them against the version installed in the virtualenv. Distributing the tests to every single installation is definitely not a requirement in order to be able to run them for people who want to run them, and decoupling them can really help with that. Projects that are shipping tests inside the package may have them removed upstream and then it is very difficult to run the suite against some binary distribution. > >> On Tue, Oct 6, 2015 at 6:14 PM, David Wilson wrote: >> On Tue, Oct 06, 2015 at 09:51:01AM +0200, Antoine Pitrou wrote: >> >> > They should be inside the module. That way, you can check an installed >> > module is ok by running e.g. "python -m mypackage.tests". Any other >> > choice makes testing installed modules more cumbersome. >> >> As Donald mentioned, this doesn't work in the general case since many >> packages ship quite substantial test data around that often doesn't end >> up installed, and in other cases since the package requires significant >> fixture setup or external resources (e.g. running SQLAlchemy tests >> without a working database server would be meaningless). >> >> The option of always shipping test data as a standard part of a package >> in a vein attempt to always ensure it can be tested (which is not always >> likely given the SQLAlchemy example above) strikes me as incredibly >> wasteful, not from some oh-precious-bytes standpoint, but from the >> perspective of distributing a Python application of any size where the >> effects of always shipping half-configured test suites has increased the >> resulting distribution size potentially by 3 or 4x. >> >> https://github.com/bennoleslie/pexif is the first hit on Google for a >> module I thought would need some test data. It's actually quite >> minimally tested, yet already the tests + data are 3.6x the size of the >> module itself. >> >> >> I appreciate arguments for inlining tests alongside a package in order >> to allow reuse of the suite's functionality by consuming applications' >> test suites, but as above, in the general case this simply isn't >> something that will always work and can be relied on by default. >> >> >> Is there perhaps a third option that was absent from the original post? >> e.g. organizing tests in a separate, optional, potentially >> pip-installable package. >> >> >> > >> > > outside the module like this: >> > > >> > > https://github.com/pypa/sampleproject/tree/master/tests >> > >> > There is no actual reason to do that except win a couple kilobytes if >> > you are distributing your package on floppy disks for consumption on >> > Z80-based machines with 64KB RAM. >> > >> > Even Python *itself* puts its test suite inside the standard library, >> > not outside it (though some Linux distros may strip it away). >> > Try "python -m test.regrtest" (again, this may fail if your distro >> > decided to ship the test suite in a separate package). >> > >> > The PyP"A" should definitely fix its sample project to reflect good >> > practices. >> > >> > Regards >> > >> > Antoine. >> > >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Sat Oct 10 14:06:01 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Sat, 10 Oct 2015 14:06:01 +0200 Subject: [Distutils] venusian pain was: Where should I put tests when packaging python modules? In-Reply-To: <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> Message-ID: <5618FF29.104@thomas-guettler.de> Am 10.10.2015 um 10:47 schrieb Wichert Akkerman: > Also having test code in the package can be very painful if you use tools like venusian which scan and try to import all Python files. Hi Wichert Akkerman, can you please explain this pain? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From ncoghlan at gmail.com Sat Oct 10 16:11:02 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2015 00:11:02 +1000 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On 9 October 2015 at 20:11, Ionel Cristian M?rie? wrote: > > On Fri, Oct 9, 2015 at 12:43 PM, Paul Moore wrote: >> >> >> The sample project is not intended for that purpose. It's a good >> starting point for something like that, but that's not its purpose. >> >> Paul >> >> PS I sympathise with your wish for a "do this to set up your first >> project" beginner's guide. I've wanted one of those myself for a long >> time now. But I don't believe such a thing exists, unfortunately. > > Should we at least give some pointers as to what people should look at for a > fully fledged project? Like a lists with examples/templates covering all the > variants. Eg: > > "PyPA's sample projects only covers the packaging aspect of a project. For > more exhaustive examples take a look at cookiecutter templates, barry's > sample, ogrisel's appveyor demo, this and that etc etc" > > How about something like that? This unfortunately isn't a good idea, at least for folks aiming to publish their first project. While it's well-intentioned, the problem with it is that it can be *tremendously* demotivating for folks that aren't already familiar with all those additional niceties, as it hits newcomers over the head with how much they still have to learn. Testing in particular is problematic, as writing tests involves writing programs about programs. That means you need to already understand a fair bit about programs *and* about how programs fail, to appreciate why it may make sense to write a second program to make sure the first program is behaving as you expect. Folks that already have that knowledge aren't going to need the packaging guide to tell them how to do it, while folks that don't have that knowledge are going to be confused by a digression on testing and documentation when they're wanting to learn about packaging and distribution. That said, what might be interesting is to have a section in the packaging user guide that talks about "The second release", as *that's* where the value of things like testing ("How do you know that everything you published the first time still works?") and semantic versioning ("How do your users know whether or not your API is stable?") start to make more sense. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 10 16:22:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Oct 2015 00:22:04 +1000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 10 October 2015 at 04:47, Chris Barker wrote: > So the question is: if I compile my third party package against numpy and > mkl, it would depend on numpy and py_mkl. > > If I went to installed this on a system that had a numpy that depends on > openblas, pip would install py_mkl (having already installed numpy and > py_openblas). And I'd have a numpy extension calling into a different BLAS > than numpy itself is calling into -- would that cause any problems? Python > would be linked to two libs with the same names -- would that cause > conflict? I'm way out of my depth here! Yes, you'd get weird binary ABI problems in that situation. There's a reason centrally built Linux distros and cross-platform distros like conda exist - we pick the common ABIs that all the components we build use, and enforce them in the build system. For "built by anyone" ABIs, the best you can hope for is to detect-and-report fundamental conflicts, as you can't solve the general case. The approach I proposed for metadata 2.0 is introducing an environmental constraints extension that caused things to fail at install time if the installer detected a binary incompatibility: https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension The equivalent in a pre-metadata 2.0 world would be yet-another-file in the dist-info directory. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From contact at ionelmc.ro Sat Oct 10 16:55:07 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 10 Oct 2015 17:55:07 +0300 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On Sat, Oct 10, 2015 at 5:11 PM, Nick Coghlan wrote: > This unfortunately isn't a good idea, at least for folks aiming to > publish their first project. While it's well-intentioned, the problem > with it is that it can be *tremendously* demotivating for folks that > aren't already familiar with all those additional niceties, as it hits > newcomers over the head with how much they still have to learn. > Testing in particular is problematic, as writing tests involves > writing programs about programs. That means you need to already > understand a fair bit about programs *and* about how programs fail, to > appreciate why it may make sense to write a second program to make > sure the first program is behaving as you expect. Folks that already > have that knowledge aren't going to need the packaging guide to tell > them how to do it, while folks that don't have that knowledge are > going to be confused by a digression on testing and documentation when > they're wanting to learn about packaging and distribution. > ?This begs the conception of yet another opinionated guide for X, Y and Z in Python. I believe we got plenty of those, and what encouraged their creation is exactly this sort of thinking. The problem is that none of them are really comprehensive, well maintained or authoritative enough (we wouldn't have this discussion if there was). They may have been "well-intentioned" but they only create confusion now ... What if there was something official-ish that takes the packaging guide further, for people that want more than just packaging? It seems that there is some demand. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Oct 10 17:18:30 2015 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 10 Oct 2015 10:18:30 -0500 Subject: [Distutils] sampleproject: use tox? In-Reply-To: References: <5616CC7C.4060601@thomas-guettler.de> <20151008233306.279700b6@fsol> <56178993.7070802@thomas-guettler.de> Message-ID: On Oct 10, 2015 9:55 AM, "Ionel Cristian M?rie?" wrote: > > > On Sat, Oct 10, 2015 at 5:11 PM, Nick Coghlan wrote: >> >> This unfortunately isn't a good idea, at least for folks aiming to >> publish their first project. While it's well-intentioned, the problem >> with it is that it can be *tremendously* demotivating for folks that >> aren't already familiar with all those additional niceties, as it hits >> newcomers over the head with how much they still have to learn. >> Testing in particular is problematic, as writing tests involves >> writing programs about programs. That means you need to already >> understand a fair bit about programs *and* about how programs fail, to >> appreciate why it may make sense to write a second program to make >> sure the first program is behaving as you expect. Folks that already >> have that knowledge aren't going to need the packaging guide to tell >> them how to do it, while folks that don't have that knowledge are >> going to be confused by a digression on testing and documentation when >> they're wanting to learn about packaging and distribution. > > > ?This begs the conception of yet another opinionated guide for X, Y and Z in Python. I believe we got plenty of those, and what encouraged their creation is exactly this sort of thinking. > > The problem is that none of them are really comprehensive, well maintained or authoritative enough (we wouldn't have this discussion if there was). They may have been "well-intentioned" but they only create confusion now ... > > What if there was something official-ish that takes the packaging guide further, for people that want more than just packaging? It seems that there is some demand. references for such a CreativeWork * http://docs.python-guide.org/en/latest/scenarios/ci/ * https://en.wikipedia.org/wiki/Continuous_integration#Best_practices * http://docs.python-guide.org/en/latest/writing/documentation/ * https://docs.python.org/devguide/documenting.html#sections * http://docs.python-guide.org/en/latest/writing/tests/ * https://packaging.python.org/en/latest/glossary/ * /peps /projects * https://westurner.org/tools/#python-packages * more links above (in this thread) * dox (docker + tox), cookiecutter- * https://westurner.org/tools/#python * https://westurner.org/wiki/awesome-python-testing > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wichert at wiggy.net Sat Oct 10 21:47:41 2015 From: wichert at wiggy.net (Wichert Akkerman) Date: Sat, 10 Oct 2015 20:47:41 +0100 Subject: [Distutils] venusian pain was: Where should I put tests when packaging python modules? In-Reply-To: <5618FF29.104@thomas-guettler.de> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> <5618FF29.104@thomas-guettler.de> Message-ID: <8711BE71-F721-46F0-A30E-097E098547C6@wiggy.net> > On 10 Oct 2015, at 13:06, Thomas G?ttler wrote: > > Am 10.10.2015 um 10:47 schrieb Wichert Akkerman: >> Also having test code in the package can be very painful if you use tools like venusian which scan and try to import all Python files. > > > Hi Wichert Akkerman, > > can you please explain this pain? Importing tests often leads to problems for two reasons: 1) tests try to import things that are not normally installed (mock, pytest, redis-mock, etc.) which breaks application startup, and 2) some tests have import side-effects which can be fine for testing purposes, but should never trigger during normal usage. Wichert. From lac at openend.se Sat Oct 10 21:53:31 2015 From: lac at openend.se (Laura Creighton) Date: Sat, 10 Oct 2015 21:53:31 +0200 Subject: [Distutils] warning about potential problem for wheels Message-ID: <201510101953.t9AJrVwx007005@fido.openend.se> (note, I currently don't have mail delivery on for distutils. I could change this, but right now I don't think I have a lot to contribute. This is just a warning). If you have old windows hardware, which does not support SSE2, and windows 7, you can build your own python 3.5. This will work. But wheels won't, you will need to build them from source as well. see: http://bugs.python.org/issue25361 This means that wheels could start failing. It would be good if the wheels themselves could detect this problem and protest in a reasonable fashion, but I have no idea if this is possible. In any case, I thought you needed to know. Laura From oscar.j.benjamin at gmail.com Sat Oct 10 23:52:58 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 10 Oct 2015 21:52:58 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510101953.t9AJrVwx007005@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> Message-ID: On Sat, 10 Oct 2015 20:53 Laura Creighton wrote: (note, I currently don't have mail delivery on for distutils. I could change this, but right now I don't think I have a lot to contribute. This is just a warning). If you have old windows hardware, which does not support SSE2, and windows 7, you can build your own python 3.5. This will work. But wheels won't, you will need to build them from source as well. see: http:// bugs.python.org /issue25361 This means that wheels could start failing. It would be good if the wheels themselves could detect this problem and protest in a reasonable fashion, but I have no idea if this is possible. In any case, I thought you needed to know. There is no way for wheels to do this. A wheel is just a zip file with a standardised layout. Pip just extracts the zip, reads the metadata and copies the files to the appropriate locations. The metadata has no way to describe the fact that it the wheel contains SSE2 dependent binaries. The standard tools used to create wheels don't know anything about the contents of the compiled binaries so they don't really have a way to detect that the wheel depends on SSE2. Really this is just a case of an unsupported platform. It's unfortunate that CPython doesn't properly support this hardware but I think it's reasonable that if you have to build your interpreter from source then you have to build your extension modules as well. I'm not sure of a robust solution to detecting the problem at install time. Extension module authors can only really guarantee that their Windows binaries are compatible with standard released binaries. So if someone builds their own interpreter using different compiler options then there's no real way for pip or the extension module author to know if the binaries will be compatible. So either pip rejects all binaries for a non standard interpreter build or it installs them and hopes for the best. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From lac at openend.se Sun Oct 11 00:37:17 2015 From: lac at openend.se (Laura Creighton) Date: Sun, 11 Oct 2015 00:37:17 +0200 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> Message-ID: <201510102237.t9AMbH5e018456@fido.openend.se> In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: >Really this is just a case of an unsupported platform. It's unfortunate >that CPython doesn't properly support this hardware but I think it's >reasonable that if you have to build your interpreter from source then you >have to build your extension modules as well. Alas that there is no easy way to detect. The situation I am imagining is where the administrators of a school build pythons for the students to run on their obsolete hardware, and then the poor students don't understand why pip doesn't work. But I suppose we will just get to deal with that problem when and if it happens. Laura From p.f.moore at gmail.com Sun Oct 11 00:47:58 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Oct 2015 23:47:58 +0100 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510102237.t9AMbH5e018456@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: You could, I suppose, patch your custom Python to *not* treat wheels built for cp35-none-win32 (or cp35-none-win_amd64, as appropriate) as compatible with that build (that requires a patch to the tag matching code in pip). You then wouldn't be able to use any binary wheels on PyPI, which is a bit heavy handed but probably necessary. But that's probably way more complex than is justified... Paul From oscar.j.benjamin at gmail.com Sun Oct 11 01:09:17 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 10 Oct 2015 23:09:17 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510102237.t9AMbH5e018456@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: On Sat, 10 Oct 2015 23:37 Laura Creighton wrote: In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: >Really this is just a case of an unsupported platform. It's unfortunate >that CPython doesn't properly support this hardware but I think it's >reasonable that if you have to build your interpreter from source then you >have to build your extension modules as well. Alas that there is no easy way to detect. The situation I am imagining is where the administrators of a school build pythons for the students to run on their obsolete hardware, and then the poor students don't understand why pip doesn't work. But I suppose we will just get to deal with that problem when and if it happens. Does it sound plausible to you that a school would build their own Pythons? I only know a few schools and I'd be very surprised if this happened at one of them but I guess there's a lot of schools in the world... The administrators at my daughter's school don't even understand how to put text into an email let alone install compilers and build Python! -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 11 01:10:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 16:10:39 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <201510102237.t9AMbH5e018456@fido.openend.se> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > >Really this is just a case of an unsupported platform. It's unfortunate > >that CPython doesn't properly support this hardware but I think it's > >reasonable that if you have to build your interpreter from source then you > >have to build your extension modules as well. > > Alas that there is no easy way to detect. The situation I am > imagining is where the administrators of a school build pythons for > the students to run on their obsolete hardware, and then the poor > students don't understand why pip doesn't work. But I suppose we > will just get to deal with that problem when and if it happens. In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: - ~0.5% of Firefox crashes are on machines that are missing SSE2 - <0.1% of machines with Steam installed are missing SSE2 I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sun Oct 11 01:24:23 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 10 Oct 2015 19:24:23 -0400 Subject: [Distutils] Uploading stdeb built debs ? References: <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> <1688702971.1798587.1444459739953.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20151010192423.416e35b3@anarchist.wooz.org> On Oct 10, 2015, at 06:48 AM, Stuart Axon via Distutils-SIG wrote: >Hi,? ? I built a package of using stdeb, any idea if these are acceptable to >debian + ubuntu, and where to start / who to contact to get a package >included in those distros ??S++ You can contact debian-python at lists.debian.org Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From njs at pobox.com Sun Oct 11 02:22:12 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 17:22:12 -0700 Subject: [Distutils] Bad setup.py dependencies on numpy, and 'pip upgrade' (was: Towards a simple and standard sdist format that isn't intertwined with distutils) Message-ID: On Sun, Oct 4, 2015 at 1:02 PM, Paul Moore wrote: [...] > A > common bug report for pip is users finding that their installs fail, > because setup.py requires numpy to be installed in order to run, and > yet pip is running setup.py egg-info precisely to find out what the > requirements are. We tell the user that the setup.py is written > incorrectly, and they should install numpy and retry the install, but > it's not a good user experience. And from a selfish point of view, > users blame *pip* for the consequences of a project whose code to > generate the metadata is buggy. Those bug reports are a drain on the > time of the pip developers, as well as a frustrating experience for > the users. [...] This is mostly orthogonal to the other discussion, but it's a problem relevant to distutils-sig where I think there's a lot of confusion, so I just wanted to try and clarify what's happening here. It's true that almost no packages accurately declare their dependencies on numpy. This is for two reasons: First, for packages that build-depend on numpy (e.g. they contain extensions that need numpy's header files), then distutils' design makes things tricky, b/c you have to call np.get_include() to find the headers and pass that to setup(). But this isn't the main issue -- solving this requires nasty hacks, but at least the more prominent projects are gradually gaining those hacks. The main issue is that people really, really, REALLY hate the recursive behavior of 'pip install -U'. They hate it *so much* -- especially the way it has a habit of suddenly trying to rebuild numpy when all you were trying to do was upgrade some little pure-python package -- that they actively refuse to accurately report their project's dependencies, because install -U can't recurse over dependencies that it can't see. E.g., here's two setup.py files for extremely mainstream projects, that both report accurate setup_requires and install_requires on numpy if and only if numpy is not already installed; if it is installed then they pretend that they don't need it: https://github.com/scipy/scipy/blob/master/setup.py#L200-L208 https://github.com/statsmodels/statsmodels/blob/master/setup.py#L96-L156 Obviously this is terrible -- it means that the wheels for these projects end up with incorrect dependencies (usually! it's kinda random!), and even if you install from source then whether future upgrades will work correctly depends on arbitrary details about how your virtualenv was configured when you did the original install. But people care about this so much that you actually have prominent developers going around and filing bugs on packages complaining that they provide accurate metadata and that this is a bug, they should lie instead. E.g.: https://github.com/pydata/patsy/issues/5 I'm not sure I agree with this conclusion, but the users have spoken. AFAICT the only way to fix this problem and start getting packages with accurate metadata is for pip to gain a non-recursive upgrade mode. The bug for that is https://github.com/pypa/pip/issues/59 AFAICT from reading that thread, work on this has stalled out because of the following reasoning: 1) Everyone agrees that pip should have 'upgrade $PKG' and 'upgrade-all' commands, and 'install -U' should be deprecated/removed. 2) But implementing 'upgrade-all' is tricky and dangerous without first fixing pip's dependency resolver. 3) Therefore we can't add 'upgrade' or 'upgrade-all' until after we fix pip's dependency resolver. I feel like there's a certain logical gap between (2) and (3)... we could defer 'upgrade-all' until later but start supporting 'upgrade $PKG' right now, couldn't we? (It'd be implemented as the equivalent of 'pip install $PKG=$LATEST_VERSION', which is not scary looking at all.) -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Sun Oct 11 02:28:40 2015 From: donald at stufft.io (Donald Stufft) Date: Sat, 10 Oct 2015 20:28:40 -0400 Subject: [Distutils] Bad setup.py dependencies on numpy, and 'pip upgrade' (was: Towards a simple and standard sdist format that isn't intertwined with distutils) In-Reply-To: References: Message-ID: On October 10, 2015 at 8:22:16 PM, Nathaniel Smith (njs at pobox.com) wrote: The bug for that is? https://github.com/pypa/pip/issues/59? AFAICT from reading that thread, work on this has stalled out because? of the following reasoning:? 1) Everyone agrees that pip should have 'upgrade $PKG' and? 'upgrade-all' commands, and 'install -U' should be deprecated/removed.? 2) But implementing 'upgrade-all' is tricky and dangerous without? first fixing pip's dependency resolver.? 3) Therefore we can't add 'upgrade' or 'upgrade-all' until after we? fix pip's dependency resolver.? I feel like there's a certain logical gap between (2) and (3)... we? could defer 'upgrade-all' until later but start supporting 'upgrade? $PKG' right now, couldn't we? (It'd be implemented as the equivalent? of 'pip install $PKG=$LATEST_VERSION', which is not scary looking at? all.)? I don?t see any reason that ``pip upgrade`` can?t be implemented prior to ``pip upgrade-all`` (or ``pip upgrade ?all``). It?s not adding a new thing that can cause problems without a dependency solver, it?s just moving the existing one and switching it so it?s non recursive by default. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Oct 11 05:26:50 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 20:26:50 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Fri, Oct 9, 2015 at 10:04 AM, Chris Barker wrote: > 1) what in the world is a "source wheel"? And how is it different than an > sdist (other than maybe in a different file format. It seemed like people had different ideas about what a platonic "sdist" even is and this was causing us to talk past each other. So I proposed the terms "source release" and "source wheel" as two somewhat abstract concepts to let us to state positions like "I think an sdist should fill the role of a source wheel, and that there's no need for source releases", or "I think source wheels and source releases are different and important, so we need one spec for each". I'm not sure how much it helped :-) This is the email that suggested distinguishing between the terms: https://mail.python.org/pipermail/distutils-sig/2015-October/026981.html and the "source wheel" term in particular was stolen from Donald's earlier email: https://mail.python.org/pipermail/distutils-sig/2015-October/026964.html -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sun Oct 11 06:01:48 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 21:01:48 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On Wed, Oct 7, 2015 at 2:41 PM, Paul Moore wrote: [...] > Possibly the nearest > we'd have to an issue is over allowing the build process to *add* > dependencies to a binary wheel (e.g. a some builds depend on > currently-hypothetical MKL wheel, which provides needed DLLs). I don't > in principle object to that, but I'd like to see a fleshed out > proposal on how wheels containing just DLLs (as opposed to Python > packages) would work in practice - until we have a mechanism for > building/distributing such wheels, I think it's premature to worry > about specifying dependencies. Just to answer this: There's no formal proposal because it doesn't involve distutils-sig -- it's something that we already know how to do and are working on implementing :-). The only reason it doesn't already exist on PyPI is that Windows is the main target, and toolchain problems mean that we aren't yet able to build numpy or scipy wheels on Windows at all. (Long story...) The basic idea though is just, we make a "python package" with trivial structure: pyopenblas/ __init__.py openblas.dll include/ ... Usage: # in downstream package setup.py Extension(..., include_dirs=... + pyopenblas.get_include(), linker_arguments=... + pyopenblas.get_linker_arguments(), ...) # in downstream package __init__.py import pyopenblas pyopenblas.enable() Implementation: pyopenblas/__init__.py contains some code like: DIR = os.path.dirname(__file__) def get_include(): return [os.path.join(DIR, "include")] def get_linker_arguments(): return ["-L" + DIR, "-lpyopenblas"] def enable(): # Platform specific code to let the runtime linker find libopenblas.so if WINDOWS: # ctypes magic to preload the DLL elif LINUX: # modify os.environ["LD_LIBRARY_PATH"] else: ... -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Sun Oct 11 06:31:27 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Oct 2015 21:31:27 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? Message-ID: Hi all, I'm finding it impossible to keep track of that other thread, and I guess I'm probably not the only one, so I figured I'd try splitting a few of the more specific discussions out :-). One thing that seems to be a key issue, but where I remain very confused, is the question of what pip actually needs from an sdist. (Not PyPI, just pip or other package install tools.) Right now, IIUC, there are three times that pip install touches sdist-related metadata: 1) it uses the name+version that are embedded in the sdist filename to select an sdist from an index like PyPI 2) after unpacking this sdist it then calls 'setup.py egg_info' to get the full metadata for the wheel (or wheel equivalent) that this sdist will eventually produce. Specifically what it does with this is extract the setup_requires and install_requires fields, and uses them to go find other wheels/sdists that also need to be installed 3) eventually it actually builds the package, and this produces a wheel (or wheel equivalent) that has its own metadata (which often matches the metadata from egg_info in step (2), but not always) Is that a correct description of current behavior? Is there anything that pip ever looks at besides name, version, dependencies? Paul says that this is broken, and that pip gets lots of bug reports that "can be traced back to needing to run setup.py egg-info to get metadata" [1]. Since AFAICT the only metadata that pip actually touches is name, version, and dependencies, and it already knows the name and version before it runs egg_info, I assume that what this means is that it's crucial for pip to have static access to dependency information? OTOH in another email Paul says that name and version are the minimum he wants [2], so maybe I'm reading too much into this :-). >From the discussion so far, it sounds like the particularly crucial question is whether pip needs to statically know dependencies before building a wheel. Trying to reason through from first principles, I can't see any reason why it would. It would be somewhat convenient if sdists did list their binary dependencies: if that were the case, then pip could take a strictly phased approach: 1) solve the complete dependency graph to find a set of packages to install / remove 2) for all the packages-to-be-installed that are sdists, turn them into wheels 3) install all the wheels OTOH if sdists have only name and version statically, but not dependency information, then you need to do something like: 1) create a fake dependency graph that contains accurate information for all known wheels, and for each sdist add a fake node that has the right name and version number but pretends not to have any dependencies. 2) solve this graph to find the set of packages to install / remove 3) if any of the packages-to-be-installed are sdists, then fetch them, run egg_info or build them or whatever to get their real dependencies, add these to the graph, and go to step 1 4) else, we have wheels for everything; install them. (This works because dependencies are constraints -- adding dependencies can only reduce the space of possible solutions, never enlarge it. Also, because by the time we decide to fetch and build any sdists, we already know that we're very likely to want to install them, so the performance penalty for building packages we turn out not to want is not high. And, crucially, we know that there exists some set of dependency metadata which would convince us to install these sdists, and dependency metadata is under the package author's control, so we already have established a trust route to the author of this package -- if they don't declare any dependencies, then we'll be installing and running arbitrary code of theirs, so running arbitrary code to check their dependencies doesn't require any additional trust.) But there's often a large difference between what we work out from first principles and how things actually work :-). Is there anything I'm missing in the analysis above? Do the relevant pip maintainers even read this mailing list? :-) -n [1] https://mail.python.org/pipermail/distutils-sig/2015-October/026960.html [2] https://mail.python.org/pipermail/distutils-sig/2015-October/026942.html -- Nathaniel J. Smith -- http://vorpus.org From p.f.moore at gmail.com Sun Oct 11 13:03:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2015 12:03:44 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: <20151005152201.4433e9a6@fsol> <20151005153910.7f921992@fsol> <9423722A-3A28-4A72-A141-107E6DB520A3@stufft.io> Message-ID: On 11 October 2015 at 05:01, Nathaniel Smith wrote: > There's no formal proposal because it doesn't involve distutils-sig -- > it's something that we already know how to do and are working on > implementing :-). The only reason it doesn't already exist on PyPI is > that Windows is the main target, and toolchain problems mean that we > aren't yet able to build numpy or scipy wheels on Windows at all. > (Long story...) > > The basic idea though is just, we make a "python package" with trivial > structure: Cool, thanks for the clarification - I hadn't realised it was as straightforward as this. Paul From p.f.moore at gmail.com Sun Oct 11 13:49:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2015 12:49:19 +0100 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 11 October 2015 at 05:31, Nathaniel Smith wrote: > Do the relevant pip maintainers > even read this mailing list? :-) Donald and I are pip maintainers (not the only ones, but probably the ones who make the most noise on this list :-)). The comments I've made that you quoted are based mostly on how I recall pip works, it's unfortunately been some time since I looked into pip's dependency resolution and install code (it's a bit of a pit of snakes) and I haven't had time to check the details when commenting on this thread, so I don't claim my comments are authoritative. I think your description of the behaviour is right. As regards what pip could do, technically you are of course correct (it's possible, it just needs someone willing to make the code changes). I don't know, however, if the approach you're proposing fits with how we currently envisage pip developing. Specifically, my personal goal [1] is that we get to a point where pip can do all of the dependency resolution steps, getting to a point where it knows exactly what it will install, *before* it starts downloading packages, running build steps, etc. (And I don't see that sitting well with your proposal). The reason I want that is so that I would like to be able to see what pip plans on doing, and audit it before it begins (a "pip install --dry-run" type of step). Typically, this lets me avoid getting part-way through an install only to find that I need a project that doesn't have a wheel and I can't build from the sdist. That's a real problem I deal with a lot - at the moment I work with "pip list --outdated", check the listing to see what provides wheels, and update just those packages individually. But that's messy and manual, and not always 100% effective - and I'd like to be able to do better. I also don't think it's reasonable to focus solely on what *pip* requires. Just as important (IMO) is allowing PyPI to provide rich metadata. In particular, I write a lot of programs that query PyPI for package information. There are a lot of useful scripts I would like to be able to write that need to query PyPI for the dependency details of a package, and that simply isn't possible at the moment. Most of these are ad-hoc or personal-use scripts, and a lot of things I simply haven't done because I can't. So it's hard to quantify the benefits. But currently, the wonderful Metadata 2.0 ideal future promises me static dependency data on PyPI. That will only happen if we get enough people interested in implementing it, but I live in hope. If I understand your proposals, they are closing off that option, and explicitly stating that we'll never have dependency metadata on PyPI for projects that don't publish wheels. Maybe you're right, and it'll never be possible. But I don't want to give up that dream without being sure it's necessary - which is why I push back so hard on the idea. Paul [1] Which is not backed up by me having the time to write the code, so the only relevance here is that I'm inclined to support proposals that work towards that goal. From steve.dower at python.org Sun Oct 11 16:17:42 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 11 Oct 2015 07:17:42 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Nathaniel Smith" Sent: ?10/?10/?2015 16:11 To: "Laura Creighton" Cc: "distutils-sig" Subject: Re: [Distutils] warning about potential problem for wheels On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > >Really this is just a case of an unsupported platform. It's unfortunate > >that CPython doesn't properly support this hardware but I think it's > >reasonable that if you have to build your interpreter from source then you > >have to build your extension modules as well. > > Alas that there is no easy way to detect. The situation I am > imagining is where the administrators of a school build pythons for > the students to run on their obsolete hardware, and then the poor > students don't understand why pip doesn't work. But I suppose we > will just get to deal with that problem when and if it happens. In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: - ~0.5% of Firefox crashes are on machines that are missing SSE2 - <0.1% of machines with Steam installed are missing SSE2 I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Oct 11 16:31:24 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 Oct 2015 10:31:24 -0400 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> Message-ID: <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? Sent from my iPhone > On Oct 11, 2015, at 10:17 AM, Steve Dower wrote: > > An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). > > That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). > > Cheers, > Steve > > Top-posted from my Windows Phone > From: Nathaniel Smith > Sent: ?10/?10/?2015 16:11 > To: Laura Creighton > Cc: distutils-sig > Subject: Re: [Distutils] warning about potential problem for wheels > > On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > > > >Really this is just a case of an unsupported platform. It's unfortunate > > >that CPython doesn't properly support this hardware but I think it's > > >reasonable that if you have to build your interpreter from source then you > > >have to build your extension modules as well. > > > > Alas that there is no easy way to detect. The situation I am > > imagining is where the administrators of a school build pythons for > > the students to run on their obsolete hardware, and then the poor > > students don't understand why pip doesn't work. But I suppose we > > will just get to deal with that problem when and if it happens. > > In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: > - ~0.5% of Firefox crashes are on machines that are missing SSE2 > - <0.1% of machines with Steam installed are missing SSE2 > > I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) > > -n > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Sun Oct 11 17:20:55 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 11 Oct 2015 15:20:55 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: On Sun, 11 Oct 2015 15:31 Donald Stufft wrote: Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? It should be fine either way as long as the CPU can handle the particular instructions used. X86 is backward compatible like that so unless the compiler does something funny when the SSE option is enabled it should be fine. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sun Oct 11 17:07:30 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 11 Oct 2015 08:07:30 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: Yes. SSE never makes it into the module ABI, it's just an implementation concern. (I believe it may be used across function/module boundaries internally when the compiler can prove that it's inaccessible from the outside - a more space effective form of inlining.) There will be some 3.5.0-built wheels that won't work on some machines running 3.5.1 without SSE, and some wheels will see a performance decrease when built with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones weren't expecting the perf increase anyway, so compatibility is likely there biggest concern. This does only affect 32-bit builds, so now I'm thinking about the possibility of treating those as highly compatible while the 64-bit ones get better performance treatment, though I'm not sure how that could actually play out. It may help remove some of the questions about which one to use though. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Donald Stufft" Sent: ?10/?11/?2015 7:31 To: "Steve Dower" Cc: "distutils-sig" ; "Laura Creighton" Subject: Re: [Distutils] warning about potential problem for wheels Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? Sent from my iPhone On Oct 11, 2015, at 10:17 AM, Steve Dower wrote: An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). Cheers, Steve Top-posted from my Windows Phone From: Nathaniel Smith Sent: ?10/?10/?2015 16:11 To: Laura Creighton Cc: distutils-sig Subject: Re: [Distutils] warning about potential problem for wheels On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: > > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: > > >Really this is just a case of an unsupported platform. It's unfortunate > >that CPython doesn't properly support this hardware but I think it's > >reasonable that if you have to build your interpreter from source then you > >have to build your extension modules as well. > > Alas that there is no easy way to detect. The situation I am > imagining is where the administrators of a school build pythons for > the students to run on their obsolete hardware, and then the poor > students don't understand why pip doesn't work. But I suppose we > will just get to deal with that problem when and if it happens. In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: - ~0.5% of Firefox crashes are on machines that are missing SSE2 - <0.1% of machines with Steam installed are missing SSE2 I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) -n _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Oct 11 17:49:58 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 11 Oct 2015 11:49:58 -0400 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: <06B5B098-7A7F-4766-BAD8-9EAEC8EE766A@stufft.io> Okay. Thanks (and Oscar too). I'm not really that great at C and compiled stuff so I wasn't sure if it would have any negative effects like that. Seems like a reasonable change. Sent from my iPhone > On Oct 11, 2015, at 11:07 AM, Steve Dower wrote: > > Yes. SSE never makes it into the module ABI, it's just an implementation concern. (I believe it may be used across function/module boundaries internally when the compiler can prove that it's inaccessible from the outside - a more space effective form of inlining.) > > There will be some 3.5.0-built wheels that won't work on some machines running 3.5.1 without SSE, and some wheels will see a performance decrease when built with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones weren't expecting the perf increase anyway, so compatibility is likely there biggest concern. > > This does only affect 32-bit builds, so now I'm thinking about the possibility of treating those as highly compatible while the 64-bit ones get better performance treatment, though I'm not sure how that could actually play out. It may help remove some of the questions about which one to use though. > > Cheers, > Steve > > Top-posted from my Windows Phone > From: Donald Stufft > Sent: ?10/?11/?2015 7:31 > To: Steve Dower > Cc: distutils-sig; Laura Creighton > Subject: Re: [Distutils] warning about potential problem for wheels > > Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What about the inverse? > > Sent from my iPhone > >> On Oct 11, 2015, at 10:17 AM, Steve Dower wrote: >> >> An extra data point is that we've had exactly one report of Python 3.5 not working due to lack of SSE, and that person was also on Windows XP (so zero reports on supported platforms). >> >> That said, I should probably just fix 3.5.1 to not use SSE for core or distutils builds. I doubt there was a huge performance increase due to it (mainly in memcpy I'd assume). >> >> Cheers, >> Steve >> >> Top-posted from my Windows Phone >> From: Nathaniel Smith >> Sent: ?10/?10/?2015 16:11 >> To: Laura Creighton >> Cc: distutils-sig >> Subject: Re: [Distutils] warning about potential problem for wheels >> >> On Oct 10, 2015 3:37 PM, "Laura Creighton" wrote: >> > >> > In a message of Sat, 10 Oct 2015 21:52:58 -0000, Oscar Benjamin writes: >> > >> > >Really this is just a case of an unsupported platform. It's unfortunate >> > >that CPython doesn't properly support this hardware but I think it's >> > >reasonable that if you have to build your interpreter from source then you >> > >have to build your extension modules as well. >> > >> > Alas that there is no easy way to detect. The situation I am >> > imagining is where the administrators of a school build pythons for >> > the students to run on their obsolete hardware, and then the poor >> > students don't understand why pip doesn't work. But I suppose we >> > will just get to deal with that problem when and if it happens. >> >> In case some numbers help: the numerical community has been tracking the deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at this point what I hear is: >> - ~0.5% of Firefox crashes are on machines that are missing SSE2 >> - <0.1% of machines with Steam installed are missing SSE2 >> >> I'm not sure what python distributions like Anaconda or Activestate are doing wrt SSE2, but even if their builds do require SSE2 then their build tooling might provide a quick way to generate a whole installable environment with custom build options for targeting older systems. (Continuum as of today still hasn't released build scripts for the bottom of their stack -- python/numpy/etc. -- but they've claimed willingness to do so and there's increasing calls to make a "centos" version of Anaconda. And all the tooling beyond that -- e.g. the actual package manager -- is FOSS.) >> >> -n >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Oct 11 18:44:07 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 11 Oct 2015 18:44:07 +0200 Subject: [Distutils] warning about potential problem for wheels References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> Message-ID: <20151011184407.22503ba9@fsol> On Sun, 11 Oct 2015 08:07:30 -0700 Steve Dower wrote: > > This does only affect 32-bit builds, so now I'm thinking about the > possibility of treating those as highly compatible while the 64-bit > ones get better performance treatment, though I'm not sure how that > could actually play out. It may help remove some of the questions > about which one to use though. That sounds reasonable to me. I don't know Windows very much, but are there still many people using 32-bit Windows these days (on x86, I mean)? Regards Antoine. From oscar.j.benjamin at gmail.com Sun Oct 11 19:43:14 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 11 Oct 2015 17:43:14 +0000 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <20151011184407.22503ba9@fsol> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: On Sun, 11 Oct 2015 17:44 Antoine Pitrou wrote: On Sun, 11 Oct 2015 08:07:30 -0700 Steve Dower wrote: > > This does only affect 32-bit builds, so now I'm thinking about the > possibility of treating those as highly compatible while the 64-bit > ones get better performance treatment, though I'm not sure how that > could actually play out. It may help remove some of the questions > about which one to use though. That sounds reasonable to me. I don't know Windows very much, but are there still many people using 32-bit Windows these days (on x86, I mean)? I don't know but I think it makes sense to follow Windows' lead. So if 3.5 supports Vista and Vista doesn't require SSE2 then CPython shouldn't either. If 3.6 or whatever drops support for Vista and if Windows 7 requires SSE2 then CPython can require it too. I assume this what happens with the OSX binaries. Note that my recently retired computer was 64 bit and had SSE but didn't have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers are going to stop adding new instructions. So this general issue isn't limited to 32 bit hardware and won't be solved by special casing that. I think it makes sense to have a general policy for architectures that will be supported by the official build in future. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Oct 11 19:48:51 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 10:48:51 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > 2) after unpacking this sdist it then calls 'setup.py egg_info' to get > the full metadata for the wheel I wouldn't say "get the full metadata for the wheel". it's not a wheel yet. `egg_info` run so we can use the pkg_resources api to find the dependencies. > Specifically what it does with this is > extract the setup_requires and install_requires fields specifically, we call `requires` on pkg_resources distribution objects https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583 > 3) eventually it actually builds the package, and this produces a > wheel (or wheel equivalent) that has its own metadata (which often > matches the metadata from egg_info in step (2), but not always) > "not always"? not following your point they're 2 different formats, but should contain the same essential information. here's the wheel code that does the conversion https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90 > name and version before it runs egg_info, I assume that what this > means is that it's crucial for pip to have static access to dependency > information? yes > It would be somewhat convenient if sdists did list their binary > dependencies: not sure about your insertion of "binary" here. pip is concerned with finding python project dependencies (i.e. name and version constraints) in the sdist and then based on the current install environment, it will further constrain the wheels chosen based on architecture and python implementation. and to be perfectly clear, none of this deals with non-python OS/Distro requirements. > 3) if any of the packages-to-be-installed are sdists, then fetch them, > run egg_info or build them or whatever to get their real dependencies, > add these to the graph, and go to step 1 > this is the pain we don't want in the future. since we're simply talking about name/version constraints (not platform/python), It's hard to conceive that we'd agree on an sdist spec that didn't include that. > Do the relevant pip maintainers > even read this mailing list? :-) > I try : ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Oct 11 20:12:51 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 11:12:51 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > since we're simply talking about name/version constraints (not platform/python) let me clarify that... since I imagine someone might bring up markers in response... markers allow you to vary your name/version dependencies by environment (for example, by platform and python version), but to be clear, it's still not a "binary" dependency declaration... i.e. the dependency is still declared by name and a version specifier alone. On Sun, Oct 11, 2015 at 10:48 AM, Marcus Smith wrote: > > >> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get >> the full metadata for the wheel > > > I wouldn't say "get the full metadata for the wheel". it's not a wheel > yet. > `egg_info` run so we can use the pkg_resources api to find the > dependencies. > > > >> Specifically what it does with this is >> extract the setup_requires and install_requires fields > > > specifically, we call `requires` on pkg_resources distribution objects > https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583 > > > >> 3) eventually it actually builds the package, and this produces a >> wheel (or wheel equivalent) that has its own metadata (which often >> matches the metadata from egg_info in step (2), but not always) >> > > "not always"? not following your point > they're 2 different formats, but should contain the same essential > information. > here's the wheel code that does the conversion > > https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90 > > > >> name and version before it runs egg_info, I assume that what this >> means is that it's crucial for pip to have static access to dependency >> information? > > > yes > > > >> It would be somewhat convenient if sdists did list their binary >> dependencies: > > > not sure about your insertion of "binary" here. > pip is concerned with finding python project dependencies (i.e. name and > version constraints) in the sdist > and then based on the current install environment, it will further > constrain the wheels chosen based on architecture and python implementation. > and to be perfectly clear, none of this deals with non-python OS/Distro > requirements. > > > >> 3) if any of the packages-to-be-installed are sdists, then fetch them, >> run egg_info or build them or whatever to get their real dependencies, >> add these to the graph, and go to step 1 >> > > this is the pain we don't want in the future. > > since we're simply talking about name/version constraints (not > platform/python), It's hard to conceive that we'd agree on an sdist spec > that didn't include that. > > > >> Do the relevant pip maintainers >> even read this mailing list? :-) >> > > I try : ) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sun Oct 11 20:18:21 2015 From: steve.dower at python.org (Steve Dower) Date: Sun, 11 Oct 2015 11:18:21 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: The general rule is the minimum requirements of all supported operating systems, so in this case requiring SSE is a bug. Note that this doesn't prevent us from having SSE optimized code paths, as long as there is a fallback to one that is more compatible. And it's also not binding on redistributors or package builders, who are free to add more restrictive requirements. As far as identifying these sort of requirements in packages, I don't think they really make sense as part of a platform tag as a non-SSE build is fully compatible with an SSE build, but as part of version ordering it would be nice to be able to prefer certain tagged builds without excluding ones without the tag (e.g. if my platform is win32_sse2 and there's only a win32 tagged build, I still want it - how we formalize this is probably very complex though. Also, my platform tag is probably 100 chars long with everything I may want to represent...) Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Oscar Benjamin" Sent: ?10/?11/?2015 10:43 To: "Antoine Pitrou" ; "Distutils-Sig at python.org" Subject: Re: [Distutils] warning about potential problem for wheels On Sun, 11 Oct 2015 17:44 Antoine Pitrou wrote: On Sun, 11 Oct 2015 08:07:30 -0700 Steve Dower wrote: > > This does only affect 32-bit builds, so now I'm thinking about the > possibility of treating those as highly compatible while the 64-bit > ones get better performance treatment, though I'm not sure how that > could actually play out. It may help remove some of the questions > about which one to use though. That sounds reasonable to me. I don't know Windows very much, but are there still many people using 32-bit Windows these days (on x86, I mean)? I don't know but I think it makes sense to follow Windows' lead. So if 3.5 supports Vista and Vista doesn't require SSE2 then CPython shouldn't either. If 3.6 or whatever drops support for Vista and if Windows 7 requires SSE2 then CPython can require it too. I assume this what happens with the OSX binaries. Note that my recently retired computer was 64 bit and had SSE but didn't have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers are going to stop adding new instructions. So this general issue isn't limited to 32 bit hardware and won't be solved by special casing that. I think it makes sense to have a general policy for architectures that will be supported by the official build in future. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Oct 11 21:22:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 11 Oct 2015 20:22:47 +0100 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 11 October 2015 at 19:37, Robert Collins wrote: > 'pip install --only-binary :all: THINGTOINSTALL' will do exactly what > you want I think in that it will only download wheels. ooh, I didn't spot that getting added! Many thanks for that. Paul From robertc at robertcollins.net Sun Oct 11 22:13:53 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 09:13:53 +1300 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 12 October 2015 at 08:22, Paul Moore wrote: > On 11 October 2015 at 19:37, Robert Collins wrote: >> 'pip install --only-binary :all: THINGTOINSTALL' will do exactly what >> you want I think in that it will only download wheels. > > ooh, I didn't spot that getting added! Many thanks for that. > > Paul You're welcome :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Sun Oct 11 23:53:03 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 10:53:03 +1300 Subject: [Distutils] sampleproject: use tox? In-Reply-To: <5616CC7C.4060601@thomas-guettler.de> References: <5616CC7C.4060601@thomas-guettler.de> Message-ID: On 9 October 2015 at 09:05, Thomas G?ttler wrote: > This is a follow up to the thread "Where should I put tests when packaging python modules?" > > I have never used tox up to now. But reading the mails of the thread, it seems > that tox is the current state of the art. > > This leads me to the conclusion that the sample project should use tox. > > Any reasons not to do it? Yes, as already said - the sample project is limited in scope to packaging. But in the larger context of 'setting up a Python project from scratch - many folk have written opinionated guides - and if you were writing one of those, even if tox isn't consensus opinion, I'd say use it, so that new folk don't need to read a chunk of prose about choosing the right-thing-for-them. The challenge with education is making the learning curve approachable :) -rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Sun Oct 11 20:37:59 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 07:37:59 +1300 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 12 October 2015 at 00:49, Paul Moore wrote: > On 11 October 2015 at 05:31, Nathaniel Smith wrote: >> Do the relevant pip maintainers >> even read this mailing list? :-) > > Donald and I are pip maintainers (not the only ones, but probably the > ones who make the most noise on this list :-)). The comments I've made > that you quoted are based mostly on how I recall pip works, it's > unfortunately been some time since I looked into pip's dependency > resolution and install code (it's a bit of a pit of snakes) and I > haven't had time to check the details when commenting on this thread, > so I don't claim my comments are authoritative. > > I think your description of the behaviour is right. I have this on my to-read-threads list, just haven't had time to do it justice yet. FWIW I am *very* familiar with the dependency/install code at the moment, having overhauled it for: - wheel caching - working proof of concept static-dependencies-in-metadata handling - working proof of concept full resolver And various related bits. > ...The reason I want that is so that I would like to be able > to see what pip plans on doing, and audit it before it begins (a "pip > install --dry-run" type of step). Typically, this lets me avoid > getting part-way through an install only to find that I need a project > that doesn't have a wheel and I can't build from the sdist. That's a > real problem I deal with a lot - at the moment I work with "pip list > --outdated", check the listing to see what provides wheels, and update > just those packages individually. But that's messy and manual, and not > always 100% effective - and I'd like to be able to do better. 'pip install --only-binary :all: THINGTOINSTALL' will do exactly what you want I think in that it will only download wheels. -Rob From robertc at robertcollins.net Mon Oct 12 02:02:10 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 13:02:10 +1300 Subject: [Distutils] Where should I put tests when packaging python modules? In-Reply-To: <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006231424.GA30512@k3> <8C29AA9E-A988-4672-B6DA-C917951E76F1@wiggy.net> Message-ID: On 10 October 2015 at 21:47, Wichert Akkerman wrote: > Also having test code in the package can be very painful if you use tools > like venusian which scan and try to import all Python files. Even if tests are not in-package, such tools need to cope with extras in general: any optionally-installed dependencies will have this issue. Venusian offers a callback on import errors - http://venusian.readthedocs.org/en/latest/#onerror-scan-callback. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 12 02:19:26 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 13:19:26 +1300 Subject: [Distutils] pbr issues (was: Where should I put tests when packaging python modules?) In-Reply-To: References: Message-ID: On 8 October 2015 at 17:47, Nathaniel Smith wrote: > On Wed, Oct 7, 2015 at 3:39 PM, Robert Collins > wrote: > [...] >> Its all about easy-install. This is why I put a proof of concept >> static-setup-requires thing together for pip (and its on our teams >> roadmap to deliver a production version of it via patches to all the >> pypa projects, we're just not at that point yet - the resolver is >> first, and we have to finish rolling out constraints within OpenStack >> before that gets large timeslices). > > I remembered Nick saying something about this at PyCon, but I couldn't > find anything when I looked -- could you point me to the PoC? https://github.com/rbtcollins/pip/tree/issue-988 - resolver - its a little horked right now as I haven't fully addressed the constraints changes in the merge with it, but its readable. https://github.com/rbtcollins/pip/tree/declarative - declarative commits POC. Totally stale code but also worth reading (and not hard to update when the time comes) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 12 06:06:45 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 17:06:45 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: EWOW, huge thread. I've read nearly all of it but in order not to make it massively worse, I'm going to reply to all the points I think need raising in one mail :). Top level thoughts here, more point fashion with only rough editing below the fold. I realise many things - like the issue between different wheels of the same package consuming different numpy abis - have been touched on, but AFAICT they are entirely orthogonal to the proposal, which was to solve 'be able to use arbitrary build systems and still install with pip'. Of the actual problems with using arbitrary build systems, 99% of them seem to boil down to 'setup-requires isn't introspectable by pip (https://github.com/pypa/pip/issues/1820 ). - If it was, then alternative build systems could be depended on reasonably; and the mooted thunk from setuptools CLI to arbitrary build system would be viable. It is, in principle a matter of one patch to teach pip *a* way to do this (and then any and all build systems that want to can utilise it). https://github.com/rbtcollins/pip/tree/declarative is a POC I did - my next steps on that were to discuss the right ecosystem stuff for it - e.g. should pip consume it via setuptools, or should pip support it as *the way* and other systems including setuptools can choose to use it? A related but separate thing is being able to *exclusively* install things without setuptools present - I've filed https://github.com/pypa/pip/issues/3175 about that, but I think its -much- lower priority than reliably enabling third party build tools. -Rob ---- " solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. " Actually pip still punts a *lot* here - we have bypasses to let things like C compiler flags be set during wheel build, and when thats done we don't cache the wheels (or even try to build wheels). " While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: ... (c) you are forced to use them anyway, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``." I don't understand the claim of (c) here - its entirely possible to write a package that doesn't use setuptools and have it do the right thing - pip uses a subprocess to drive package installation, and the interface is documented. The interface might be fugly as, but it exists and works. It is missing setup-requires handling, but so is setup.py itself. The only thing we'd really need to do AFAICT is make our setuptools monkeypatching thunk handle setuptools not being installed (which would be a sensible thing to Just Do anyhow). " - query for build dependencies - run a build, producing wheels as output - set up the current source tree so that it can be placed on ``sys.path`` in "develop mode" " So we have that already. setup.py egg-info, setup.py bdist_wheel, setup.py develop. "A version 1-or-greater format source tree can be identified by the presence of a file ``_pypackage/_pypackage.cfg``. " I really don't like this. Its going to be with us forever, and its intrusive (its visible), and so far isn't shown to be fixing anything. "to scatter files around willy-nilly never works, so we adopt the convention that names starting with an underscore are reserved for official use, and non-underscored names are available for idiosyncratic use by individual projects." I can see the motivation here, but is it really solving a problem we have? On the specifics of the format: I don't want to kibbitz over strawman aspects at this point. Having the extension mechanism be both pip specific and in Python means that we're going to face significant adoption issues: the former because pip is not by any means the only thing around - and some distros have until very recently been actively hostile to pip (which in turn means we need to wait a decade or two for them to age-out and stop being used). The latter because we'll face all the headaches of running arbitrary untrusted code and dealing with two deps with different versions of the same hook and so on: I think its an intrinsically unsafe design. @dstufft "problem with numpy.distutils, as I know you?re aware!). We could do a minimal extension and add another defacto-ish standard of allowing pip and setuptools to process additional setup_requires like arguments from a setup.cfg to solve that problem though. The flip side to this is that since it involves new capabilities in pip/setuptools/any other installer is that it you?ll have several years until you can depend on setup.cfg based setup_requires from being able to be depended on. " Well. For *any* proposal that involves modifying pip, we have to assume that all existing things keep working, and that anyone wanting to utilise the new thing will have to either a) include a local compatibility thunk, or b) error when being used from a too-old toolchain. I don't think that should really be a factor in design since its intrinsic to the quagmire. "Longer term, I think the answer is sdist 2.0 which has proper metadata inside of it (name, version, dependencies, etc) but which also includes a hook like this PEP has to specify the build system that should be used to build a wheel out of this source distribution." Any reason that can't just be setup.cfg ? @Daniel "I thought Robert Collins had a working setup-requires implementation already? I have a worse but backwards compatible one too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py" - https://github.com/rbtcollins/pip/tree/declarative - I'll be updating that probably early next year at this rate - after issue-988 anyhow. The issue with your approach is that pip doesn't handle having concurrent installs done well - and in fact it will end up locking its environment somehow. @Paul " I can understand that a binary wheel may need a certain set of libraries installed - but that's about the platform tags that are part of the wheel definition, not about dependencies. Platform tags are an ongoing discussion, and a good example of a partial solution that" - thats where the draft PEP tennessee and I start is aimed - at making those libraries be metadata, not platform tags. @Chris " A given package might depend on numpy, as you say, and it may work with all numpy versions 1.6 to 1.9. Fine, so we specify that in install_requires. And this shodl be the dependency in the sdist, too. If the package is pur python, this is fine and done. But if the package has some extensions code that used the numpy C API ( a very common occurrence), then when it is built, it will only work (reliably) with the version of numpy it was built with. So the project itself, and the sdist depend on numpy >=1.6, but a build binary wheel depends on numpy == 1.7 (for instance). Which requires a binary (wheel) dependency that is somewhat different than the source dependency. " - so yes, that is where bdist_wheel should be creating different metadata for that wheel. The issue that arises is that we need unique file names so that they can coexist on PyPI or local archives - which is where wheel tags come in. I'd be in favour of not using semantic tags for this - rather hash the deps or something and just make a unique file name. Use actual metadata for metadata. @Nathaniel "I know that one unpleasant aspect of the current design is that the split between egg-info and actual building creates the possibility for time-of-definition-to-time-of-use bugs, where the final wheel hopefully matches what egg-info said it would, but in practice there could be skew. (Of course this is true in any system which represents" - actually see https://bugs.launchpad.net/pbr/+bug/1502692 for a bug where this 'skew' is desirable: for older environments we want tailored deps with no markers, for anything supporting markers we want them - so the wheel will have markers and egg_info won't. @Nathaniel " (Part of the intuition for the last part is that we also have a not-terribly-secret-conspiracy here for writing a PEP to get Linux wheels onto PyPI and at least achieve feature parity with Windows / OS X. Obviously there will always be weird platforms -- iOS and FreeBSD and Linux-without-glibc and ... -- but this should dramatically reduce the frequency with which people need sdist dependencies.)" - I think a distinction between sdist and binary names for dependencies would be a terrible mistake. It will raise complexity for reasoning and describing things without solving any concrete problem that I can see. @Nathaniel "I guess to make progress in this conversation I need some more detailed explanations. I totally get that there's a long history of thought and conversations behind the various assertions here like "a sdist is fundamentally different from a VCS checkout", "there must be a 1-1 mapping between sdists and wheels", "pip needs sdists that have full wheel metadata in static form", and I'm barging in from the outside with no context, but I literally have no idea why the specific design features you're asking for are desirable or even viable. Right now if I were to try and write the PEP you're asking for, then the rationale section would just be "because Donald said so" over and over :-). I couldn't write the motivation section, because I don't know any problems that the PEP you're describing would fix for me as a package author (which doesn't mean they don't exist, but!)." -- VCS trees are (generally) by-humans for humans. They are the primary source of data and can do thinks like inferring versions from commit data. sdists are derived from the VCS tree and can include extra data (such as statically defined version data). Wheels are derived from a tree on disk and can (today) be built from either VCS trees or sdists. I'm not sure that forcing an sdist step is beneficial - the egg-info step we have today is basically that without the cost of compressing and decompressing potentially large trees for no reason. @Jeremy "An sdist is an installable package which just happens to _look_ a lot like a source release tarball, but trying to pretend that downstream packagers will want to use it as such leads to a variety of pain points in the upstream/downstream relationship. For better or worse a lot of distros don't want generated files in upstream source code releases, since they need to confirm that they also ship the necessary tooling to regenerate any required files and that the generated files they ship match what their packaged tooling produces." - Well, pbr doesn't work if you just tar up or git export your VCS tree: it requires the chance to add metadata. And while distros have whinged about pbr in a number of contexts, that hasn't been one so far. Downstreams are pretty used to receiving tarballs with generated files in them - as long as they *have the option* to recreate those, so the source material isn't lost. [And for version data, 'grab from git' is a valid answer there']. OTOH perhaps ftpmaster just hasn't noticed and we're about to get a bug report ;) From robertc at robertcollins.net Mon Oct 12 06:37:36 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 17:37:36 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 17:06, Robert Collins wrote: > EWOW, huge thread. > > I've read nearly all of it but in order not to make it massively > worse, I'm going to reply to all the points I think need raising in > one mail :). And a bugfix :) - I didn't link to the docs for the build system interface we have today - https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 06:53:36 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 21:53:36 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 4:49 AM, Paul Moore wrote: [...] > As regards what pip could do, technically you are of course correct > (it's possible, it just needs someone willing to make the code > changes). I don't know, however, if the approach you're proposing fits > with how we currently envisage pip developing. Specifically, my > personal goal [1] is that we get to a point where pip can do all of > the dependency resolution steps, getting to a point where it knows > exactly what it will install, *before* it starts downloading packages, > running build steps, etc. Thanks for stating this so clearly. Unfortunately I just don't see any way this can possibly be achieved. It seems to unambiguously rule out the various "compromise" proposals from the other thread (e.g., your suggestion that packages would have to specify most dependencies statically, but would have the option of adding some extra ones at build time, would not accomplish the goal stated above). To accomplish this, it really is necessary that we be able to *exactly* predict the full dependency set (and also environment specifiers and potential external system requirements and ... -- anything that possibly affects whether a wheel is installable) before we download any packages or build any wheels. But as discussed at length in the other thread, it's a fact of life that the same source release may be configured in different ways that create different resulting dependencies. NumPy is one example of this, but it's hardly unusual -- pretty much any optional dependency on a C library works like this. And these kinds of issues will presumably only get more complicated if we start adding more complex dependency structures (e.g. "provides", per-package abi compatibility tracking, ...). Are you aware of any other systems that have accomplished anything like this? From a quick skim, it looks like .deb, .rpm, gentoo, freebsd ports, and ruby gems all allow for arbitrary code execution inside "source" packages when specifying dependencies, and none of the systems I looked at have the property that you're looking for. This doesn't prove that it's impossible, but... I do see one clear path to accomplish what you want: 1) enable linux wheels on pypi 2) build an autobuilder infrastructure for pypi 3) now that 95% of packages have wheels, flip the switch so that pip ignores sdists when auto-installing dependencies This strategy at least has the advantage that it only requires we do things that have been done before and we know are possible :-). And the alternative is -- what? As far as pip goes, it sounds like we all agree that there's a perfectly sound algorithm for solving the installation problem without access to static dependencies (i.e., wrapping the solve-then-build cycle in a loop); it's just that this leads to a bit more complicated code in pip. But pip clearly has to implement and debug this code no matter what, because we are committed to handling traditional sdists for some years yet. It seems like the best we can hope for is that if we impose these constraints on wheels, and we somehow manage to make this work at all (which seems to be a research problem), then we might eventually be able to replace some working code with some slightly simpler working code. (Plus whatever other benefits there are of having static dependency metadata on pypi, like interesting global static analyses of the package ecosystem.) I know "debian packagers think source wheels with static metadata would be great" was cited in the other thread as an example of an advantage, but note that for numpy, the dependencies in the configuration that debian will want to use are exactly *not* the dependencies in the configuration we'll put on pypi, because the pypi version wants to be self-contained inside the pypi ecosystem but the debian version wants to rely as much as possible on debian-distributed libraries. So even if numpy has a "source wheel" that has static dependency metadata, then this will be exactly what debian *can't* use; they'll need a traditional non-static-metadata source release instead. I dunno -- I'm sure there must exist some other ways forward that don't require dropping the dream of static dependencies. At one extreme, we had a birds-of-a-feature at SciPy this year on "the future of numpy", and the most vocal audience contingent was in favor of numpy simply dropping upstream support for pip/wheels/pypi entirely and requiring all downstream packages/users to switch to conda or building by hand. It sounds like a terrible idea to me. But I would find it easier to believe in the pypi/pip ecosystem if there were some concrete plan for how this all-static world was actually going to work, and that it wasn't just chasing rainbows. -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 12 07:05:01 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 22:05:01 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:48 AM, Marcus Smith wrote: > >> >> 2) after unpacking this sdist it then calls 'setup.py egg_info' to get >> the full metadata for the wheel > > > I wouldn't say "get the full metadata for the wheel". it's not a wheel yet. > `egg_info` run so we can use the pkg_resources api to find the dependencies. > > >> >> Specifically what it does with this is >> extract the setup_requires and install_requires fields > > > specifically, we call `requires` on pkg_resources distribution objects > https://github.com/pypa/pip/blob/develop/pip/req/req_set.py#L583 > > >> >> 3) eventually it actually builds the package, and this produces a >> wheel (or wheel equivalent) that has its own metadata (which often >> matches the metadata from egg_info in step (2), but not always) > > > "not always"? not following your point > they're 2 different formats, but should contain the same essential > information. > here's the wheel code that does the conversion > https://bitbucket.org/pypa/wheel/src/1cb7374c9ea4d5c82992f1d9b24cf7168ca87707/wheel/metadata.py?at=default&fileviewer=file-view-default#metadata.py-90 I just meant that due to the fact that you're running two chunks of arbitrary code, there's no way to statically guarantee that the metadata produced in the two calls to 'setup.py' is going to match. In the message here: https://mail.python.org/pipermail/distutils-sig/2015-October/027161.html I linked to some packages whose setup.py's actually report different dependency information in the egg_info and build phases. I'm not saying this is a good or bad or even particularly relevant thing, just observing that it's a fact about how the current system works :-). (And this is part of why my draft PEP that set all this off just has a single atomic "build a wheel" operation instead of splitting this into two phases.) > > >> >> name and version before it runs egg_info, I assume that what this >> means is that it's crucial for pip to have static access to dependency >> information? > > > yes > > >> >> It would be somewhat convenient if sdists did list their binary >> dependencies: > > > not sure about your insertion of "binary" here. > pip is concerned with finding python project dependencies (i.e. name and > version constraints) in the sdist > and then based on the current install environment, it will further constrain > the wheels chosen based on architecture and python implementation. > and to be perfectly clear, none of this deals with non-python OS/Distro > requirements. I didn't mean anything in particular by "binary", sorry if that threw you off. It is probably because I am still finding it useful to think of source packages and binary packages as logically distinct, i.e., a source package is a black box that can produce a binary package and when I have a dependency it names a binary package that I want (which can be obtained either by downloading it or by finding a source package that will produce that binary package and building it). >> 3) if any of the packages-to-be-installed are sdists, then fetch them, >> run egg_info or build them or whatever to get their real dependencies, >> add these to the graph, and go to step 1 > > > this is the pain we don't want in the future. > > since we're simply talking about name/version constraints (not > platform/python), It's hard to conceive that we'd agree on an sdist spec > that didn't include that. > > >> >> Do the relevant pip maintainers >> even read this mailing list? :-) > > > I try : ) Sorry, I realized that may have come across as much more negative than I intended :-). I just realized at the end of typing this whole thing that I wasn't even sure if I was sending it to the right place :-). -- Nathaniel J. Smith -- http://vorpus.org From qwcode at gmail.com Mon Oct 12 07:12:05 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 22:12:05 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > > > it's a fact of life that the same > source release may be configured in different ways that create > different resulting dependencies. NumPy is one example of this, but > it's hardly unusual I've tried to respond to this point twice.. but let me try again. : ) If I'm confused, please help me. aren't the "different resulting dependencies" non-python, non-pypi system dependencies? the dependencies the sdist would cover are the python/pypi dependencies, and they don't usually vary based on building extensions as for representing non-python distro dependencies and build variances, I mentioned before that PEP426 extensions might be used to cover that, but that's still an open question I think.. --Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Oct 12 07:32:10 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 22:32:10 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:12 PM, Marcus Smith wrote: >> >> it's a fact of life that the same >> source release may be configured in different ways that create >> different resulting dependencies. NumPy is one example of this, but >> it's hardly unusual > > > I've tried to respond to this point twice.. but let me try again. : ) > If I'm confused, please help me. > > aren't the "different resulting dependencies" non-python, non-pypi system > dependencies? > > the dependencies the sdist would cover are the python/pypi dependencies, and > they don't usually vary based on building extensions > > as for representing non-python distro dependencies and build variances, I > mentioned before that PEP426 extensions might be used to cover that, but > that's still an open question I think.. Ah, sorry, there was some discussion of this buried deep inside the giant thread :-) For NumPy/SciPy/scikit-learn/etc. we have the problem that we need to somehow distribute the standard linear algebra libraries like BLAS and LAPACK. For e.g. debian packages, or a self-built version on Linux, these are generally treated as distro dependencies -- the user is responsible for making sure that they're (somehow) installed, and then we link to them directly. For wheels, though, we can't assume this -- not just because there's no spec for distro dependencies, but because even if there would then it'd probably be fragile (no guarantee that different distros have compatible ABIs) and it certainly wouldn't work on Windows or OS X. So instead, the current plan is that we're going to drop the libraries inside a wheel and upload it to PyPI: https://mail.python.org/pipermail/distutils-sig/2015-October/027164.html and our wheels on pypi will have an install_requires on this special library wheel. Compared to PEP426-extensions idea, this has the advantages that (a) it doesn't require a PEP, (b) there's no question about whether it will work :-). Another example of where this might arise is if a package [1] wants to optionally use dynd as its array backend instead of numpy [2], so it would need to optionally link to the dynd C API instead of or in addition to the numpy API. But dynd is a young and rapidly changing project, so you might want to support this as an experimental build configuration while still uploading "official" wheels that depend on numpy only. But the configuration that does link to dynd would still want to fetch dynd using python-level dependencies [3]. Does that help? -n [1] This isn't terribly hypothetical, pandas is one of the most popular data processing libraries in the world and they have an experimental branch doing exactly this: https://github.com/pydata/pandas/issues/8643 [2] https://github.com/libdynd/libdynd [3] https://pypi.python.org/pypi/dynd -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 12 07:36:37 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 22:36:37 -0700 Subject: [Distutils] Sources of truth Message-ID: Hi all, Again trying to split out some more focused discussion from the big thread about sdists... One big theme there has been the problem of "sources of truth": e.g. in current sdists, there is a PKG-INFO file that has lots of static metadata in it, but because the "real" version of that metadata is in setup.py, everyone ignores PKG-INFO. A clear desideratum for a new sdist format is that we avoid this problem, by having static metadata that is actually trustworthy. I see two fundamentally different strategies that we might use to accomplish this. In time honored mailing list tradition, these are of course the one that I hear other people advocating and the one that I like ;-). The first strategy is: sdists and the wheels they generate logically share the same metadata; so, we need some mechanism to enforce that whatever static metadata is in the sdist will match the metadata in the resulting wheel. (The wheel might potentially have additional metadata beyond what is in the sdist, but anything that overlaps has to match.) An open question is what this mechanism will look like -- if everyone used distutils/setuptools, then we could write the code in distutils/setuptools so that when it generated wheel metadata, it always copied it directly out of the sdist metadata (when present). But not everyone will use distutils/setuptools, because distutils delenda est. So we need some mechanism to statically analyze an arbitrary build system and prove things about the data it outputs. Which sounds... undecideable. Or we could have some kind of after-the-fact enforcement mechanism, where tools like pip are required -- as the last step when building a wheel from an sdist -- to double-check that all the metadata matches, and if it doesn't then they produce a hard error and refuse to continue. But even this wouldn't necessarily guarantee that PyPI can trust the metadata, since PyPI is not going to run this enforcement mechanism... The second strategy is: put static metadata in both sdists and wheels, but treat them as logically distinct things: the static metadata in sdists is the source of truth for information *about that sdist* (sdist name, sdist version, sdist description, sdist authors, etc.), and the static metadata in wheels is the source of truth for information about that wheel, but we think of these as distinct things and don't pretend that we can statically guarantee that they will match. I mean, in practice, they basically always will match. But IMO making this distinction in our minds leads to clearer thinking. When PyPI needs to know the name/version/description for an sdist, it can still do that; and since we've lowered our ambitions to only finding the sdist name instead of the wheel name, it can actually do it reliably in a totally static way, without having to run arbitrary code to validate this. OTOH pip will always have to be prepared to handle the possibility of mismatch between what it was expecting based on the sdist metadata and what it actually got after building it, so we might as well acknowledge that in our mental model. One potential advantage of this approach is that we might be able to talk ourselves into trusting the existing PKG-INFO as providing static metadata about the sdist, and thus PyPI at least could start trusting it for things like the "description" field, and if we define a new sdist format then it would be possible to generate its static metadata from current setup.py files (e.g. by modifying setuptools's sdist command). Contrast this with the other approach, where getting any kind of static source-of-truth would require rewriting almost all existing setup.py files. The challenge, of course, is that there are a few places where pip actually does need to know something about wheels based on examining an sdist -- in particular name and version and (controversially) dependencies. But this can/should be addressed explicitly, e.g. by writing down a special rule about the name and version fields. -n -- Nathaniel J. Smith -- http://vorpus.org From donald at stufft.io Mon Oct 12 07:49:47 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 12 Oct 2015 01:49:47 -0400 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On October 12, 2015 at 12:54:07 AM, Nathaniel Smith (njs at pobox.com) wrote: I dunno -- I'm sure there must exist some other ways forward that? don't require dropping the dream of static dependencies. At one? extreme, we had a birds-of-a-feature at SciPy this year on "the future? of numpy", and the most vocal audience contingent was in favor of? numpy simply dropping upstream support for pip/wheels/pypi entirely? and requiring all downstream packages/users to switch to conda or? building by hand. It sounds like a terrible idea to me. But I would? find it easier to believe in the pypi/pip ecosystem if there were some? concrete plan for how this all-static world was actually going to? work, and that it wasn't just chasing rainbows.? FTR, I plan on making some sort of auto builder for PyPI so it?s possible that we can get pip to a point where almost all things it downloads are binary wheels and we don?t need to worry too much about needing to optimize the sdist case. I also think that part of the problem with egg-info and setup.y?s and pip?s attempted use of them is just down to the setup.py interface being pretty horrible and setup.py?s serving too many masters where it needs to be both VCS entry point, sdist metadata entry point, installation entry point, and wheel building entry point. I also think that it would be a terrible idea to have the science stack leave the ?standard? Python packaging ecosystem and go their own way and I think it?d make the science packages essentially useless to the bulk of the non science users. I think a lot of the chasing rainbows like stuff comes mostly from: We have some desires from our experiences but we haven?t yet taken the time to fully flesh out what the impact of those desires are, nor are any of us science stack users (or contributors) to my knowledge, so we don?t deal with the complexities of that much [1]. One possible idea that I?ve thought of here, which may or may not be a good idea: Require packages to declare up front conditional dependencies (I?m assuming the list of dependencies that a project *could* have is both finite and known ahead of time) and let them give groups of these dependencies a name. I?m thinking something similar to setuptools extras where you might be able to put a list of dependencies to a named group. The build interface could include a way for the thing that?s calling the build tool to say ?I require the feature represented by this named group of dependencies?[2], and then the build tool can hard fail if it detects it can?t be build in a way that requires those dependencies at runtime. When the final build is done, it could put into the Wheel a list of all of the additional named groups of dependencies it built with. The runtime dependencies of a wheel would then be the combination of all of those named groups of dependencies + the typical install_requires dependencies. This could possibly even be presented nicely on PyPI as a sort of ?here are the extra features you can get with this library, and what that does to it?s dependencies?. Would something like that solve Numpy?s dependency needs? Or is it the case that you don?t know ahead of time what the entire list of the dependency specifiers could be (or that it would be unreasonable to have to declare them all up front?). I think I recall someone saying that something might depend on something like ?Numpy >= 1.0? in their sdist, but once it?s been built then they?ll need to depend on something like ?Numpy >= $VERSION_IT_WAS_BUILT_AGAINST?. If this is something that?s needed, then we might not be able to satisfy this particular thing. I think I should point out too, that I?m not dead set on having static dependency information inside of a sdist/source wheel/whatever. What I am dead set on having is that all of the metadata *inside* of the sdist/source wheel/whatever should be static and it should include as much as possible which isn?t specific to a particular wheel. This means that things like name, version, summary, description, project URLs, etc. these are obviously (to me anyways) not going to be specific to a wheel and should be kept static inside of the sdist (and then copied over to resulting wheel as static as well [3]) and you simply can?t get information out of a sdist that is inherent to wheels. Obvious (again, to me) examples of data like that are things like build number, ABI the wheel was compiled against, etc. Is the list of runtime dependencies one of the things that are specific to a particular wheel? I don?t know, it?s not obvious to me but maybe it is. [4] [1] That being said, I?d love it if someone who does deal with this things would become more involved with the ?standard? ecosystem so we can have experts who deal with that side of things as well, because I do think it?s an important use case. [2] And probably: ?Even if you can do the feature suggested by this named group of dependencies, I don?t want it? [3] This rule should be setup so that we can have an assertion put into place that this data will remain the exact same when building a wheel from a sdist. [4] I think there?s possibly some confusion in what is causing problems. I think that the entirety of ``setup.py`` is full of problems with executing in different environments and isn?t very well designed to enable robust packages. A better designed interface would likely resolve a good number of the problems that pip currently has either way. It is true however that needing to download the sdist prior to resolving dependencies makes things a lot slower, but I also think that doing it correctly is more important than doing it quickly. We could possibly resolve some of this by pushing more people to publish Wheels, expanding the cases where Wheels are possible to be uploaded, and creating a wheel builder service. These things could make it possible to reduce the situations where you *need* to download + build locally to do a dependency resolution. I think that something like the pip wheel building cache can reduce the need for this as well, since we?ll already have built wheels available locally in the wheel cache that we can query for dependency information.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Mon Oct 12 07:55:03 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 11 Oct 2015 22:55:03 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: > > > So instead, the current plan is that we're going > to drop the libraries inside a wheel and upload it to PyPI: > aha... ok, now it's clearer where you're coming from. but using what platform in the wheel tag? linux wheels are blocked currently regardless of the tagging though, I'm not so sure about using PyPI like this to hold all these "python" wrapper packages for binary extensions -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Oct 12 08:14:43 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 11 Oct 2015 23:14:43 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:55 PM, Marcus Smith wrote: >> >> So instead, the current plan is that we're going >> to drop the libraries inside a wheel and upload it to PyPI: > > aha... ok, now it's clearer where you're coming from. > but using what platform in the wheel tag? > linux wheels are blocked currently Windows is the first goal, likely OS X after that (OS X provides a bit more in the base system so distributing these external dependencies isn't as urgent, but it would be good eventually), and yeah, doing this for Linux would obviously require unblocking Linux wheels. (We do have a plan for unblocking them, but the critical person is a bit busy so I'm not sure how long it will be before it can be unveiled :-). It's not really a secret, the plan is just: take advantage of Continuum and Enthought's hard-won knowledge from distributing hundreds of thousands of pre-built python package binaries on Linux to define a named "common Linux platform" abi tag that is known to be abi-compatible with >99.99% of Linux installs out there. This doesn't prevent the use of other abi tags -- e.g. distro-specific ones -- or the development of more detailed metadata about "distro dependencies", and wouldn't require that every single Linux box be willing to install wheels with this abi tag, but it would at least bring Linux support up to parity with the other platforms.) > regardless of the tagging though, I'm not so sure about using PyPI like this > to hold all these "python" wrapper packages for binary extensions If someone else invents a better solution before this is implemented then of course we'd switch to that instead... -n -- Nathaniel J. Smith -- http://vorpus.org From robertc at robertcollins.net Mon Oct 12 08:00:51 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 19:00:51 +1300 Subject: [Distutils] Sources of truth In-Reply-To: References: Message-ID: On 12 October 2015 at 18:36, Nathaniel Smith wrote: > Hi all, > > Again trying to split out some more focused discussion from the big > thread about sdists... > > One big theme there has been the problem of "sources of truth": e.g. > in current sdists, there is a PKG-INFO file that has lots of static > metadata in it, but because the "real" version of that metadata is in > setup.py, everyone ignores PKG-INFO. > > A clear desideratum for a new sdist format is that we avoid this > problem, by having static metadata that is actually trustworthy. I see > two fundamentally different strategies that we might use to accomplish > this. In time honored mailing list tradition, these are of course the > one that I hear other people advocating and the one that I like ;-). > The first strategy is: sdists and the wheels they generate logically > share the same metadata; so, we need some mechanism to enforce that This is false: they don't share the same metadata. Some portions are the same, but deps, supported platforms, those will differ (and perhaps more than that). In particular, an sdist doesn't have a dependency on an ABI, and a wheel doesn't have a dependency on an API. Some APIs are ABIs (approximately true for all pure Python packages, for instance), but some are not (numpy). > The second strategy is: put static metadata in both sdists and wheels, > but treat them as logically distinct things: the static metadata in > sdists is the source of truth for information *about that sdist* > (sdist name, sdist version, sdist description, sdist authors, etc.), > and the static metadata in wheels is the source of truth for > information about that wheel, but we think of these as distinct things > and don't pretend that we can statically guarantee that they will > match. I mean, in practice, they basically always will match. The analgous current data won't match for pbr using packages when we fix https://bugs.launchpad.net/pbr/+bug/1502692 (older pip's don't support PEP-426 environment markers, but don't error when they are used either, leading to silent failure to install dependencies). Now, you might say 'hey, but the new shiny will support markers from day one'. Well the problem is backwards compat: we're going to have future things that change, and the more we split things out the more the changes are likely to need skewed results like this approach to deal with it. ... > the sdist name instead of the wheel name, it can actually do it but the sdist and the wheel have to have the same name- or do you mean the filename on disk, vs the distribution name? > reliably in a totally static way, without having to run arbitrary code > to validate this. OTOH pip will always have to be prepared to handle > the possibility of mismatch between what it was expecting based on the > sdist metadata and what it actually got after building it, so we might > as well acknowledge that in our mental model. > > One potential advantage of this approach is that we might be able to > talk ourselves into trusting the existing PKG-INFO as providing static > metadata about the sdist, and thus PyPI at least could start trusting > it for things like the "description" field, and if we define a new The challenge is the 40K broken packages up there on PyPI. Basically pip has a bugfix for any of: sdists built using distutils sdists built using random build systems that don't understand what an sdist is (e.g. automake) sdists built using versions of setuptools that had a bug in this area There is no corrective mechanism for broken packages other than route-around-it-while-you-ask-the-author-to-upload-a-fix. So I think to tackle the 'please trust the metadata in the sdist' problem, one needs to have a graceful ramp-up of that trust with robust backoff mechanisms that don't involve 50% of PyPI users hating on that one old project in the corner everyone has a dep on but that is actually moribund and not doing uploads. I can imagine several such routes, including a crowdsourced blacklist - but its going to be (like we're dealing with with the automatic wheel cache already) years of bug reports until things age out. > sdist format then it would be possible to generate its static metadata > from current setup.py files (e.g. by modifying setuptools's sdist > command). Contrast this with the other approach, where getting any > kind of static source-of-truth would require rewriting almost all > existing setup.py files. We already generate static metadata from current setup.py files: setup.py egg_info does precisely that. There, bug fixed ;). > The challenge, of course, is that there are a few places where pip > actually does need to know something about wheels based on examining > an sdist -- in particular name and version and (controversially) > dependencies. But this can/should be addressed explicitly, e.g. by > writing down a special rule about the name and version fields. I'm sorry, I don't follow. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 09:23:46 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 00:23:46 -0700 Subject: [Distutils] Sources of truth In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 11:00 PM, Robert Collins wrote: > On 12 October 2015 at 18:36, Nathaniel Smith wrote: [...] >> the sdist name instead of the wheel name, it can actually do it > > but the sdist and the wheel have to have the same name- or do you mean > the filename on disk, vs the distribution name? I mean the distribution name - there's no way to guarantee that building foo-1.0.zip won't spit out bar-7.4.whl, where by "no way" I mean "it's literally undecideable". I mean, if someone actually did this it would be super weird and we would all shun them, but our code and specs still need to be prepared for the possibility. IIUC this is why PyPI can't trust PKG-INFO: 99.9% of the time the metadata in PKG-INFO matches what you will get when you run setup.py, but right now PyPI wants to know what setup.py will do, and there's no way to know if it will be the same as what PKG-INFO says, so it just doesn't trust PKG-INFO. OTOH if we redefine PyPI's goal as being, figure out what's in PKG-INFO (or whatever replaces it), and declare that it's okay (for PyPI's purposes) if that doesn't match what the build system will eventually do, then that's a viable way forward. >> reliably in a totally static way, without having to run arbitrary code >> to validate this. OTOH pip will always have to be prepared to handle >> the possibility of mismatch between what it was expecting based on the >> sdist metadata and what it actually got after building it, so we might >> as well acknowledge that in our mental model. >> >> One potential advantage of this approach is that we might be able to >> talk ourselves into trusting the existing PKG-INFO as providing static >> metadata about the sdist, and thus PyPI at least could start trusting >> it for things like the "description" field, and if we define a new > > The challenge is the 40K broken packages up there on PyPI. Basically > pip has a bugfix for any of: > sdists built using distutils > sdists built using random build systems that don't understand what an > sdist is (e.g. automake) > sdists built using versions of setuptools that had a bug in this area > > There is no corrective mechanism for broken packages other than > route-around-it-while-you-ask-the-author-to-upload-a-fix. IIUC what PyPI wants to do with PKG-INFO is read out stuff like the description and trove classifiers fields. Are there really 40K sdists on PyPI that have PKG-INFO files and where those files contain incorrect descriptions and so forth? I mean, obviously someone would have to check :-) But it seems unlikely, since almost everyone uploads by running 'sdist upload' or twine or something similarly automated. > So I think to tackle the 'please trust the metadata in the sdist' > problem, one needs to have a graceful ramp-up of that trust with > robust backoff mechanisms that don't involve 50% of PyPI users hating > on that one old project in the corner everyone has a dep on but that > is actually moribund and not doing uploads. I can imagine several such > routes, including a crowdsourced blacklist - but its going to be (like > we're dealing with with the automatic wheel cache already) years of > bug reports until things age out. > >> sdist format then it would be possible to generate its static metadata >> from current setup.py files (e.g. by modifying setuptools's sdist >> command). Contrast this with the other approach, where getting any >> kind of static source-of-truth would require rewriting almost all >> existing setup.py files. > > We already generate static metadata from current setup.py files: > setup.py egg_info does precisely that. There, bug fixed ;). I'm pretty sure that merely making it so 'setup.py sdist' created a file that contained the output from egg_info would not solve the current problem. That's pretty much exactly what the existing PKG-INFO *is*, isn't it? Yet apparently no-one trusts it. >> The challenge, of course, is that there are a few places where pip >> actually does need to know something about wheels based on examining >> an sdist -- in particular name and version and (controversially) >> dependencies. But this can/should be addressed explicitly, e.g. by >> writing down a special rule about the name and version fields. > > I'm sorry, I don't follow. E.g., we can document that if you have a sdist foo-1.0, then pip and similar tools will expect this to generate a foo-1.0 wheel (but be prepared to do something sensible if this doesn't happen, like give an error message or whatever). That's really all pip needs, right? -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 12 10:24:35 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 01:24:35 -0700 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On Sun, Oct 11, 2015 at 10:49 PM, Donald Stufft wrote: > FTR, I plan on making some sort of auto builder for PyPI so it?s possible > that we can get pip to a point where almost all things it downloads are > binary wheels and we don?t need to worry too much about needing to optimize > the sdist case. That would be 1000% awesomesauce. > I also think that part of the problem with egg-info and > setup.y?s and pip?s attempted use of them is just down to the setup.py > interface being pretty horrible and setup.py?s serving too many masters > where it needs to be both VCS entry point, sdist metadata entry point, > installation entry point, and wheel building entry point. Yeah, I'm glad that Robert linked https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface because I hadn't seen it, and it's great that it's documented, but what reading that documentation mostly convinced me is that I definitely could not write a new implementation of that interface and get it right without significant pain and trial-and-error :-). I mean... you have to support both --single-version-externally-managed enabled and disabled? develop has to know the right arcane magic to get the directory added to sys.path? (how do you know where in sys.path to add the magic?) what is egg_info even supposed to do -- the linked to docs don't actually define what egg metadata looks like, and IIRC it's not standardized, is it? (as opposed to the dist-info used in wheels) Like, obviously all that should be needed to implement a new build system is the ability to generate a wheel, right? Wheels are well-documented and one honking great idea. My current best guess at where we want to end up with sdists is: - there's a static file like the _pypackage/_pypackage.cfg in my original proposal (ignore the name, we can change it, I don't care) - it's written in some simple well-specified language like json or toml (if you look at the spec, toml is basically json encoded in an INI-style syntax -- it's definitely nowhere near as complex as yaml) - it includes static metadata for name, version (required for pypi uploads, optional otherwise to accomodate VCS checkouts), description, long_description, authors, URLs, trove, etc. etc., but not install_depends - it also includes the setup-dependencies / setup-dependencies-dynamic / build-wheel / build-in-place entries/hooks from my original proposal (or the moral equivalent) - it lives inside the single top-level directory next to the source code (instead of having a meta/ + source/ split, because that way it works for VCS checkouts too and AFAICT there aren't any other advantages to this split. We can afford to steal a single name -- everyone use used to having a setup.py or Makefile or Gemfile or whatever cluttering up the top-level of their source, and this is just a replacement for that.) That gives you static metadata for PyPI and solves the setup_depends problem while simultaneously cleaning up the "fugly" setup.py interface so that it becomes easy to write your own build system. (I think instead of just arguing about tests, we should have big debates here about which of the 3-4 highly-regarded competing build systems we should be recommending to newbies. But this will only happen if we make designing a new build system something that dozens of people do and a few take seriously, which requires making it accessible.) I *think* the above interface also makes it possible to write a shim in either direction (a setup.py that speaks the current "fugly" interface to old versions of pip while using the new files to actually do its work, or a new-style sdist that delegates all the actual work to setup.py), and even for setuptools to start generating sdists that are simultaneously both old-style and new-style for compatibility. So I think we can at least solve the metadata-for-PyPI problem, the setup_requires problem, and the let-a-thousand-build-systems-bloom problems now, without solving the static-install_requires-in-sdists problem (but also without creating any particular obstacles to solving it in the future, if it's even possible). And in the mean time we can keep working on autobuilders and linux wheels that will remove much of the demand for automagic sdist building -- it's difficult to make predictions, especially about the future, so who knows, maybe in the end we'll decide we just don't care that much about static-install_requires-in-sdists. > I also think that it would be a terrible idea to have the science stack > leave the ?standard? Python packaging ecosystem and go their own way and I > think it?d make the science packages essentially useless to the bulk of the > non science users. I think a lot of the chasing rainbows like stuff comes > mostly from: We have some desires from our experiences but we haven?t yet > taken the time to fully flesh out what the impact of those desires are, nor > are any of us science stack users (or contributors) to my knowledge, so we > don?t deal with the complexities of that much [1]. > > One possible idea that I?ve thought of here, which may or may not be a good > idea: > > Require packages to declare up front conditional dependencies (I?m assuming > the list of dependencies that a project *could* have is both finite and > known ahead of time) and let them give groups of these dependencies a name. > I?m thinking something similar to setuptools extras where you might be able > to put a list of dependencies to a named group. The build interface could > include a way for the thing that?s calling the build tool to say ?I require > the feature represented by this named group of dependencies?[2], and then > the build tool can hard fail if it detects it can?t be build in a way that > requires those dependencies at runtime. When the final build is done, it > could put into the Wheel a list of all of the additional named groups of > dependencies it built with. The runtime dependencies of a wheel would then > be the combination of all of those named groups of dependencies + the > typical install_requires dependencies. This could possibly even be presented > nicely on PyPI as a sort of ?here are the extra features you can get with > this library, and what that does to it?s dependencies?. > > Would something like that solve Numpy?s dependency needs? The answer is, I don't know! I am not at all confident in my ability to anticipate every possible difficult case, and NumPy is really not the only complicated library out there, it just gets most of the press. :-) Your idea definitely sounds like something worth exploring as part of the general debate around things like "Provides:" and extras, but I'd be hesitant to commit to locking something in right now. Also: > Or is it the case > that you don?t know ahead of time what the entire list of the dependency > specifiers could be (or that it would be unreasonable to have to declare > them all up front?). I think I recall someone saying that something might > depend on something like ?Numpy >= 1.0? in their sdist, but once it?s been > built then they?ll need to depend on something like ?Numpy >= > $VERSION_IT_WAS_BUILT_AGAINST?. If this is something that?s needed, then we > might not be able to satisfy this particular thing. Yes, it is true right now that packages that use the NumPy C API should have a wheel dependency on Numpy >= $VERSION_IT_WAS_BUILT_AGAINST. We'd like it if wheels/pip had even more flexible metadata in this regard that could express things like "package A needs numpy ABI version 3, and numpy 1.8, 1.9, and 1.10 say that they can provide ABI version 3, but 1.11 and 1.7 don't", but we shouldn't worry about the exact details of that kind of system right now -- I'm thinking any more flexible solution to the NumPy ABI problem would be at least as problematic for pip's sdist resolution rules as the current situation, and it sounds like the current situation is already bad enough to rule out a lot of cases. -n -- Nathaniel J. Smith -- http://vorpus.org From robertc at robertcollins.net Mon Oct 12 10:49:10 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 21:49:10 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 19:23, Ralf Gommers wrote: > > > On Mon, Oct 12, 2015 at 6:37 AM, Robert Collins > wrote: >> >> On 12 October 2015 at 17:06, Robert Collins >> wrote: >> > EWOW, huge thread. >> > >> > I've read nearly all of it but in order not to make it massively >> > worse, I'm going to reply to all the points I think need raising in >> > one mail :). >> >> And a bugfix :) - I didn't link to the docs for the build system >> interface we have today - >> >> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > > > From that link: > """ > In order for pip to install a package from source, setup.py must implement > the following commands: > ... > The install command should implement the complete process of installing the > package to the target directory XXX. > """ > That just sounds so wrong. You want the build system to build, not install. Right so - with the automatic wheel cache we added, that is only used if building a wheel failed. So you can consider it to be legacy cruft - the preferred interface is build-a-wheel then install-that-wheel. > And if "install" actually means "build to a tempdir so pip can copy it over > it to its final location", then how does that address something like > installing docs to a different dir than the package itself? Well, how do we install docs to a different dir with wheels? If we've got an answer for that, I think we're in a good position to figure something out for install (even if that is 'fix your package so it can build wheels). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Mon Oct 12 11:53:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Oct 2015 10:53:33 +0100 Subject: [Distutils] Sources of truth In-Reply-To: References: Message-ID: On 12 October 2015 at 08:23, Nathaniel Smith wrote: > I mean the distribution name - there's no way to guarantee that > building foo-1.0.zip won't spit out bar-7.4.whl, where by "no way" I > mean "it's literally undecideable". I mean, if someone actually did > this it would be super weird and we would all shun them, but our code > and specs still need to be prepared for the possibility. But the whole point of a spec is to *make* that guarantee. Pip doesn't expect to ever work with random code that can do literally anything, that's silly. Pip works with code that conforms to a spec. At the moment, the spec is "it does what setuptools/distutils does", which is a lousy spec because (a) nobody else can implement the spec without reading the huge mess that is distutils/setuptools, (b) it's too broad (we don't *really* need every corner case that distutils covers) and (c) distutils/setuptools don't satisfy the needs of a number of communities, notably the scientific community. So we want to define a more usable spec for the future. Pip will always have to deal with backward compatibility, and all of the hacks that implies, but we can at least say "if a project declares that it follows the new spec, we can rely on certain things". One of those things we can rely on is that building foo-1.0.zip won't spit out bar-7.4.whl - that would be part of the contract of that spec. The debate is over on the one hand, we want to be able to rely on when writing packaging tools, vs what flexibility we want to have when writing build tools. But it's not over ignoring the other use cases, it's about agreeing a *contract* that satisfies the needs of *both* packaging tools and build tools. At the moment, the dial is 100% over to the packaging side - builders have zero flexibility, the rule is "use distutils and setuptools". That doesn't feel like zero flexibility, because distutils/setuptools let you do a lot of things. But when you do finally hit the limits you're stopped cold. It also is a bad choice for packaging tools, because distutils and setuptools are a dreadful API for automation. So nobody wins. Two suggestions that have been around for a long while that give more flexibility to build tools while at the same time giving packaging tools a better interface to work with are: 1. Declarative setup - you have to put your metadata in file X in format Y, and it's static. But when you do, your build tool can do whatever it wants as long as it spits out binaries that confirm to your declared metadata. The downside to this is that it's totally static, and developers don't like that ("what if I want to generate my version number from my VCS tags?") 2. A fixed API - we document a set of commands that packaging tools can use and build tools have to provide. This is where we talk about documenting the distutils commands that pip relies on ("setup.py egg-info", "setup.py install", "setup.py develop"...) This one usually falls down because nobody likes the idea of writing a "dummy" setup.py that translates the interface, and because no-one has done anything more with this idea than suggest documenting the setuptools commands that pip *currently* uses, even though these are probably a lousy API design. It also does very little for packaging tools *other* than pip (for example, it does nothing for PyPI, which cannot accept an API that requires running user-supplied code). The key drivers here have been about defining something that packaging tools can use effectively, and build tools can cope with being required to provide. While not constraining *either* type of tool beyond the minimum needed for the contract. Paul From p.f.moore at gmail.com Mon Oct 12 12:06:30 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Oct 2015 11:06:30 +0100 Subject: [Distutils] What metadata does pip actually need about sdists? In-Reply-To: References: Message-ID: On 12 October 2015 at 07:14, Nathaniel Smith wrote: > On Sun, Oct 11, 2015 at 10:55 PM, Marcus Smith wrote: >>> >>> So instead, the current plan is that we're going >>> to drop the libraries inside a wheel and upload it to PyPI: >> >> aha... ok, now it's clearer where you're coming from. >> but using what platform in the wheel tag? >> linux wheels are blocked currently > > Windows is the first goal, likely OS X after that (OS X provides a bit > more in the base system so distributing these external dependencies > isn't as urgent, but it would be good eventually), and yeah, doing > this for Linux would obviously require unblocking Linux wheels. I'm running out of time to keep up with this mega-thread, so I'm starting to only skim at this point. Apologies if I've missed something that has already been said as a result... My view is that if you distribute a library (e.g., openblas) in a wheel, then it's essentially a "Python dependency" at that point, insofar as it can participate in the standard package dependency process in exactly the same way as (say) numpy itself does. So if you use that approach, I no longer see *any* reason why you can't work within the existing dependency process. Is that right? > We do have a plan for unblocking them, but the critical person is a > bit busy so I'm not sure how long it will be before it can be unveiled > :-). Obviously, as Marcus said, you need Linux wheels unblocked, and that means being able to specify compatibility cleanly, but as you say here, you're looking at that. Paul From robertc at robertcollins.net Mon Oct 12 12:01:53 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 12 Oct 2015 23:01:53 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 21:15, Thomas Kluyver wrote: > On 12 October 2015 at 05:37, Robert Collins > wrote: >> >> And a bugfix :) - I didn't link to the docs for the build system >> interface we have today - >> >> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > > > I'm happy to see that the command line interface pip requires from setup.py > is now documented. But the first thing it describes is the egg_info command, > and the description of what that involves is basically 'whatever setuptools > does'. Egg info is not any kind of standard, AFAIK - unlike dist info. One https://www.python.org/dev/peps/pep-0314/ describes PKG_INFO, which can in principle include dependency data. I don't know why they're not being fully populated - I suspect its another case of forward-looking-spec-not-quite-right-and-thus-never-used. Someone with more history on Python packaging will have to fill that in. One option would be to push on that. > of my main goals in writing flit is to build wheels without involving > setuptools at all, so I'm probably never going to implement that. Well thats fair enough. but: pip uses pkg_resources to find out whats installed and what it depends on, and that pulls out of egg-info on the file system for non-wheel-installs (dist-info for wheel installs). So, if that isn't well enough documented, it would be entirely reasonable to make a PEP to capture it so that you can emit it with confidence. There's already buy-in from everyone in the ecosystem to use PEPs to document interoperability impacting changes across these tools, so I can't imagine it being a contentious proposal. > I also don't want to have a setup.py in the VCS at all, because it's an > invitation for people to run setup.py whatever, and then file bug reports > that it doesn't do exactly what setuptools/distutils would do. Ok, so I understand that frustration. You're going to get those through pip too though - right now there's a giant escape clause in pip where arbitrary options are passed through to setup.py, because we don't have semantic modelling of all the things that people want to do. And until thats achieved - and the options aged out - folk are going to use those options and then be surprised when flit using packages don't behave the same way. > This is what I like about Nathaniel's proposal. By looking for a new file, > it clearly indicates that this is a new style Python source tree, without > potential confusion over what setup.py does, and it lets us specify a simple > build system interface based on standardised pieces like wheels rather than > messing around with the details of setuptools. The problem with 'simple' is that we've got a rich (not just old: fully featured) interface. Building something that isn't complected out from behind that is both valuable and hard. An 85% solution or 95% solution will retain the trap-doors that let folk tickle the system from the outside, and thats going to lead back to the same bug reports you don't want IMO :/. I'm not against a spec that avoids the need for each tree to have a setup.py - I just don't think its going to be the end of the story in any deep fashion :). Here's a draft one I'd be happy with (with no boilerplate or anything). For commands we can define a small number environment variables like e.g. OUTPUT_PATH and PYTHON have those supplied when invoking the command. Python packaging config in $root/pypa.yaml Defined keys: ---- version: # not needed yet, since its a new file, defaults to 1. setup-requires: - requirement - requirement[extra] - requirement:markers - requirement[extra]:markers build-tool: # basic command that will spit a hunk of json back to the caller defining the # commands to use with the build tool. ---- Build tool output json (in yaml format for ease of reading): version: # not needed yet, defaults to 1 egg-info: # comand to run to generate an egg-info directory # only used if dist-info is not defined dist-info: # command to run to generate a dist-info directory wheel: # command to build a wheel develop: # command to do an in-place installation install: # command to do a direct installation # only used if wheeling failed provided-by: # the requirement that provides the build system. # this is used to facilitate caching the build tool output # so that we don't need to probe everytime. (e.g. if the # same dependency version is present in two cases # we can presume the output - I threw this in as an # obvious tweak for large dependency chains but it is # probably not worth it for typical 10-20 package things -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ralf.gommers at gmail.com Mon Oct 12 08:23:03 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 12 Oct 2015 08:23:03 +0200 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Mon, Oct 12, 2015 at 6:37 AM, Robert Collins wrote: > On 12 October 2015 at 17:06, Robert Collins > wrote: > > EWOW, huge thread. > > > > I've read nearly all of it but in order not to make it massively > > worse, I'm going to reply to all the points I think need raising in > > one mail :). > > And a bugfix :) - I didn't link to the docs for the build system > interface we have today - > https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > >From that link: """ In order for pip to install a package from source, setup.py must implement the following commands: ... The install command should implement the complete process of installing the package to the target directory XXX. """ That just sounds so wrong. You want the build system to build, not install. And if "install" actually means "build to a tempdir so pip can copy it over it to its final location", then how does that address something like installing docs to a different dir than the package itself? +1 for your main point of focusing more on enabling other build systems though. Ralf > -Rob > > > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Mon Oct 12 10:15:00 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 12 Oct 2015 09:15:00 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 05:37, Robert Collins wrote: > And a bugfix :) - I didn't link to the docs for the build system > interface we have today - > https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface > I'm happy to see that the command line interface pip requires from setup.py is now documented. But the first thing it describes is the egg_info command, and the description of what that involves is basically 'whatever setuptools does'. Egg info is not any kind of standard, AFAIK - unlike dist info. One of my main goals in writing flit is to build wheels without involving setuptools at all, so I'm probably never going to implement that. I also don't want to have a setup.py in the VCS at all, because it's an invitation for people to run setup.py whatever, and then file bug reports that it doesn't do exactly what setuptools/distutils would do. This is what I like about Nathaniel's proposal. By looking for a new file, it clearly indicates that this is a new style Python source tree, without potential confusion over what setup.py does, and it lets us specify a simple build system interface based on standardised pieces like wheels rather than messing around with the details of setuptools. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Oct 12 14:52:12 2015 From: dholth at gmail.com (Daniel Holth) Date: Mon, 12 Oct 2015 12:52:12 +0000 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: I could be wrong but if egg-info actually generated a dist-info directory it would probably still work. On Mon, Oct 12, 2015 at 8:50 AM Thomas Kluyver wrote: > On 12 October 2015 at 05:37, Robert Collins > wrote: > >> And a bugfix :) - I didn't link to the docs for the build system >> interface we have today - >> >> https://pip.pypa.io/en/latest/reference/pip_install/#build-system-interface >> > > I'm happy to see that the command line interface pip requires from > setup.py is now documented. But the first thing it describes is the > egg_info command, and the description of what that involves is basically > 'whatever setuptools does'. Egg info is not any kind of standard, AFAIK - > unlike dist info. One of my main goals in writing flit is to build wheels > without involving setuptools at all, so I'm probably never going to > implement that. > > I also don't want to have a setup.py in the VCS at all, because it's an > invitation for people to run setup.py whatever, and then file bug reports > that it doesn't do exactly what setuptools/distutils would do. > > This is what I like about Nathaniel's proposal. By looking for a new file, > it clearly indicates that this is a new style Python source tree, without > potential confusion over what setup.py does, and it lets us specify a > simple build system interface based on standardised pieces like wheels > rather than messing around with the details of setuptools. > > Thomas > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon Oct 12 15:12:46 2015 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 12 Oct 2015 08:12:46 -0500 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 11, 2015 11:07 PM, "Robert Collins" wrote: > > EWOW, huge thread. > > I've read nearly all of it but in order not to make it massively > worse, I'm going to reply to all the points I think need raising in > one mail :). > > Top level thoughts here, more point fashion with only rough editing > below the fold. > > I realise many things - like the issue between different wheels of the > same package consuming different numpy abis - have been touched on, > but AFAICT they are entirely orthogonal to the proposal, which was to > solve 'be able to use arbitrary build systems and still install with > pip'. > > Of the actual problems with using arbitrary build systems, 99% of them > seem to boil down to 'setup-requires isn't introspectable by pip > (https://github.com/pypa/pip/issues/1820 ). - If it was, then > alternative build systems could be depended on reasonably; and the > mooted thunk from setuptools CLI to arbitrary build system would be > viable. > > It is, in principle a matter of one patch to teach pip *a* way to do > this (and then any and all build systems that want to can utilise it). > https://github.com/rbtcollins/pip/tree/declarative is a POC I did - my > next steps on that were to discuss the right ecosystem stuff for it - > e.g. should pip consume it via setuptools, or should pip support it as > *the way* and other systems including setuptools can choose to use it? as a standard RDF graph representation, JSON-LD would be uniquely portable here. "PEP 426: Define a JSON-LD context as part of the proposal" https://github.com/pypa/interoperability-peps/issues/31 > > A related but separate thing is being able to *exclusively* install > things without setuptools present - I've filed > https://github.com/pypa/pip/issues/3175 about that, but I think its > -much- lower priority than reliably enabling third party build tools. peep may not need setuptools? * SHA256 * --no-deps * https://pypi.python.org/pypi/peep * wheels > > -Rob > > ---- > > > " > solved many of the hard problems here -- e.g. it's no longer necessary > that a build system also know about every possible installation > configuration -- so pretty much all we really need from a build system > is that it have some way to spit out standard-compliant wheels. > " > > Actually pip still punts a *lot* here - we have bypasses to let things > like C compiler flags be set during wheel build, and when thats done > we don't cache the wheels (or even try to build wheels). > > " > While ``distutils`` / ``setuptools`` have taken us a long way, they > suffer from three serious problems: ... > (c) you are forced to use them anyway, because they provide the > standard interface for installing python packages expected by both > users and installation tools like ``pip``." > > I don't understand the claim of (c) here - its entirely possible to > write a package that doesn't use setuptools and have it do the right > thing - pip uses a subprocess to drive package installation, and the > interface is documented. The interface might be fugly as, but it > exists and works. It is missing setup-requires handling, but so is > setup.py itself. The only thing we'd really need to do AFAICT is make > our setuptools monkeypatching thunk handle setuptools not being > installed (which would be a sensible thing to Just Do anyhow). > > " > - query for build dependencies > - run a build, producing wheels as output > - set up the current source tree so that it can be placed on > ``sys.path`` in "develop mode" > " > > So we have that already. setup.py egg-info, setup.py bdist_wheel, > setup.py develop. > > "A version 1-or-greater format source tree can be identified by the > presence of a file ``_pypackage/_pypackage.cfg``. > " > > I really don't like this. Its going to be with us forever, and its > intrusive (its visible), and so far isn't shown to be fixing anything. > > > "to scatter files around willy-nilly never works, so we adopt the > convention that names starting with an underscore are reserved for > official use, and non-underscored names are available for > idiosyncratic use by individual projects." > > I can see the motivation here, but is it really solving a problem we have? > > > On the specifics of the format: I don't want to kibbitz over strawman > aspects at this point. > > Having the extension mechanism be both pip specific and in Python > means that we're going to face significant adoption issues: the former > because pip is not by any means the only thing around - and some > distros have until very recently been actively hostile to pip (which > in turn means we need to wait a decade or two for them to age-out and > stop being used). The latter because we'll face all the headaches of > running arbitrary untrusted code and dealing with two deps with > different versions of the same hook and so on: I think its an > intrinsically unsafe design. > > @dstufft "problem with numpy.distutils, as I know you?re aware!). We > could do a minimal extension and add another defacto-ish standard of > allowing pip and setuptools to process additional setup_requires like > arguments from a setup.cfg to solve that problem though. The flip side > to this is that since it involves new capabilities in > pip/setuptools/any other installer is that it you?ll have several > years until you can depend on setup.cfg based setup_requires from > being able to be depended on. > " > > Well. For *any* proposal that involves modifying pip, we have to > assume that all existing things keep working, and that anyone wanting > to utilise the new thing will have to either a) include a local > compatibility thunk, or b) error when being used from a too-old > toolchain. I don't think that should really be a factor in design > since its intrinsic to the quagmire. > > "Longer term, I think the answer is sdist 2.0 which has proper > metadata inside of it (name, version, dependencies, etc) but which > also includes a hook like this PEP has to specify the build system > that should be used to build a wheel out of this source distribution." a composed JSON-LD document indicating provenance (who, what, when) for each part of the build chain [VCS archive, egg-info, sdist, wheel, bdist] pydist.jsonld? > > Any reason that can't just be setup.cfg ? > > @Daniel "I thought Robert Collins had a working setup-requires > implementation already? I have a worse but backwards compatible one > too at https://bitbucket.org/dholth/setup-requires/src/tip/setup.py" - > https://github.com/rbtcollins/pip/tree/declarative - I'll be updating > that probably early next year at this rate - after issue-988 anyhow. > The issue with your approach is that pip doesn't handle having > concurrent installs done well - and in fact it will end up locking its > environment somehow. > > @Paul " > I can understand that a binary wheel may need a certain set of > libraries installed - but that's about the platform tags that are part > of the wheel definition, not about dependencies. Platform tags are an > ongoing discussion, and a good example of a partial solution that" - > thats where the draft PEP tennessee and I start is aimed - at making > those libraries be metadata, not platform tags. > > @Chris " > A given package might depend on numpy, as you say, and it may work > with all numpy versions 1.6 to 1.9. Fine, so we specify that in > install_requires. And this shodl be the dependency in the sdist, too. > If the package is pur python, this is fine and done. > > But if the package has some extensions code that used the numpy C API > ( a very common occurrence), then when it is built, it will only work > (reliably) with the version of numpy it was built with. > > So the project itself, and the sdist depend on numpy >=1.6, but a > build binary wheel depends on numpy == 1.7 (for instance). > > Which requires a binary (wheel) dependency that is somewhat different > than the source dependency. > " - so yes, that is where bdist_wheel should be creating different > metadata for that wheel. The issue that arises is that we need unique > file names so that they can coexist on PyPI or local archives - which > is where wheel tags come in. I'd be in favour of not using semantic > tags for this - rather hash the deps or something and just make a > unique file name. Use actual metadata for metadata. > > @Nathaniel "I know that one unpleasant aspect of the current design is that the > split between egg-info and actual building creates the possibility for > time-of-definition-to-time-of-use bugs, where the final wheel > hopefully matches what egg-info said it would, but in practice there > could be skew. (Of course this is true in any system which represents" > - actually see https://bugs.launchpad.net/pbr/+bug/1502692 for a bug > where this 'skew' is desirable: for older environments we want > tailored deps with no markers, for anything supporting markers we want > them - so the wheel will have markers and egg_info won't. > > @Nathaniel " > (Part of the intuition for the last part is that we also have a > not-terribly-secret-conspiracy here for writing a PEP to get Linux > wheels onto PyPI and at least achieve feature parity with Windows / OS > X. Obviously there will always be weird platforms -- iOS and FreeBSD > and Linux-without-glibc and ... -- but this should dramatically reduce > the frequency with which people need sdist dependencies.)" - I think a > distinction between sdist and binary names for dependencies would be a > terrible mistake. It will raise complexity for reasoning and > describing things without solving any concrete problem that I can see. > > @Nathaniel "I guess to make progress in this conversation I need some > more detailed explanations. I totally get that there's a long history > of thought and conversations behind the various assertions here like > "a sdist is fundamentally different from a VCS checkout", "there must > be a 1-1 mapping between sdists and wheels", "pip needs sdists that > have full wheel metadata in static form", and I'm barging in from the > outside with no context, but I literally have no idea why the specific > design features you're asking for are desirable or even viable. Right > now if I were to try and write the PEP you're asking for, then the > rationale section would just be "because Donald said so" over and over > :-). I couldn't write the motivation section, because I don't know any > problems that the PEP you're describing would fix for me as a package > author (which doesn't mean they don't exist, but!)." -- VCS trees are > (generally) by-humans for humans. They are the primary source of data > and can do thinks like inferring versions from commit data. sdists are > derived from the VCS tree and can include extra data (such as > statically defined version data). Wheels are derived from a tree on > disk and can (today) be built from either VCS trees or sdists. I'm not > sure that forcing an sdist step is beneficial - the egg-info step we > have today is basically that without the cost of compressing and > decompressing potentially large trees for no reason. > > @Jeremy "An sdist is an installable package which just happens to _look_ a > lot like a source release tarball, but trying to pretend that > downstream packagers will want to use it as such leads to a variety > of pain points in the upstream/downstream relationship. For better > or worse a lot of distros don't want generated files in upstream > source code releases, since they need to confirm that they also ship > the necessary tooling to regenerate any required files and that the > generated files they ship match what their packaged tooling > produces." - Well, pbr doesn't work if you just tar up or git export > your VCS tree: it requires the chance to add metadata. And while > distros have whinged about pbr in a number of contexts, that hasn't > been one so far. Downstreams are pretty used to receiving tarballs > with generated files in them - as long as they *have the option* to > recreate those, so the source material isn't lost. [And for version > data, 'grab from git' is a valid answer there']. OTOH perhaps > ftpmaster just hasn't noticed and we're about to get a bug report ;) another interesting use case for [not-] pip: https://github.com/mitsuhiko/pipsi > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Mon Oct 12 15:33:03 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Mon, 12 Oct 2015 14:33:03 +0100 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 12 October 2015 at 11:01, Robert Collins wrote: > Python packaging config in $root/pypa.yaml > > Defined keys: > ---- > ... > build-tool: > # basic command that will spit a hunk of json back to the caller > defining the > # commands to use with the build tool. > ---- > > Build tool output json (in yaml format for ease of reading): > I would be quite happy with something along the lines of this proposal, though I'd bikeshed about some of the details. I like the idea of the source tree having a single reference to the build tool, and the build tool describing itself to pip. I'd probably use references to Python functions/modules rather than specifying shell-style commands, though; it seems like there's less to go wrong that way. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Oct 12 19:11:10 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 13 Oct 2015 06:11:10 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 13 October 2015 at 01:52, Daniel Holth wrote: > I could be wrong but if egg-info actually generated a dist-info directory it > would probably still work. I'd worry about fallout - since pip doesn't need to change (much) either way, I'd be inclined to be specific and allow setuptools to merely define the bootstrap code and opt in to later changes without pressure. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 12 19:16:53 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 13 Oct 2015 06:16:53 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 13 October 2015 at 02:33, Thomas Kluyver wrote: > On 12 October 2015 at 11:01, Robert Collins > wrote: >> >> Python packaging config in $root/pypa.yaml >> >> Defined keys: >> ---- >> ... >> build-tool: >> # basic command that will spit a hunk of json back to the caller >> defining the >> # commands to use with the build tool. >> ---- >> >> Build tool output json (in yaml format for ease of reading): > > > I would be quite happy with something along the lines of this proposal, > though I'd bikeshed about some of the details. I like the idea of the source > tree having a single reference to the build tool, and the build tool > describing itself to pip. I'd probably use references to Python > functions/modules rather than specifying shell-style commands, though; it > seems like there's less to go wrong that way. One of the fundamental things that emerged during the review of the design of my static setup-requires implementation in pip was that setuptools behaviour of not installing setup requirements into the target environment was deliberate design: it permits the use of different, mutually incompatible, versions of a given setup requirement by packages in the same dependency tree. E.g. imagine A and B both use setuptools-vcs, and setuptools-vcs does an incompatible 2.0 release. When A upgrades to that and B hasn't, if B install-requires A, pip installing B needs to install both those setuptools-vcs versions transiently, not permanently. (Even if one version is already installed, the build-time actions around the other of A|B need to have the other version installed). [My branch of pip doesn't do this - its one of the differences between proof of concept and production ready] So until we solve the problems related to unloading something loaded into Python and loading a different version in and all the related pain that can occur - I think using Python function calls is a non-starter. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 19:23:53 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 10:23:53 -0700 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On Oct 12, 2015 10:16 AM, "Robert Collins" wrote: > > On 13 October 2015 at 02:33, Thomas Kluyver wrote: > > On 12 October 2015 at 11:01, Robert Collins > > wrote: > >> > >> Python packaging config in $root/pypa.yaml > >> > >> Defined keys: > >> ---- > >> ... > >> build-tool: > >> # basic command that will spit a hunk of json back to the caller > >> defining the > >> # commands to use with the build tool. > >> ---- > >> > >> Build tool output json (in yaml format for ease of reading): > > > > > > I would be quite happy with something along the lines of this proposal, > > though I'd bikeshed about some of the details. I like the idea of the source > > tree having a single reference to the build tool, and the build tool > > describing itself to pip. I'd probably use references to Python > > functions/modules rather than specifying shell-style commands, though; it > > seems like there's less to go wrong that way. > > One of the fundamental things that emerged during the review of the > design of my static setup-requires implementation in pip was that > setuptools behaviour of not installing setup requirements into the > target environment was deliberate design: it permits the use of > different, mutually incompatible, versions of a given setup > requirement by packages in the same dependency tree. E.g. imagine A > and B both use setuptools-vcs, and setuptools-vcs does an incompatible > 2.0 release. When A upgrades to that and B hasn't, if B > install-requires A, pip installing B needs to install both those > setuptools-vcs versions transiently, not permanently. (Even if one > version is already installed, the build-time actions around the other > of A|B need to have the other version installed). [My branch of pip > doesn't do this - its one of the differences between proof of concept > and production ready] > > So until we solve the problems related to unloading something loaded > into Python and loading a different version in and all the related > pain that can occur - I think using Python function calls is a > non-starter. I don't see the contradiction here. If you look at the original draft PEP then it exactly specifies that builds get isolated environments, and build tools are supposed to spawn a child and then have that child do a function call using whatever mechanism they prefer. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Oct 12 19:47:33 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 13 Oct 2015 06:47:33 +1300 Subject: [Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils In-Reply-To: References: Message-ID: On 13 October 2015 at 06:23, Nathaniel Smith wrote: > On Oct 12, 2015 10:16 AM, "Robert Collins" > wrote: ... >> So until we solve the problems related to unloading something loaded >> into Python and loading a different version in and all the related >> pain that can occur - I think using Python function calls is a >> non-starter. > > I don't see the contradiction here. If you look at the original draft PEP > then it exactly specifies that builds get isolated environments, and build > tools are supposed to spawn a child and then have that child do a function > call using whatever mechanism they prefer. Ok, so here's a worked example to let us debug the disconnect. Given: A at 1.0: setup-requires: S~=1.0 install-requires: B B at 1.0: setup-requires: S~=2.0 S at 1.0: no dependencies at all. S at 2.0: no dependencies at all. and no binaries of A or B... then: pip install A will do the following (key bits related to this proposal only, happy path only): - download the A at 1.0 sdist - read pypa.yaml and read the setup-requires + build-tool keys - download S at 1.0 and prepare a built version of it - place S at 1.0 into PYTHONPATH and the built versions bin into PATH - run the build-tool command to determine how to use it - run the resulting wheel-build command to build a wheel for A at 1.0 - read the wheel metadata in to determine A's install-requires - download the B at 1.0 sdist - read pypa.yaml and read the setup-requires + build-tool keys - download S at 2.0 and prepare a built version of it - place S at 2.0 into PYTHONPATH and the built versions bin into PATH - run the build-tool command to determine how to use it - run the resulting wheel-build command to build a wheel for B at 1.0 - read the wheel metadata in to determine B's install-requires - install the B wheel into the target environment - install the A wheel into the target environment Note the two places where PYTHONPATH and PATH need to be something other than the environment of pip itself. pip may not be installed in the target environment (or may be a different version than the pip installed there). I don't understand how you propose to have S at 1.0 and S at 2.0 co-exist in pip's Python process. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 12 20:01:14 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 12 Oct 2015 11:01:14 -0700 Subject: [Distutils] Meeting info re: sdists Message-ID: Hangouts URL: https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a Shared document for notes: https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Tue Oct 13 18:09:19 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 13 Oct 2015 09:09:19 -0700 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: Hi all, I was going to send around a summary from this meeting, but I seem to have come down with the flu last night. So if someone else wants to do it while it's fresh in our minds, that'd be great, or else I'll try to get to it next week or so. Cheers, -n On Oct 12, 2015 11:01 AM, "Nathaniel Smith" wrote: > Hangouts URL: > https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a > > Shared document for notes: > > https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Oct 13 18:59:05 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 13 Oct 2015 09:59:05 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: On Sun, Oct 11, 2015 at 11:18 AM, Steve Dower wrote: > And it's also not binding on redistributors or package builders, who are > free to add more restrictive requirements. > not binding, but presumably a simple: setup.py build with distutils (or setuptools) will NOT use SSE2 in this case? That's the beauty of distutils of course, that is builds extensions with all the same settings as the host python. So even is SSE2 provides little for Python itself, in the usual case, we'll see performance hits n compiled extensions that are not compiled by particularly smart people. Also -- this thread started with "I can build Python3 myself, but now I can't install arbitrary binary wheels" -- if the goal is to fix that, then we don't WANT people tweaking things to build their wheels with SSE2 -- it would just get uglier. I certanly thought the "official" policy was that binary wheels put up on PyPi be (fully) compatible with the python.org build(s). What I'm getting at is that a decision to not use SSE2 in the main python build is a decision to not use it for any PyPi packages -- which could be a Bad Thing. > As far as identifying these sort of requirements in packages, I don't > think they really make sense as part of a platform tag as a non-SSE build > is fully compatible with an SSE build, but as part of version ordering it > would be nice to be able to prefer certain tagged builds without excluding > ones without the tag (e.g. if my platform is win32_sse2 and there's only a > win32 tagged build, I still want it - how we formalize this is probably > very complex though. > yeah, that's the trick with trying to shoehorn dependencies liek this into platform tags. But even if so do we WANT to now be supporting 4 Windows platforms? win32 win32_sse2 win64 win64_sse2 and then, what about sse3, etc??? My thoughts -- pick one, and either: * SSE2 is a requirement for python3.5 on Windows - then schools and the like can build their own Python3.5, AND their own package repository - on the one hand, institutions should be able to do this - on the other hand, the same institutions that NEED to do this (nes with old hardware),. are probably short on other resources, like admins savvy enough to be setting up build systems... * SSE2 is NOT used for the Windows builds. - then all packages on PyPi should also not use SSE2 - Other organizations, like Anaconda, Enthought, etc can build their own python and dependent packages can be build against that. - At least these folks DO have the resources to support that. -CHB > Also, my platform tag is probably 100 chars long with everything I may > want to represent...) > > Cheers, > Steve > > Top-posted from my Windows Phone > ------------------------------ > From: Oscar Benjamin > Sent: ?10/?11/?2015 10:43 > To: Antoine Pitrou ; Distutils-Sig at python.org > Subject: Re: [Distutils] warning about potential problem for wheels > > On Sun, 11 Oct 2015 17:44 Antoine Pitrou wrote: > > On Sun, 11 Oct 2015 08:07:30 -0700 > Steve Dower wrote: > > > > This does only affect 32-bit builds, so now I'm thinking about the > > possibility of treating those as highly compatible while the 64-bit > > ones get better performance treatment, though I'm not sure how that > > could actually play out. It may help remove some of the questions > > about which one to use though. > > That sounds reasonable to me. I don't know Windows very much, but are > there still many people using 32-bit Windows these days (on x86, I > mean)? > > > > I don't know but I think it makes sense to follow Windows' lead. So if 3.5 > supports Vista and Vista doesn't require SSE2 then CPython shouldn't > either. If 3.6 or whatever drops support for Vista and if Windows 7 > requires SSE2 then CPython can require it too. I assume this what happens > with the OSX binaries. > > Note that my recently retired computer was 64 bit and had SSE but didn't > have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after > SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers > are going to stop adding new instructions. So this general issue isn't > limited to 32 bit hardware and won't be solved by special casing that. I > think it makes sense to have a general policy for architectures that will > be supported by the official build in future. > > -- > Oscar > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Oct 13 19:12:35 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 13 Oct 2015 13:12:35 -0400 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: On October 13, 2015 at 1:07:44 PM, Chris Barker (chris.barker at noaa.gov) wrote: yeah, that's the trick with trying to shoehorn dependencies liek this into platform tags. But even if so do we WANT to now be supporting 4 Windows platforms? win32 win32_sse2 win64 win64_sse2 and then, what about sse3, etc??? Longer term, more platforms (particularly more platforms that really only differ by some compiler flags) shouldn?t be a very big deal, because my dream for PyPI is that authors upload a source package and PyPI builds all their wheels for them (optionally of course). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 00:31:19 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 11:31:19 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 05:09, Nathaniel Smith wrote: > Hi all, > > I was going to send around a summary from this meeting, but I seem to have > come down with the flu last night. So if someone else wants to do it while > it's fresh in our minds, that'd be great, or else I'll try to get to it next > week or so. > > Cheers, > -n > > On Oct 12, 2015 11:01 AM, "Nathaniel Smith" wrote: >> >> Hangouts URL: >> https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a >> >> Shared document for notes: >> >> https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing So that ^ is worth a read for any interested party. The summary: We agreed on the following design constraints(MUSTS): * Graceful forward compat (old pip either can install, or fails cleanly, on new VCS and sdists, pypi can be changed ) * Backwards compat (new pip can install both old and new VCS/sdists, new pypi can handle old and new sdists) * Setup-requires static to the point the build system can be installed by pip * Setup-requirements need to be isolated: must be able to use different incompatible versions of the same setup-requirement during one install run. * Describing how to use the build system has to be user friendly. [e.g. json configs are out] And the following design goals(SHOULDs): * Be able to use things like flit cleanly within pip without needing a setup.py in-tree. * Long term - reduce variation in code paths * Long term - reduce amount of dynamism? in install pipeline? * We may never be able to remove it entirely, but being able to not run egg-info in the common case should be achievable * Things that have reason to change (deps) are more reasonable to be dynamic (even with PEP-426 markers there are exceptions) * project description, version etc in sdist have no reason to be dynamic * Files in VCS should not need to be different in sdist - ever We had broad agreement on the basic design: * There will be a file that describes how to install the build tool, and how pip or other systems should invoke it * It will be minimal: this file is strictly additive to the system, no overlap with existing things (even setup-requires - because setup-requires presumes the buildtool (setuptools) is already installed) I *think* we had broad agreement that its not desirable, overall, to try and skip steps like metadata generation - because with a resolver we'd end up potentially doing things like building dozens of versions of scipy just to extract metadata about dependencies. I like the term 'bootstrap-requires' -because we're bootstrapping the ability to use the build tool. The build tool interface needs to be sufficient that one could write a shim to existing setuptools in it. Previous posts have described what that might look like, but to reprise my sketch it had the following entries schema: #was version egg-info: dist-info: wheel: develop: install: provided-by: We have a design decision to make. We could make bootstrap-requires be a superset of setup-requires. That is, projects should put anything that is functionally a setup-requires thing into bootstrap-requires. This had the ad-hoc effect of dictating to setuptools how static setup-requires would be expressed. I don't tlike that. Alternatively we can make the interface permit querying for things to install before running any of the other things in the build system interface: e.g. one way would be build-requires: # command to query for package specific build dependencies # returns a PEP-426 JSON file, only the keys https://www.python.org/dev/peps/pep-0426/#metadata-version and "build_requires" are consulted. Alternatives here include: - just generating {dist.egg}-info twice - once to get setup_requires, once to get everything once thats resolved - drop dist-info and egg-info and use PEP-426 for those calls as well - don't use pep-426 at all, just use requires.txt format which is already well established - Use a pep-345 file I think pep-426 is ok since we're already using it within wheels (metadata.json). I'm a little nervous about dropping egg-info/dist-info - but hell, lets try it (since its common to have a proof of concept before accepting peps, we'll know it works for setuptools since that should be in the proof of concept). That would give a build tool interface of: schema: #was version build-requires: # command to get a pep-426 json blob on stdout with build_requires populated dist-info: # command to get a pep-426 json blob on stdout with everything populated wheel: # build a wheel develop: # do a develop install install: # do a direct install provided-by: # allows caching the build tool interface We need to define the exact environment that the build tool will be run in: will it inherit already installed things from the install target? Will it be isolated or hermetic? Being able to fix things at a distance is important - and remembering we have a lot of legacy paths, that means a greenfield approach isn't very viable. Separately we need to define how-and-when dist-info will be called in the case of sdists which have valid static metadata. Thats sufficiently different we think it should be a separate PEP. Go forward: Nathaniel to write two PEPs: * bootstrap-requirements and build system interface - enables multiple build systems * usable static metadata in sdists - allow the common case to do less work by ensuring valid metadata in sdists -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From chris.barker at noaa.gov Wed Oct 14 01:31:32 2015 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 13 Oct 2015 16:31:32 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: <-1380413487201379509@unknownmsgid> But even if so do we WANT to now be supporting 4 Windows platforms? win32 win32_sse2 win64 win64_sse2 and then, what about sse3, etc??? Longer term, more platforms (particularly more platforms that really only differ by some compiler flags) shouldn?t be a very big deal, because my dream for PyPI is that authors upload a source package and PyPI builds all their wheels for them (optionally of course). That would be pretty nice. I'm still a bit wary though -- the more platforms you have, the more you have to test and debug.... Testing, off course would be part of the automated build. But debugging could get painful ... CHB ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 14 01:39:28 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 13 Oct 2015 18:39:28 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 13, 2015 6:01 PM, "Robert Collins" wrote: > > On 14 October 2015 at 05:09, Nathaniel Smith wrote: > > Hi all, > > > > I was going to send around a summary from this meeting, but I seem to have > > come down with the flu last night. So if someone else wants to do it while > > it's fresh in our minds, that'd be great, or else I'll try to get to it next > > week or so. > > > > Cheers, > > -n > > > > On Oct 12, 2015 11:01 AM, "Nathaniel Smith" wrote: > >> > >> Hangouts URL: > >> https://hangouts.google.com/call/fzzbnwpdujlkhtcv6lmcopnqj4a > >> > >> Shared document for notes: > >> > >> https://docs.google.com/document/d/11MZVPaayaFD3dd1NRjcKpyzlk_1IXkgQDqtkxkXuoNk/edit?usp=sharing > > So that ^ is worth a read for any interested party. > > The summary: > > We agreed on the following design constraints(MUSTS): > > * Graceful forward compat (old pip either can install, or fails > cleanly, on new VCS and sdists, pypi can be changed ) > * Backwards compat (new pip can install both old and new VCS/sdists, > new pypi can handle old and new sdists) > * Setup-requires static to the point the build system can be installed by pip > * Setup-requirements need to be isolated: must be able to use > different incompatible versions of the same setup-requirement during > one install run. > * Describing how to use the build system has to be user friendly. > [e.g. json configs are out] (.cfg | .yml) > JSONLD Should these build settings all be included in e.g. PEP 426 + JSONLD metadata.jsonld? - for backward compat, there would then be both metadata.json and metadata.jsonld (because of normative JSONLD form and PEP 426+) .cfg (ConfigParse): (section, key, value) .yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF triple graph (of linked typed packages with attributes, generated from [...])) > > And the following design goals(SHOULDs): > > * Be able to use things like flit cleanly within pip without needing a > setup.py in-tree. > * Long term - reduce variation in code paths > * Long term - reduce amount of dynamism? in install pipeline? * OS Packaging compat (--prefix) * Conda solves for this. (static files with package-level checksums) > * We may never be able to remove it entirely, but being able to not > run egg-info in the common case should be achievable > * Things that have reason to change (deps) are more reasonable to be > dynamic (even with PEP-426 markers there are exceptions) > * project description, version etc in sdist have no reason to be dynamic > * Files in VCS should not need to be different in sdist - ever > > > We had broad agreement on the basic design: > * There will be a file that describes how to install the build tool, > and how pip or other systems should invoke it > * It will be minimal: this file is strictly additive to the system, no > overlap with existing things (even setup-requires - because > setup-requires presumes the buildtool (setuptools) is already > installed) > > I *think* we had broad agreement that its not desirable, overall, to > try and skip steps like metadata generation - because with a resolver > we'd end up potentially doing things like building dozens of versions > of scipy just to extract metadata about dependencies. > > I like the term 'bootstrap-requires' -because we're bootstrapping the > ability to use the build tool. > > The build tool interface needs to be sufficient that one could write a > shim to existing setuptools in it. Previous posts have described what > that might look like, but to reprise my sketch it had the following > entries > schema: #was version > egg-info: > dist-info: > wheel: > develop: > install: > provided-by: > > We have a design decision to make. > > We could make bootstrap-requires be a superset of setup-requires. That > is, projects should put anything that is functionally a setup-requires > thing into bootstrap-requires. This had the ad-hoc effect of dictating > to setuptools how static setup-requires would be expressed. I don't > tlike that. > > Alternatively we can make the interface permit querying for things to > install before running any of the other things in the build system > interface: e.g. one way would be > > build-requires: > # command to query for package specific build dependencies > # returns a PEP-426 JSON file, only the keys > https://www.python.org/dev/peps/pep-0426/#metadata-version and > "build_requires" are consulted. > > Alternatives here include: > - just generating {dist.egg}-info twice - once to get setup_requires, > once to get everything once thats resolved > - drop dist-info and egg-info and use PEP-426 for those calls as well > - don't use pep-426 at all, just use requires.txt format which is > already well established > - Use a pep-345 file > > I think pep-426 is ok since we're already using it within wheels > (metadata.json). I'm a little nervous about dropping > egg-info/dist-info - but hell, lets try it (since its common to have a > proof of concept before accepting peps, we'll know it works for > setuptools since that should be in the proof of concept). That would > give a build tool interface of: > > schema: #was version > build-requires: # command to get a pep-426 json blob on stdout with > build_requires populated > dist-info: # command to get a pep-426 json blob on stdout with > everything populated > wheel: # build a wheel > develop: # do a develop install > install: # do a direct install > provided-by: # allows caching the build tool interface > > We need to define the exact environment that the build tool will be > run in: will it inherit already installed things from the install > target? Will it be isolated or hermetic? Being able to fix things at a > distance is important - and remembering we have a lot of legacy paths, > that means a greenfield approach isn't very viable. > > Separately we need to define how-and-when dist-info will be called in > the case of sdists which have valid static metadata. Thats > sufficiently different we think it should be a separate PEP. > > Go forward: > > Nathaniel to write two PEPs: > * bootstrap-requirements and build system interface - enables > multiple build systems > * usable static metadata in sdists - allow the common case to do less > work by ensuring valid metadata in sdists > > -Rob > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Oct 14 02:25:21 2015 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 13 Oct 2015 17:25:21 -0700 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: thanks for the summary! * Things that have reason to change (deps) are more reasonable to be > dynamic (even with PEP-426 markers there are exceptions) > as we know, for *many* cases, run-time deps aren't dynamic. is there a consensus for those cases? exist in the sdist metadata? or no? or maybe with the "auto-building" Donald mentioned, we'd rely solely on wheel "built" metadata as the vehicle for dependency resolution also, did Nathaniel's idea of uploading c libraries into PyPI as python wrapper projects get discussed? --Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 02:46:11 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 13:46:11 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 13:25, Marcus Smith wrote: > > > thanks for the summary! > > >> * Things that have reason to change (deps) are more reasonable to be >> dynamic (even with PEP-426 markers there are exceptions) > > > as we know, for *many* cases, run-time deps aren't dynamic. > is there a consensus for those cases? exist in the sdist metadata? or no? The plan we hashed out (this would be the new PEP on static metadata in sdists) - pip etc to lint and error if they encounter a malformed dist-info in an sdist - then start putting dist-info, not egg-info into sdists - for any field if the field is entirely absent, we'll get it by running the dist-info build-tool command I described in the other mails Concretely: {'build_requires': []} -> no build requirements {} -> get build requirements by running the build system > or maybe with the "auto-building" Donald mentioned, we'd rely solely on > wheel "built" metadata as the vehicle for dependency resolution I don't believe thats feasible in a performant fashion. > also, did Nathaniel's idea of uploading c libraries into PyPI as python > wrapper projects get discussed? Nope :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 14 02:50:14 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 13:50:14 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 12:39, Wes Turner wrote: > .... > (.cfg | .yml) > JSONLD > > Should these build settings all be included in e.g. PEP 426 + JSONLD > metadata.jsonld? > - for backward compat, there would then be both metadata.json and > metadata.jsonld (because of normative JSONLD form and PEP 426+) I'm sorry, I have literally no idea what you're talking about. I know you have a thing about JSONLD, but I don't understand how or why it is relevant to these design discussions, which are 99% fine tuning backwards/forwards compat needs around the existing Python code thats out there and will be interacting with what we produce. If you want to shift the needle here and get discussions talking about things with JSONLD rather than PEP-426 (or some future leaner and less enterprisy version) ... then for me at least, I need more clear communication, less assumption that I have *any* overlap with where you're coming from, and at least some connection to the problems we're working on in your mails. > .cfg (ConfigParse): (section, key, value) > > .yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF > triple graph (of linked typed packages with attributes, generated from > [...])) Still not following. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From wes.turner at gmail.com Wed Oct 14 04:04:16 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 13 Oct 2015 21:04:16 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > > On 14 October 2015 at 12:39, Wes Turner wrote: > > > .... > > (.cfg | .yml) > JSONLD > > > > Should these build settings all be included in e.g. PEP 426 + JSONLD > > metadata.jsonld? > > - for backward compat, there would then be both metadata.json and > > metadata.jsonld (because of normative JSONLD form and PEP 426+) > > I'm sorry, I have literally no idea what you're talking about. I know > you have a thing about JSONLD, but I don't understand how or why it is > relevant to these design discussions, which are 99% fine tuning > backwards/forwards compat needs around the existing Python code thats > out there and will be interacting with what we produce. > > If you want to shift the needle here and get discussions talking about > things with JSONLD rather than PEP-426 (or some future leaner and less > enterprisy version) ... then for me at least, I need more clear > communication, less assumption that I have *any* overlap with where > you're coming from, and at least some connection to the problems we're > working on in your mails. > > > .cfg (ConfigParse): (section, key, value) > > > > .yml (YAML lib xyz): easily maps to JSONLD (which always maps to an RDF > > triple graph (of linked typed packages with attributes, generated from > > [...])) > > Still not following. * egg-info and dist-info should generate JSONLD * sdist, bdist, and wheel should update **append to** the same JSONLD metadata document * if there are additional parameters which would be necessary to reproduce a package(s), those should be included in the package metadata ... the *provenance* of these build artifacts would then be represented in the package metadata document is this specific to sdist? nope, sorry, now is the time. > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 04:40:41 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 15:40:41 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 15:04, Wes Turner wrote: > > On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > * egg-info and dist-info should generate JSONLD The .egg-info and .dist-info directories are existing defined formats, I don't see any way that they can be retroactively defined as being JSONLD. I understand you have a JSONLD spec being discussed - this would be a successor to PEP-426, and Nick has put the breaks on all such things until we *catch up* with the already designed and defined work - so I'm not sure where your spec fits in things: but even assuming its all approved, we still can't change the meaning of existing in the wild artifacts. > * sdist, bdist, and wheel should update **append to** the same JSONLD > metadata document Why? > * if there are additional parameters which would be necessary to reproduce a > package(s), those should be included in the package metadata Why? > ... the *provenance* of these build artifacts would then be represented in > the package metadata document I hate to sound like a broken record, but why? We could list the hashes of all the headers the C compiler consulted in generating a numpy extension module, but what problem are we solving. > is this specific to sdist? nope, sorry, now is the time. The time for what? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From wes.turner at gmail.com Wed Oct 14 05:05:13 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 13 Oct 2015 22:05:13 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 13, 2015 9:40 PM, "Robert Collins" wrote: > > On 14 October 2015 at 15:04, Wes Turner wrote: > > > > On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > > > * egg-info and dist-info should generate JSONLD > > The .egg-info and .dist-info directories are existing defined formats, > I don't see any way that they can be retroactively defined as being > JSONLD. I understand you have a JSONLD spec being discussed - this > would be a successor to PEP-426, and Nick has put the breaks on all > such things until we *catch up* with the already designed and defined > work - so I'm not sure where your spec fits in things: but even > assuming its all approved, we still can't change the meaning of > existing in the wild artifacts. > > > * sdist, bdist, and wheel should update **append to** the same JSONLD > > metadata document > > Why? So that the project artifacts are traceable through the build process; in order to (identify sources of variance) [Requirements Traceability, 5Ws] This is called "provenance" in real metadata circles. [W3C PROV]. > > > * if there are additional parameters which would be necessary to reproduce a > > package(s), those should be included in the package metadata > > Why? Because otherwise one has no idea what they are working with. Because, for reproducibility, it's optimal to record all inputs as structured data. Why would we be throwing this process knowledge away? dash-delimited filenames to indicate a phantom ABI parameter? > > > ... the *provenance* of these build artifacts would then be represented in > > the package metadata document > > I hate to sound like a broken record, but why? We could list the > hashes of all the headers the C compiler consulted in generating a > numpy extension module, but what problem are we solving. This could be useful; we currently must use actual OS packaging to verify per-file on-disk checksums (e.g. debsums). e.g. we actually currently have no idea if the uncompressed archive files differ from the checksummed archive, unless we re-download and compare with an archive manifest; because there is not yet a manifest containing (path, checksum, [perms], [fsattrs]) > > > is this specific to sdist? nope, sorry, now is the time. > > The time for what? > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 14 10:33:17 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Oct 2015 09:33:17 +0100 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 01:46, Robert Collins wrote: > On 14 October 2015 at 13:25, Marcus Smith wrote: >> >> >> thanks for the summary! Agreed. >> >>> * Things that have reason to change (deps) are more reasonable to be >>> dynamic (even with PEP-426 markers there are exceptions) >> >> >> as we know, for *many* cases, run-time deps aren't dynamic. >> is there a consensus for those cases? exist in the sdist metadata? or no? > > The plan we hashed out (this would be the new PEP on static metadata in sdists) > - pip etc to lint and error if they encounter a malformed dist-info in an sdist > - then start putting dist-info, not egg-info into sdists > - for any field if the field is entirely absent, we'll get it by > running the dist-info build-tool command I described in the other > mails > > Concretely: > {'build_requires': []} -> no build requirements > {} -> get build requirements by running the build system One use case that I don't think is covered here is publishing dependency metadata via PyPI. I believe distlib needs this for some functions (at the moment, I think it uses an externally hosted set of package dependency data that's maintained by Vinay), and I know there have been a number of utility scripts I've needed to write that I simply haven't been able to because doing so would involve a significant amount of extra work downloading and running package code. If there are dependencies that are only detectable at wheel build time, then so be it (I'm still looking for an example, but it's clear that the potential is there) but I'd like some way of getting at whatever dependency information a wheel (source or binary) provides via the PyPI JSON API - and I'd like an assurance that if dependency information *can* be determined statically, it will be. Paul From robertc at robertcollins.net Wed Oct 14 10:50:06 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 14 Oct 2015 21:50:06 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 21:33, Paul Moore wrote: > On 14 October 2015 at 01:46, Robert Collins wrote: ... >> Concretely: >> {'build_requires': []} -> no build requirements >> {} -> get build requirements by running the build system > > One use case that I don't think is covered here is publishing > dependency metadata via PyPI. I believe distlib needs this for some > functions (at the moment, I think it uses an externally hosted set of > package dependency data that's maintained by Vinay), and I know there > have been a number of utility scripts I've needed to write that I > simply haven't been able to because doing so would involve a > significant amount of extra work downloading and running package code. Where thats static (expressable for all installs using PEP-426 conditions) we will be able to do that for sdists. Where its dependent on the build environment, we can't do that today (but we already can for wheels). For illustration, consider a numpy extension, the version of numpy present in the build environment determines some of the install-requires. The sdist can build against a much wider range of numpy versions than the resulting binary can run against. > If there are dependencies that are only detectable at wheel build > time, then so be it (I'm still looking for an example, but it's clear > that the potential is there) but I'd like some way of getting at > whatever dependency information a wheel (source or binary) provides > via the PyPI JSON API - and I'd like an assurance that if dependency > information *can* be determined statically, it will be. That is exactly the intent. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Wed Oct 14 11:32:21 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Oct 2015 10:32:21 +0100 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 09:50, Robert Collins wrote: > For illustration, consider a numpy extension, the version of numpy > present in the build environment determines some of the > install-requires. The sdist can build against a much wider range of > numpy versions than the resulting binary can run against. For my purposes, having the sdist depend on "numpy" and the wheel depend on a specific version of numpy would be fine. In general, for the uses I have in mind I'm not really interested in versions, mostly just "package X depends on package Y". Think of things like scripts to build a local package repository with everything needed to install a particular set of packages. I'm not worried about getting a minimal set of files, just getting all files available for each package needed is good enough. (I know "pip wheel" is more or less what I need for this specific case, my actual requirements tend to be a bit more fiddly). >> If there are dependencies that are only detectable at wheel build >> time, then so be it (I'm still looking for an example, but it's clear >> that the potential is there) but I'd like some way of getting at >> whatever dependency information a wheel (source or binary) provides >> via the PyPI JSON API - and I'd like an assurance that if dependency >> information *can* be determined statically, it will be. > > That is exactly the intent. Excellent, thanks. Paul From solipsis at pitrou.net Wed Oct 14 12:40:48 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Oct 2015 12:40:48 +0200 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> Message-ID: <20151014124048.2cea5551@fsol> On Tue, 13 Oct 2015 09:59:05 -0700 Chris Barker wrote: > > So even is SSE2 provides little for Python itself, in the usual case, we'll > see performance hits n compiled extensions that are not compiled by > particularly smart people. Since the question is only for 32-bit builds, does this even matter? 32-bit builds on x86 generally bring you poorer performance by themselves, as the CPU has only access to 8 general-purpose registers (instead of 16) in that mode. If you want better performance (or want to efficiently address 3+GB of RAM, which is an extremely common situation nowadays), you probably want 64-bit builds. Regards Antoine. From nico.schloemer at gmail.com Tue Oct 13 19:03:34 2015 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Tue, 13 Oct 2015 17:03:34 +0000 Subject: [Distutils] rst not rendered on pypi Message-ID: Hi everyone, I'm trying to get my README.rst [1] display properly on my PyPy project [2], but for some reason it's displayed as raw text. The file was created from a README.md [3] via ``` pandoc README.md -o README.rst ``` Any hints on what may have been going wrong? Cheers, Nico [1] http://chunk.io/f/f4385eda77de4ed78f8eba8cc8d8c8b6 [2] https://pypi.python.org/pypi/voropy [3] https://github.com/nschloe/voropy/blob/master/README.md -------------- next part -------------- An HTML attachment was scrubbed... URL: From schettino72 at gmail.com Wed Oct 14 14:03:04 2015 From: schettino72 at gmail.com (Eduardo Schettino) Date: Wed, 14 Oct 2015 20:03:04 +0800 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 1:03 AM, Nico Schl?mer wrote: > Hi everyone, > > I'm trying to get my README.rst [1] display properly on my PyPy project > [2], but for some reason it's displayed as raw text. The file was created > from a README.md [3] via > ``` > pandoc README.md -o README.rst > ``` > Any hints on what may have been going wrong? > > You can check it using: https://pypi.python.org/pypi/readme -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 14 13:43:10 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 14 Oct 2015 14:43:10 +0300 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: This looks suspicious: *A twice-folded M??bius strip, created with VoroPy's ``moebius_tri -i 2 out.e``. Visualization with `ParaView `__.* Regardless, you should use https://pypi.python.org/pypi/readme to check it (that's what PyPI uses for rendering). Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Tue, Oct 13, 2015 at 8:03 PM, Nico Schl?mer wrote: > Hi everyone, > > I'm trying to get my README.rst [1] display properly on my PyPy project > [2], but for some reason it's displayed as raw text. The file was created > from a README.md [3] via > ``` > pandoc README.md -o README.rst > ``` > Any hints on what may have been going wrong? > > Cheers, > Nico > > > [1] http://chunk.io/f/f4385eda77de4ed78f8eba8cc8d8c8b6 > [2] https://pypi.python.org/pypi/voropy > [3] https://github.com/nschloe/voropy/blob/master/README.md > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nico.schloemer at gmail.com Wed Oct 14 16:19:22 2015 From: nico.schloemer at gmail.com (=?UTF-8?Q?Nico_Schl=C3=B6mer?=) Date: Wed, 14 Oct 2015 14:19:22 +0000 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: Exactly what I needed. Thanks! --Nico On Wed, Oct 14, 2015 at 2:03 PM Eduardo Schettino wrote: > > On Wed, Oct 14, 2015 at 1:03 AM, Nico Schl?mer > wrote: > >> Hi everyone, >> >> I'm trying to get my README.rst [1] display properly on my PyPy project >> [2], but for some reason it's displayed as raw text. The file was created >> from a README.md [3] via >> ``` >> pandoc README.md -o README.rst >> ``` >> Any hints on what may have been going wrong? >> >> You can check it using: https://pypi.python.org/pypi/readme > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From randy at thesyrings.us Wed Oct 14 16:29:37 2015 From: randy at thesyrings.us (Randy Syring) Date: Wed, 14 Oct 2015 10:29:37 -0400 Subject: [Distutils] rst not rendered on pypi In-Reply-To: References: Message-ID: <561E66D1.9060907@thesyrings.us> I've also found restview (https://pypi.python.org/pypi/restview) very easy to use. I believe it uses readme behind the scenes (at least, it's included as a dependency in restviews setup.py). *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 10/14/2015 10:19 AM, Nico Schl?mer wrote: > Exactly what I needed. Thanks! > > --Nico > > On Wed, Oct 14, 2015 at 2:03 PM Eduardo Schettino > > wrote: > > > On Wed, Oct 14, 2015 at 1:03 AM, Nico Schl?mer > > wrote: > > Hi everyone, > > I'm trying to get my README.rst [1] display properly on my > PyPy project [2], but for some reason it's displayed as raw > text. The file was created from a README.md [3] via > ``` > pandoc README.md -o README.rst > ``` > Any hints on what may have been going wrong? > > You can check it using: https://pypi.python.org/pypi/readme > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 14 16:35:28 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Oct 2015 00:35:28 +1000 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 14 October 2015 at 12:40, Robert Collins wrote: > On 14 October 2015 at 15:04, Wes Turner wrote: >> >> On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: > >> * egg-info and dist-info should generate JSONLD > > The .egg-info and .dist-info directories are existing defined formats, > I don't see any way that they can be retroactively defined as being > JSONLD. I understand you have a JSONLD spec being discussed - this > would be a successor to PEP-426, and Nick has put the breaks on all > such things until we *catch up* with the already designed and defined > work - so I'm not sure where your spec fits in things: but even > assuming its all approved, we still can't change the meaning of > existing in the wild artifacts. I finally found time to investigate JSON-LD a while back, and I think there's potentially value in having the formal specification of our metadata formats be in the form of a JSON-LD context definition: https://github.com/pypa/interoperability-peps/issues/31 However, doing that doesn't actually solve any immediate problems for us, as none of our tools are JSON-LD aware, they're all based on ad hoc JSON processing. Thus it's firmly in the "might be nice to do" category for me, rather than "we need to do this as an urgent priority". One thing we do need to be careful of is the fact that PEP 426 is still a draft spec - if there are things we want to formalise *now* in a JSON format aimed at the *current* capabilities of the ecosystem, we may want to push some of the ideas like metadata extensions and the finer granularity of dependencies out to a more speculative metadata 3.0 spec, and descope 2.0 to a "only incremental additions to current capabilities, wrapped up in a better defined format" effort. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Wed Oct 14 18:25:43 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 09:25:43 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <20151014124048.2cea5551@fsol> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> <20151014124048.2cea5551@fsol> Message-ID: On Wed, Oct 14, 2015 at 3:40 AM, Antoine Pitrou wrote: > On Tue, 13 Oct 2015 09:59:05 -0700 > Chris Barker wrote: > > > > So even is SSE2 provides little for Python itself, in the usual case, > we'll > > see performance hits n compiled extensions that are not compiled by > > particularly smart people. > > Since the question is only for 32-bit builds, IS that the case: """ Note that my recently retired computer was 64 bit and had SSE but didn't have SSE2 (I'm fairly sure - CPU was some budget AMD model) """ granted, such machines are probably really really rare, but maybe it does matter for 64 bit, too? > does this even matter? > 32-bit builds on x86 generally bring you poorer performance by > themselves, If a user has a 32 bit machine, they have no choice -- we could argue that anyone for whom performance matters probably isn't running an old, cheap machine, but still... > If you want better performance (or want > to efficiently address 3+GB of RAM, which is an extremely common > situation nowadays), you probably want 64-bit builds. > I thought that wasn't the case (well it certainly is if you want to access that memory), but for raw performance, 64 bit is pushing a lot more memory, in the form of larger pointers, around -- so is sometimes slower. I'm sure it depends on the use-case, as it always does.... Also a note -- I'm lost on the details, but the most recent numpy release required SSE2: """ * Compiling with msvc9 or msvc10 for 32 bit Windows now requires SSE2. This was the easiest fix for what looked to be some miscompiled code when SSE2 was not used. If you need to compile for 32 bit Windows systems without SSE2 support, mingw32 should still work. """ and that's not py3.5 (msvc9 is the py2.7 compiler, and msvc10 is py3.4, yes???) -Chris > > Regards > > Antoine. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 14 18:43:25 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 09:43:25 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames Message-ID: Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 14 18:54:57 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Oct 2015 18:54:57 +0200 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> <20151014124048.2cea5551@fsol> Message-ID: <20151014185457.1e6c63b9@fsol> On Wed, 14 Oct 2015 09:25:43 -0700 Chris Barker wrote: > On Wed, Oct 14, 2015 at 3:40 AM, Antoine Pitrou wrote: > > > On Tue, 13 Oct 2015 09:59:05 -0700 > > Chris Barker wrote: > > > > > > So even is SSE2 provides little for Python itself, in the usual case, > > we'll > > > see performance hits n compiled extensions that are not compiled by > > > particularly smart people. > > > > Since the question is only for 32-bit builds, > > IS that the case: > """ > Note that my recently retired computer was 64 bit and had SSE but didn't > have SSE2 (I'm fairly sure - CPU was some budget AMD model) > """ > > granted, such machines are probably really really rare, but maybe it does > matter for 64 bit, too? Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which was originally devised by AMD), so I'm a bit skeptical about a SSE2-less 64-bit CPU. Do you have any reference? > > does this even matter? > > 32-bit builds on x86 generally bring you poorer performance by > > themselves, > > If a user has a 32 bit machine, they have no choice -- we could argue that > anyone for whom performance matters probably isn't running an old, cheap > machine, but still... The actual question is whether we want to introduce a significant amount of complexity and overhead for packagers and distributors, for the benefit of an extremely small users demography that will probably disappear altogether in a couple of years. Regards Antoine. From fungi at yuggoth.org Wed Oct 14 19:00:49 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 14 Oct 2015 17:00:49 +0000 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: <20151014170049.GQ4731@yuggoth.org> On 2015-10-14 09:43:25 -0700 (-0700), Chris Barker wrote: [...] > I propose that PyPi allow projects to replace existing files if > they REALLY REALLY want to. > > You should have to jump through all sorts of hoops, and make it > really clear that it is a BAD IDEA in the general case, but it'd > be good to have it be possible. [...] It used to be possible. If that possibility returns, I'm not looking forward to more of the bugs I used to semi-regularly file explaining to package authors why their assumptions about the safety of silently replacing their broken releases were flawed. -- Jeremy Stanley From contact at ionelmc.ro Wed Oct 14 19:04:21 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 14 Oct 2015 20:04:21 +0300 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker wrote: > some packages were unable to work with the postN suffix. ?Can you elaborate a bit more here?? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 14 19:06:25 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 14 Oct 2015 12:06:25 -0500 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On Oct 14, 2015 3:33 AM, "Paul Moore" wrote: > > On 14 October 2015 at 01:46, Robert Collins wrote: > > On 14 October 2015 at 13:25, Marcus Smith wrote: > >> > >> > >> thanks for the summary! > > Agreed. > > >> > >>> * Things that have reason to change (deps) are more reasonable to be > >>> dynamic (even with PEP-426 markers there are exceptions) > >> > >> > >> as we know, for *many* cases, run-time deps aren't dynamic. > >> is there a consensus for those cases? exist in the sdist metadata? or no? > > > > The plan we hashed out (this would be the new PEP on static metadata in sdists) > > - pip etc to lint and error if they encounter a malformed dist-info in an sdist > > - then start putting dist-info, not egg-info into sdists > > - for any field if the field is entirely absent, we'll get it by > > running the dist-info build-tool command I described in the other > > mails so, validate and transform to OrderedDicts which can then be json.dump'd to metadata.jsonld? > > > > Concretely: > > {'build_requires': []} -> no build requirements > > {} -> get build requirements by running the build system > > One use case that I don't think is covered here is publishing > dependency metadata via PyPI. I believe distlib needs this for some > functions (at the moment, I think it uses an externally hosted set of > package dependency data that's maintained by Vinay), and I know there > have been a number of utility scripts I've needed to write that I > simply haven't been able to because doing so would involve a > significant amount of extra work downloading and running package code. > > If there are dependencies that are only detectable at wheel build > time, then so be it (I'm still looking for an example, but it's clear > that the potential is there) but I'd like some way of getting at > whatever dependency information a wheel (source or binary) provides > via the PyPI JSON API - and I'd like an assurance that if dependency > information *can* be determined statically, it will be. * the ship has already sailed on a static declarative dependency model (because 'if sys.platform:' in setup.py * I agree that we should preserve all (dynamically determine at setup.py runtime) requires edges as static JSON-LD > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 14 19:36:31 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 14 Oct 2015 10:36:31 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Oct 14, 2015 10:04 AM, "Ionel Cristian M?rie?" wrote: > > > On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker wrote: >> >> some packages were unable to work with the postN suffix. > > > ?Can you elaborate a bit more here?? > Apparently some packages were making assumptions about the format of the numpy.__version__ string, and having .postN in there caused errors when they tried to process it. (It would be helpful if there were a little permissively licensed standalone implementation of PEP 440 comparisons, suitable for the "if pkg.version > ...:" checks that people insist on doing -- I couldn't find one in some quick searches.) IIUC, the specific problems numpy ran into that caused the creation of .postN releases were: - oops, didn't sign the uploads, re-upload identical file with proper signature attached -> not allowed. (I'm not sure if these were embedded or detached signatures. Either way it'd be nice if pypi allowed it, but for embedded signatures in particular I can see how this might be a hassle.) - our OS X maintainer tried to use twine to upload OS X wheels for the existing release; instead it created a new release. Not sure if a bug was filed on twine, but if not then one probably should be. As a workaround our release docs now say "always upload wheels by hand using the web interface, never use setup.py upload or twine". My feeling is that pypi is correct to disallow the mutation of releases once they become public, but that the ergonomics around this could probably be improved :-). A more general solution that might be nice to have Someday would be if you could upload a release in one step, and then get a private link to poke at what was uploaded and make sure it looks correct, before making it public in a second step. CC'ing the release manager and OS X maintainer in question, in case I got something wrong or more details are wanted... -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Oct 14 19:39:08 2015 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 14 Oct 2015 12:39:08 -0500 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Oct 14, 2015 12:05 PM, "Ionel Cristian M?rie?" wrote: > > > On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker wrote: >> >> some packages were unable to work with the postN suffix. > > > ?Can you elaborate a bit more here?? AFAIU, the community best practice here is to issue a new release with a bumped version number/pre/post/releasecode. * https://www.python.org/dev/peps/pep-0440/#version-scheme * https://github.com/peritus/bumpversion * http://docs.openstack.org/developer/pbr/semver.html#changes-vs-semver-2-0 MAJOR.MINOR.PATCH The reasons for this are many: * HTTP caches * local pip/buildout pkg caches * it is actually a different resource, with a different URI (repo,pkg,version,[pkgtype,platform,buildtags]) because it contains different files > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 14 19:59:28 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 10:59:28 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: <20151014185457.1e6c63b9@fsol> References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> <20151014124048.2cea5551@fsol> <20151014185457.1e6c63b9@fsol> Message-ID: On Wed, Oct 14, 2015 at 9:54 AM, Antoine Pitrou wrote: > > IS that the case: > > """ > > Note that my recently retired computer was 64 bit and had SSE but didn't > > have SSE2 (I'm fairly sure - CPU was some budget AMD model) > > """ > > > > granted, such machines are probably really really rare, but maybe it does > > matter for 64 bit, too? > > Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which > was originally devised by AMD), so I'm a bit skeptical about a > SSE2-less 64-bit CPU. Do you have any reference? That was a quote from this thread... I have no idea beyond that. > > > does this even matter? > > > 32-bit builds on x86 generally bring you poorer performance by > > > themselves, > > > > If a user has a 32 bit machine, they have no choice -- we could argue > that > > anyone for whom performance matters probably isn't running an old, cheap > > machine, but still... > > The actual question is whether we want to introduce a significant > amount of complexity and overhead for packagers and distributors, for > the benefit of an extremely small users demography that will probably > disappear altogether in a couple of years. > Actually, I thought you were advocating the opposite: SSE2 on 64bit -- cause 64 bit machines without SSE2 are really rare, if they exist at all. no SSE2 on 32 bit -- because anyone to whom performance really matters won't be running an old 32bit machine. Seems reasonable to me. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 14 20:12:55 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 14:12:55 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 1:37:05 PM, Nathaniel Smith (njs at pobox.com) wrote: On Oct 14, 2015 10:04 AM, "Ionel Cristian M?rie?" wrote: > > > On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker wrote: >> >> some packages were unable to work with the postN suffix. > > > ?Can you elaborate a bit more here?? > Apparently some packages were making assumptions about the format of the numpy.__version__ string, and having .postN in there caused errors when they tried to process it. (It would be helpful if there were a little permissively licensed standalone implementation of PEP 440 comparisons, suitable for the "if pkg.version > ...:" checks that people insist on doing -- I couldn't find one in some quick searches.) https://github.com/pypa/packaging It?s what both pip and setuptools use (though we embed it, but it?s fine to depend on it too). IIUC, the specific problems numpy ran into that caused the creation of .postN releases were: - oops, didn't sign the uploads, re-upload identical file with proper signature attached -> not allowed. (I'm not sure if these were embedded or detached signatures. Either way it'd be nice if pypi allowed it, but for embedded signatures in particular I can see how this might be a hassle.) I don?t think we allow embedded signatures, it would be reasonable to allow uploading detached signatures after the fact though. - our OS X maintainer tried to use twine to upload OS X wheels for the existing release; instead it created a new release. Not sure if a bug was filed on twine, but if not then one probably should be. As a workaround our release docs now say "always upload wheels by hand using the web interface, never use setup.py upload or twine". This shouldn?t create a new release unless you?ve changed the version number (including adding post releases). If you can reproduce on Test PyPI I can fix it. My feeling is that pypi is correct to disallow the mutation of releases once they become public, but that the ergonomics around this could probably be improved :-). A more general solution that might be nice to have Someday would be if you could upload a release in one step, and then get a private link to poke at what was uploaded and make sure it looks correct, before making it public in a second step. The ergonomics of it is pretty bad. Once we kill PyPI legacy and Warehouse is the thing we?re using, improving things like that will be a lot simpler. CC'ing the release manager and OS X maintainer in question, in case I got something wrong or more details are wanted... -n _______________________________________________? Distutils-SIG maillist - Distutils-SIG at python.org? https://mail.python.org/mailman/listinfo/distutils-sig? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 20:16:17 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 15 Oct 2015 07:16:17 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 15 October 2015 at 03:35, Nick Coghlan wrote: > On 14 October 2015 at 12:40, Robert Collins wrote: >> On 14 October 2015 at 15:04, Wes Turner wrote: >>> >>> On Oct 13, 2015 7:50 PM, "Robert Collins" wrote: >> >>> * egg-info and dist-info should generate JSONLD >> >> The .egg-info and .dist-info directories are existing defined formats, >> I don't see any way that they can be retroactively defined as being >> JSONLD. I understand you have a JSONLD spec being discussed - this >> would be a successor to PEP-426, and Nick has put the breaks on all >> such things until we *catch up* with the already designed and defined >> work - so I'm not sure where your spec fits in things: but even >> assuming its all approved, we still can't change the meaning of >> existing in the wild artifacts. > > I finally found time to investigate JSON-LD a while back, and I think > there's potentially value in having the formal specification of our > metadata formats be in the form of a JSON-LD context definition: > https://github.com/pypa/interoperability-peps/issues/31 I can see potential value too, but the current conversations are not revising our existing metadata formats - in fact, I think its essential for quick deployment (which we need, because decades are too long) that we don't trigger cascading work across the consumers of existing formats. > However, doing that doesn't actually solve any immediate problems for > us, as none of our tools are JSON-LD aware, they're all based on ad > hoc JSON processing. Thus it's firmly in the "might be nice to do" > category for me, rather than "we need to do this as an urgent > priority". > > One thing we do need to be careful of is the fact that PEP 426 is > still a draft spec - if there are things we want to formalise *now* in > a JSON format aimed at the *current* capabilities of the ecosystem, we > may want to push some of the ideas like metadata extensions and the > finer granularity of dependencies out to a more speculative metadata > 3.0 spec, and descope 2.0 to a "only incremental additions to current > capabilities, wrapped up in a better defined format" effort. I believe the '426 is draft' ship has sailed: metadata.json is present in .dist-info in wheels and has been for some time. (e.g. see https://pypi.python.org/packages/2.7/r/reddit_bot/reddit_bot-0.1.2-py2.py3-none-any.whl#md5=86518fc20388c1d8e567cf2d4cfe1a03 ). As such we're going to need to take the 'compatible changes to it in-place, incompatible changes need a new schema version' approach IMO. Otherwise, when we start consuming it, we'll be reading draft versions and assigning different meanings. I think future specs should issue the final version in the schema only when the PEP is made non-draft. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From dholth at gmail.com Wed Oct 14 20:22:25 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Oct 2015 18:22:25 +0000 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: > > I believe the '426 is draft' ship has sailed: metadata.json is present > in .dist-info in wheels and has been for some time. (e.g. see > > https://pypi.python.org/packages/2.7/r/reddit_bot/reddit_bot-0.1.2-py2.py3-none-any.whl#md5=86518fc20388c1d8e567cf2d4cfe1a03 > ). As such we're going to need to take the 'compatible changes to it > in-place, incompatible changes need a new schema version' approach > IMO. > > Otherwise, when we start consuming it, we'll be reading draft versions > and assigning different meanings. > > I think future specs should issue the final version in the schema only > when the PEP is made non-draft. > bdist_wheel intentionally generates metadata.json but the PEP specifies pydist.json. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 14 20:25:27 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 14 Oct 2015 11:25:27 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote: > [...] >> Apparently some packages were making assumptions about the format of the numpy.__version__ string, and having .postN in there caused errors when they tried to process it. (It would be helpful if there were a little permissively licensed standalone implementation of PEP 440 comparisons, suitable for the "if pkg.version > ...:" checks that people insist on doing -- I couldn't find one in some quick searches.) > > https://github.com/pypa/packaging > > It?s what both pip and setuptools use (though we embed it, but it?s fine to depend on it too). That's under Apache 2, so it can't be used by GPLv2 packages, or any package that might be used by GPLv2 packages. >> >> IIUC, the specific problems numpy ran into that caused the creation of .postN releases were: >> - oops, didn't sign the uploads, re-upload identical file with proper signature attached -> not allowed. (I'm not sure if these were embedded or detached signatures. Either way it'd be nice if pypi allowed it, but for embedded signatures in particular I can see how this might be a hassle.) > > I don?t think we allow embedded signatures, it would be reasonable to allow uploading detached signatures after the fact though. >> >> >> - our OS X maintainer tried to use twine to upload OS X wheels for the existing release; instead it created a new release. Not sure if a bug was filed on twine, but if not then one probably should be. As a workaround our release docs now say "always upload wheels by hand using the web interface, never use setup.py upload or twine". > > This shouldn?t create a new release unless you?ve changed the version number (including adding post releases). If you can reproduce on Test PyPI I can fix it. Matthew? Any thoughts? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 14 20:27:39 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 14:27:39 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote: > > > [...] > >> Apparently some packages were making assumptions about the format of the > numpy.__version__ string, and having .postN in there caused errors when > they tried to process it. (It would be helpful if there were a little > permissively licensed standalone implementation of PEP 440 comparisons, > suitable for the "if pkg.version > ...:" checks that people insist on doing > -- I couldn't find one in some quick searches.) > > > > https://github.com/pypa/packaging > > > > It?s what both pip and setuptools use (though we embed it, but it?s fine > to depend on it too). > > That's under Apache 2, so it can't be used by GPLv2 packages, or any > package that might be used by GPLv2 packages. I suspect it?d be trivial to relicense it. There?s a total of 6 contributors and I think I know how to get ahold of all of them. > > >> > >> IIUC, the specific problems numpy ran into that caused the creation of > .postN releases were: > >> - oops, didn't sign the uploads, re-upload identical file with proper > signature attached -> not allowed. (I'm not sure if these were embedded or > detached signatures. Either way it'd be nice if pypi allowed it, but for > embedded signatures in particular I can see how this might be a hassle.) > > > > I don?t think we allow embedded signatures, it would be reasonable to > allow uploading detached signatures after the fact though. > >> > >> > >> - our OS X maintainer tried to use twine to upload OS X wheels for the > existing release; instead it created a new release. Not sure if a bug was > filed on twine, but if not then one probably should be. As a workaround our > release docs now say "always upload wheels by hand using the web interface, > never use setup.py upload or twine". > > > > This shouldn?t create a new release unless you?ve changed the version > number (including adding post releases). If you can reproduce on Test PyPI > I can fix it. > > Matthew? Any thoughts? > > -n > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From robertc at robertcollins.net Wed Oct 14 20:35:14 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 15 Oct 2015 07:35:14 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 15 October 2015 at 06:06, Wes Turner wrote: > > On Oct 14, 2015 3:33 AM, "Paul Moore" wrote: >> >> On 14 October 2015 at 01:46, Robert Collins >> wrote: >> > On 14 October 2015 at 13:25, Marcus Smith wrote: >> >> >> >> >> >> thanks for the summary! >> >> Agreed. >> >> >> >> >>> * Things that have reason to change (deps) are more reasonable to be >> >>> dynamic (even with PEP-426 markers there are exceptions) >> >> >> >> >> >> as we know, for *many* cases, run-time deps aren't dynamic. >> >> is there a consensus for those cases? exist in the sdist metadata? or >> >> no? >> > >> > The plan we hashed out (this would be the new PEP on static metadata in >> > sdists) >> > - pip etc to lint and error if they encounter a malformed dist-info in >> > an sdist >> > - then start putting dist-info, not egg-info into sdists >> > - for any field if the field is entirely absent, we'll get it by >> > running the dist-info build-tool command I described in the other >> > mails > > so, validate and transform to OrderedDicts which can then be json.dump'd to > metadata.jsonld? That would be one strategy going forward for utilising jsonld. However, *this current effort* is not altering the existing definitions of metadata that we have today - and doing so would effectively put the initiative to make things like flit have an interopability standard back 5-10 **years** [because the half life of enterprise software environments is ridiculously long). >> > >> > Concretely: >> > {'build_requires': []} -> no build requirements >> > {} -> get build requirements by running the build system >> >> One use case that I don't think is covered here is publishing >> dependency metadata via PyPI. I believe distlib needs this for some >> functions (at the moment, I think it uses an externally hosted set of >> package dependency data that's maintained by Vinay), and I know there >> have been a number of utility scripts I've needed to write that I >> simply haven't been able to because doing so would involve a >> significant amount of extra work downloading and running package code. >> > >> If there are dependencies that are only detectable at wheel build >> time, then so be it (I'm still looking for an example, but it's clear >> that the potential is there) but I'd like some way of getting at >> whatever dependency information a wheel (source or binary) provides >> via the PyPI JSON API - and I'd like an assurance that if dependency >> information *can* be determined statically, it will be.----------* > > * the ship has already sailed on a static declarative dependency model > (because 'if sys.platform:' in setup.py Our strategy here is to provide newer features and encourage folk to adopt them. Where the feature will work correctly in pre-its-existence environments, we can expect better uptake than where a flag day is required (and we get back into that decade long wait...). PEP-426 conditionals address most (but not all) of the dynamic dependencies cases. We can't remove dynamic dependencies until a) -all- the cases for things we want on PyPI are solved and b) every single package that needs it has migrated to use the new feature. We haven't delivered (a) yet, so we have to design any initiatives that we want to be available now, to work with dynamic dependencies: but its not a statement that we've stopped pushing for static expression of them all. > * I agree that we should preserve all (dynamically determine at setup.py > runtime) requires edges as static JSON-LD There's an existing metadata format: PEP-426. Changing that would be a different discussion, even though its technically draft, because changing it imposes adoption and interopability challenges. Finally I don't see any new data schemas in the proposed bootstrap mechanisms - PEP-426 already covers requirement specifiers, and the command line string templates we need are trivial. so - I don't see anything that JSON-LD will help us with at this point - other than introducing churn, when minimising the impact (to permit adoption) is an explicit design point. I don't know about the other contributors, but I don't have enough bandwidth to tackle many proposals at once (and do them justice) - so at least from my part, I want to defer any and all JSON-LD discussion until we're making non-trivial changes to PEP-426. E.g. not for metadata 2.1 (which is what I expect the fixed handling of python 2.7.10 and markers to end up in), but for metadata 3.x. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From donald at stufft.io Wed Oct 14 20:35:43 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 14:35:43 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 2:27:40 PM, Donald Stufft (donald at stufft.io) wrote: > On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > > On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote: > > > > > [...] > > >> Apparently some packages were making assumptions about the format of the > > numpy.__version__ string, and having .postN in there caused errors when > > they tried to process it. (It would be helpful if there were a little > > permissively licensed standalone implementation of PEP 440 comparisons, > > suitable for the "if pkg.version > ...:" checks that people insist on doing > > -- I couldn't find one in some quick searches.) > > > > > > https://github.com/pypa/packaging > > > > > > It?s what both pip and setuptools use (though we embed it, but it?s fine > > to depend on it too). > > > > That's under Apache 2, so it can't be used by GPLv2 packages, or any > > package that might be used by GPLv2 packages. > > I suspect it?d be trivial to relicense it. There?s a total of 6 contributors and I think > I know how to get ahold of all of them. https://github.com/pypa/packaging/issues/42 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From robertc at robertcollins.net Wed Oct 14 20:36:00 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 15 Oct 2015 07:36:00 +1300 Subject: [Distutils] Meeting info re: sdists In-Reply-To: References: Message-ID: On 15 October 2015 at 07:22, Daniel Holth wrote: >> I believe the '426 is draft' ship has sailed: metadata.json is present >> in .dist-info in wheels and has been for some time. (e.g. see >> >> https://pypi.python.org/packages/2.7/r/reddit_bot/reddit_bot-0.1.2-py2.py3-none-any.whl#md5=86518fc20388c1d8e567cf2d4cfe1a03 >> ). As such we're going to need to take the 'compatible changes to it >> in-place, incompatible changes need a new schema version' approach >> IMO. >> >> Otherwise, when we start consuming it, we'll be reading draft versions >> and assigning different meanings. >> >> I think future specs should issue the final version in the schema only >> when the PEP is made non-draft. > > > bdist_wheel intentionally generates metadata.json but the PEP specifies > pydist.json. So that was to stop folk consuming the draft data? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Wed Oct 14 20:36:44 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Oct 2015 19:36:44 +0100 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On 14 October 2015 at 19:27, Donald Stufft wrote: > I suspect it?d be trivial to relicense it. There?s a total of 6 contributors and I think I know how to get ahold of all of them. I only contributed a tiny bit, but I'm happy for you to relicense if you want to. Paul From robertc at robertcollins.net Wed Oct 14 21:05:54 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 15 Oct 2015 08:05:54 +1300 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On 15 October 2015 at 07:27, Donald Stufft wrote: > On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (njs at pobox.com) wrote: >> On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote: >> > >> [...] >> >> Apparently some packages were making assumptions about the format of the >> numpy.__version__ string, and having .postN in there caused errors when >> they tried to process it. (It would be helpful if there were a little >> permissively licensed standalone implementation of PEP 440 comparisons, >> suitable for the "if pkg.version > ...:" checks that people insist on doing >> -- I couldn't find one in some quick searches.) >> > >> > https://github.com/pypa/packaging >> > >> > It?s what both pip and setuptools use (though we embed it, but it?s fine >> to depend on it too). >> >> That's under Apache 2, so it can't be used by GPLv2 packages, or any >> package that might be used by GPLv2 packages. > > I suspect it?d be trivial to relicense it. There?s a total of 6 contributors and I think I know how to get ahold of all of them. Presumably to MIT + ApacheV2 ? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From donald at stufft.io Wed Oct 14 21:06:57 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 15:06:57 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 3:05:57 PM, Robert Collins (robertc at robertcollins.net) wrote: > On 15 October 2015 at 07:27, Donald Stufft wrote: > > On October 14, 2015 at 2:25:31 PM, Nathaniel Smith (njs at pobox.com) wrote: > >> On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote: > >> > > >> [...] > >> >> Apparently some packages were making assumptions about the format of the > >> numpy.__version__ string, and having .postN in there caused errors when > >> they tried to process it. (It would be helpful if there were a little > >> permissively licensed standalone implementation of PEP 440 comparisons, > >> suitable for the "if pkg.version > ...:" checks that people insist on doing > >> -- I couldn't find one in some quick searches.) > >> > > >> > https://github.com/pypa/packaging > >> > > >> > It?s what both pip and setuptools use (though we embed it, but it?s fine > >> to depend on it too). > >> > >> That's under Apache 2, so it can't be used by GPLv2 packages, or any > >> package that might be used by GPLv2 packages. > > > > I suspect it?d be trivial to relicense it. There?s a total of 6 contributors and I think > I know how to get ahold of all of them. > > Presumably to MIT + ApacheV2 ? 2 Clause BSD and Apache v2.0. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Wed Oct 14 21:52:16 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 12:52:16 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac wrote: > This was discussed recently here: > https://github.com/pypa/packaging-problems/issues/74 > and on this list at other times. Though the above issue was pretty focused on restoring a deleted file without any changes -- which seems like a no-brainer to me, as long as someone wants to put the effort into the infrastructure. (the soft delete option seems like a good idea to me). But I'm talking about the cases of "whoops! I really wish I hadn't uploaded that one". We can improve the tooling (some discussion on this in this thread right now...), but people are people and some of us are stupid and/or careless. So this WILL happen. And it just seems pedantic to say: hey -- you've already put that one there -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If it happens quickly, then no one has downloaded it, it hasn't made its way to the mirrors, etc... And again -- we are all adults here: if you as the package maintainer want to do somethign that is sonfusing to your users, is it up to PyPi to never let you do that? (I think it IS up to PyPi to strongly recommend that you don't -- i.e. make it hard to do, and impossible to do thoughtlessly) On Wed, Oct 14, 2015 at 10:00 AM, Jeremy Stanley wrote: > > You should have to jump through all sorts of hoops, and make it > > really clear that it is a BAD IDEA in the general case, but it'd > > be good to have it be possible. > [...] > > It used to be possible. Was it also easy to do without very careful consideration? Or were the hoops I mentioned in place? I can't find it right now, but I think someone in this thread suggested a "staging area", so we could essentially do a trail run: upload to PyPi, tell a few trusted friends about it, have them test it, then, and only then, push it to the official channel. Maybe the infrastructure for that would be more work than it's worth, but I like it. This would fit into the basic principle that you should always be able to test something in as close the final environment as possible, before you commit to it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 14 22:04:54 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 16:04:54 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 3:58:52 PM, Chris Barker (chris.barker at noaa.gov) wrote: > On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac > wrote: > > > This was discussed recently here: > > https://github.com/pypa/packaging-problems/issues/74 > > > > and on this list at other times. Though the above issue was pretty focused > on restoring a deleted file without any changes -- which seems like a > no-brainer to me, as long as someone wants to put the effort into the > infrastructure. > > (the soft delete option seems like a good idea to me). I plan on implementing a soft delete in Warehouse. > > But I'm talking about the cases of "whoops! I really wish I hadn't uploaded > that one". We can improve the tooling (some discussion on this in this > thread right now...), but people are people and some of us are stupid > and/or careless. So this WILL happen. > > And it just seems pedantic to say: hey -- you've already put that one there > -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If > it happens quickly, then no one has downloaded it, it hasn't made its way > to the mirrors, etc? Generally within 60-120 seconds it?s available in mirrors (most of them resync once a minute). If anyone has downloaded it then they will have pretty much permanently cached the package, first in the download cache and then again in the wheel cache (assuming it wasn?t a wheel already, and they had that enabled). The original package was NumPy. It had?30,982 downloads in the last day, so we can average that out to 1290 downloads an hour or 21 downloads a minute. If it takes you two minutes to notice it and delete it, then there are ~40 people who already have the original version cached and who will not notice the updated version. Version numbers are free, use more of them. If you can?t just issue new releases quickly, then test your release before you upload it (and then upload it with twine) and you can even upload it to Test PyPI to test things earlier than that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From oscar.j.benjamin at gmail.com Wed Oct 14 22:16:33 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 14 Oct 2015 21:16:33 +0100 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> <20151014124048.2cea5551@fsol> <20151014185457.1e6c63b9@fsol> Message-ID: On 14 Oct 2015 19:00, "Chris Barker" wrote: > > On Wed, Oct 14, 2015 at 9:54 AM, Antoine Pitrou wrote: >> >> > IS that the case: >> > """ >> > Note that my recently retired computer was 64 bit and had SSE but didn't >> > have SSE2 (I'm fairly sure - CPU was some budget AMD model) >> > """ >> > >> > granted, such machines are probably really really rare, but maybe it does >> > matter for 64 bit, too? >> >> Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which >> was originally devised by AMD), so I'm a bit skeptical about a >> SSE2-less 64-bit CPU. Do you have any reference? > > > That was a quote from this thread... I have no idea beyond that. I wrote that but now I think about it Antoine is right. SSE2 is fully supported on all x86-64 CPUs. I must have been confusing my old home computer with my old work computer (which was 32 bit and ran XP). No way to check now though... The problem with SSE2 may go away soon but then we have the problem with SSE3 and so on. Most of us have CPUs that support instruction sets beyond those used in the lowest common denominator builds of Python provided in the Windows binaries (and distro binaries etc). Likewise for extension modules on PyPI. Numpy's Windows installer bundles several BLAS binaries with different levels of SSE and this was the initial reason for not providing Windows wheels. The problem is being solved though by switching from ATLAS to OpenBLAS which selects different levels of SSE at runtime. Maybe that approach (runtime machine code selection) is the only way to marry the needs of packaging with the desire to fully utilise CPU capabilities. It keeps the packaging side simple at the expense of pushing the complexity onto the project authors. Though it's probably only viable for something like a BLAS library which would often contain a load of hand crafted assembly code anyway. -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 14 22:20:01 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 14 Oct 2015 23:20:01 +0300 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 11:04 PM, Donald Stufft wrote: > Generally within 60-120 seconds it?s available in mirrors (most of them > resync once a minute). If anyone has downloaded it then they will have > pretty much permanently cached the package, first in the download cache and > then again in the wheel cache (assuming it wasn?t a wheel already, and they > had that enabled). The original package was NumPy. It had 30,982 downloads > in the last day, so we can average that out to 1290 downloads an hour or 21 > downloads a minute. If it takes you two minutes to notice it and delete it, > then there are ~40 people who already have the original version cached and > who will not notice the updated version. > ?This reminds me of Gmail's "unsend" feature where email would be delayed 10 seconds or something, giving a window to press the unsend button. Maybe something like that could be implemented? Like a minute or two in which you could unpublish? And a --no-regrets mode of course?, for people that want to live the moment :-) Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Oct 14 22:59:25 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 14 Oct 2015 21:59:25 +0100 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 8:52 PM, Chris Barker wrote: > > > On Wed, Oct 14, 2015 at 9:56 AM, Dave Forgac > wrote: > >> This was discussed recently here: >> https://github.com/pypa/packaging-problems/issues/74 >> > > and on this list at other times. Though the above issue was pretty focused > on restoring a deleted file without any changes -- which seems like a > no-brainer to me, as long as someone wants to put the effort into the > infrastructure. > > (the soft delete option seems like a good idea to me). > > But I'm talking about the cases of "whoops! I really wish I hadn't > uploaded that one". We can improve the tooling (some discussion on this in > this thread right now...), but people are people and some of us are stupid > and/or careless. So this WILL happen. > > And it just seems pedantic to say: hey -- you've already put that one > there -- maybe even two minutes ago, so there is NO WAY to fix your > mistake. If it happens quickly, then no one has downloaded it, it hasn't > made its way to the mirrors, etc... > It is not pendantic, for reasons mentioned by Donald. I have not done numpy releases for half a decade now, but it was already automated enough that putting a new version was not very costly then. And you did not have travis-ci, appveyor, tox, ubiquitous AWS 5 years ago ... I am sure there are things we can do to improve numpy's release process to avoid this in the future. David > And again -- we are all adults here: if you as the package maintainer want > to do somethign that is sonfusing to your users, is it up to PyPi to never > let you do that? (I think it IS up to PyPi to strongly recommend that you > don't -- i.e. make it hard to do, and impossible to do thoughtlessly) > > On Wed, Oct 14, 2015 at 10:00 AM, Jeremy Stanley > wrote: > >> > You should have to jump through all sorts of hoops, and make it >> > really clear that it is a BAD IDEA in the general case, but it'd >> > be good to have it be possible. >> [...] >> >> It used to be possible. > > > Was it also easy to do without very careful consideration? Or were the > hoops I mentioned in place? > > I can't find it right now, but I think someone in this thread suggested a > "staging area", so we could essentially do a trail run: upload to PyPi, > tell a few trusted friends about it, have them test it, then, and only > then, push it to the official channel. Maybe the infrastructure for that > would be more work than it's worth, but I like it. > > This would fit into the basic principle that you should always be able to > test something in as close the final environment as possible, before you > commit to it. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Oct 14 23:01:45 2015 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 14 Oct 2015 14:01:45 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> > On Oct 14, 2015, at 1:04 PM, Donald Stufft wrote: > > Generally within 60-120 seconds it?s available in mirrors (most of them resync once a minute). If anyone has downloaded it then they will have pretty much permanently cached the package, first in the download cache and then again in the wheel cache (assuming it wasn?t a wheel already, and they had that enabled). The original package was NumPy. It had 30,982 downloads in the last day, so we can average that out to 1290 downloads an hour or 21 downloads a minute. If it takes you two minutes to notice it and delete it, then there are ~40 people who already have the original version cached and who will not notice the updated version. While I don't think PyPI should allow modification of uploaded packages necessarily, I do think that Pip's caching is (A) too aggressive and (B) too opaque. For example: https://github.com/pypa/pip/issues/3127 https://github.com/pypa/pip/issues/3034 https://github.com/pypa/pip/issues/3025 https://github.com/pypa/pip/issues/2908 https://github.com/pypa/pip/issues/2882 etc, etc. I know there are some directories platform-specific directories I can delete, but almost once a day I want a command like `pip cache show? which can show me what is cached and when/where it was built, `pip cache clear? or `pip cache remove twisted? or `pip cache remove cffi>=1.0?. I don't want to have to care if it's in the HTTP cache or the wheel cache, or how it got there; I also don't want to have to bust a ~200 megabyte cache that saves me hours a day just because there's one bad entry in there. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Oct 14 22:55:27 2015 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 14 Oct 2015 13:55:27 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: > On Oct 14, 2015, at 10:36 AM, Nathaniel Smith wrote: > > My feeling is that pypi is correct to disallow the mutation of releases once they become public, but that the ergonomics around this could probably be improved :-). A more general solution that might be nice to have Someday would be if you could upload a release in one step, and then get a private link to poke at what was uploaded and make sure it looks correct, before making it public in a second step. > IMHO it would be really neat if 'pip' could authenticate to PyPI, and the authentication could affect package visibility, so that you could end-to-end test 'pip install foo' and ensure that you get what you expect before pushing the go-live switch. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 23:04:18 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 15 Oct 2015 10:04:18 +1300 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On 15 October 2015 at 09:55, Glyph Lefkowitz wrote: > > On Oct 14, 2015, at 10:36 AM, Nathaniel Smith wrote: > > My feeling is that pypi is correct to disallow the mutation of releases once > they become public, but that the ergonomics around this could probably be > improved :-). A more general solution that might be nice to have Someday > would be if you could upload a release in one step, and then get a private > link to poke at what was uploaded and make sure it looks correct, before > making it public in a second step. > > IMHO it would be really neat if 'pip' could authenticate to PyPI, and the > authentication could affect package visibility, so that you could end-to-end > test 'pip install foo' and ensure that you get what you expect before > pushing the go-live switch. And if its bust for you? Surely testpypi is equally good and its an existing solution? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From donald at stufft.io Wed Oct 14 23:06:03 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 17:06:03 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: <71FE499A-6593-4BF1-9C86-5AA0F60DE0CF@stufft.io> > On Oct 14, 2015, at 4:55 PM, Glyph Lefkowitz wrote: > > >> On Oct 14, 2015, at 10:36 AM, Nathaniel Smith wrote: >> >> My feeling is that pypi is correct to disallow the mutation of releases once they become public, but that the ergonomics around this could probably be improved :-). A more general solution that might be nice to have Someday would be if you could upload a release in one step, and then get a private link to poke at what was uploaded and make sure it looks correct, before making it public in a second step. >> > > IMHO it would be really neat if 'pip' could authenticate to PyPI, and the authentication could affect package visibility, so that you could end-to-end test 'pip install foo' and ensure that you get what you expect before pushing the go-live switch. Can't be done without a pretty huge regression in uptime and speed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 14 23:07:51 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 15 Oct 2015 10:07:51 +1300 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> References: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> Message-ID: On 15 October 2015 at 10:01, Glyph Lefkowitz wrote: > > On Oct 14, 2015, at 1:04 PM, Donald Stufft wrote: > > Generally within 60-120 seconds it?s available in mirrors (most of them > resync once a minute). If anyone has downloaded it then they will have > pretty much permanently cached the package, first in the download cache and > then again in the wheel cache (assuming it wasn?t a wheel already, and they > had that enabled). The original package was NumPy. It had 30,982 downloads > in the last day, so we can average that out to 1290 downloads an hour or 21 > downloads a minute. If it takes you two minutes to notice it and delete it, > then there are ~40 people who already have the original version cached and > who will not notice the updated version. > > > While I don't think PyPI should allow modification of uploaded packages > necessarily, I do think that Pip's caching is (A) too aggressive and (B) too > opaque. For example: > > https://github.com/pypa/pip/issues/3127 > https://github.com/pypa/pip/issues/3034 > https://github.com/pypa/pip/issues/3025 > https://github.com/pypa/pip/issues/2908 > https://github.com/pypa/pip/issues/2882 > > etc, etc. > > I know there are some directories platform-specific directories I can > delete, but almost once a day I want a command like `pip cache show? which > can show me what is cached and when/where it was built, `pip cache clear? or > `pip cache remove twisted? or `pip cache remove cffi>=1.0?. I don't want to > have to care if it's in the HTTP cache or the wheel cache, or how it got > there; I also don't want to have to bust a ~200 megabyte cache that saves me > hours a day just because there's one bad entry in there. So - none of those need PEPs - just dive in and put a patch forward. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Wed Oct 14 23:17:18 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 14 Oct 2015 14:17:18 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Oct 14, 2015 1:55 PM, "Glyph Lefkowitz" wrote: > > >> On Oct 14, 2015, at 10:36 AM, Nathaniel Smith wrote: >> >> My feeling is that pypi is correct to disallow the mutation of releases once they become public, but that the ergonomics around this could probably be improved :-). A more general solution that might be nice to have Someday would be if you could upload a release in one step, and then get a private link to poke at what was uploaded and make sure it looks correct, before making it public in a second step. > > IMHO it would be really neat if 'pip' could authenticate to PyPI, and the authentication could affect package visibility, so that you could end-to-end test 'pip install foo' and ensure that you get what you expect before pushing the go-live switch. The slick way to do this would be to provide a private index URL that gives you a view of what pypi would look like after your release goes live, and could be used like pip install --index-url https://pypi.python.org/tmp/foo/acd1538afe267/ foo That way you skip all the issues of adding an authentication system to pip, plus get a URL that you can pass around to other people ("hey #twisted, can someone who's on Windows test the new release? Just type: ...") You'd want to expire these after some time to prevent them being abused as private package indices for proprietary software, but that's easy enough. @Donald: is there somewhere we should be filing these warehouse wish list things so they don't get lost? :-) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 14 23:46:17 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 17:46:17 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> References: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> Message-ID: On October 14, 2015 at 5:01:49 PM, Glyph Lefkowitz (glyph at twistedmatrix.com) wrote: > > > On Oct 14, 2015, at 1:04 PM, Donald Stufft wrote: > > > > Generally within 60-120 seconds it?s available in mirrors (most of them resync once > a minute). If anyone has downloaded it then they will have pretty much permanently cached > the package, first in the download cache and then again in the wheel cache (assuming it > wasn?t a wheel already, and they had that enabled). The original package was NumPy. It > had 30,982 downloads in the last day, so we can average that out to 1290 downloads an hour > or 21 downloads a minute. If it takes you two minutes to notice it and delete it, then there > are ~40 people who already have the original version cached and who will not notice the > updated version. > > While I don't think PyPI should allow modification of uploaded packages necessarily, > I do think that Pip's caching is (A) too aggressive and (B) too opaque. For example: > > https://github.com/pypa/pip/issues/3127 > https://github.com/pypa/pip/issues/3034 These two are the same thing, and a bug. Fix the bug and the http cache is perfectly fine IMO. It never caches without headers or an Etag saying that it can. PyPI uses really long cache-control headers (1y+) on a package, and it can do this because it can promise that this URL will either continue to exist, or it?ll just go away. There are not, AFAIK, any problems with the download cache being too aggressive other than one bug where it doesn?t correctly identify a failed download. > https://github.com/pypa/pip/issues/3025 This is a problem with the wheel cache, with a path to fix it on the ticket. Just needs someone to implement the PR. > https://github.com/pypa/pip/issues/2908 I think this is probably just a wontfix issue tbh. The cache isn?t designed to be used that way really. > https://github.com/pypa/pip/issues/2882 Another problem with the wheel cache :/ > > etc, etc. > > I know there are some directories platform-specific directories I can delete, but almost > once a day I want a command like `pip cache show? which can show me what is cached and when/where > it was built, `pip cache clear? or `pip cache remove twisted? or `pip cache remove cffi>=1.0?. > I don't want to have to care if it's in the HTTP cache or the wheel cache, or how it got there; > I also don't want to have to bust a ~200 megabyte cache that saves me hours a day just because > there's one bad entry in there. Absent that one bug, I?m not aware of any reason why someone would need to ever purge their http cache except for if they were using a non PyPI index and they sent cache headers when they shouldn?t have (which is not pip?s fault in the slightest). The wheel cache is new and still has some issues to work out. I suspect it will need some method to inspect and remove items from the cache, but that work won?t get done until someone does it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Thu Oct 15 00:02:26 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 18:02:26 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 5:17:42 PM, Nathaniel Smith (njs at pobox.com) wrote: > > @Donald: is there somewhere we should be filing these warehouse > wish list things so they don't get lost? :-) https://github.com/pypa/warehouse/issues ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Thu Oct 15 00:05:02 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 15:05:02 -0700 Subject: [Distutils] warning about potential problem for wheels In-Reply-To: References: <201510101953.t9AJrVwx007005@fido.openend.se> <201510102237.t9AMbH5e018456@fido.openend.se> <006C8BDC-9849-43DE-B786-D079FBB4AC3D@stufft.io> <20151011184407.22503ba9@fsol> <20151014124048.2cea5551@fsol> <20151014185457.1e6c63b9@fsol> Message-ID: On Wed, Oct 14, 2015 at 1:16 PM, Oscar Benjamin wrote: > Numpy's Windows installer bundles several BLAS binaries with different > levels of SSE and this was the initial reason for not providing Windows > wheels. > indeed -- but IIRC, SSE2 made a pretty big difference, but SSE3 not so much -- though, of course, who knows how fabulously useful some future extension might be... > The problem is being solved though by switching from ATLAS to OpenBLAS > which selects different levels of SSE at runtime. > > Maybe that approach (runtime machine code selection) is the only way to > marry the needs of packaging with the desire to fully utilise CPU > capabilities. It keeps the packaging side simple at the expense of pushing > the complexity onto the project authors. Though it's probably only viable > for something like a BLAS library which would often contain a load of hand > crafted assembly code anyway. > only viable ,and probably only necessary... I'd be really surprised if SSE2 made any notabel difference in regular old python code. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Oct 15 00:12:32 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 15:12:32 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 1:04 PM, Donald Stufft wrote: > > And it just seems pedantic to say: hey -- you've already put that one > there > > -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If > > it happens quickly, then no one has downloaded it, it hasn't made its way > > to the mirrors, etc? > > Generally within 60-120 seconds it?s available in mirrors (most of them > resync once a minute). interesting -- it doesn't show up in the PyPi search nearly that fast :-) If anyone has downloaded it Honestly, I'm really not worried about people having downloaded it -- but if it gets mirrored almost instantaneously, then there's a problem. > The original package was NumPy. It had 30,982 downloads in the last day, well, let's face it, most packages are not nearly as popular as numpy... But again, if it's been mirrored, we're stuck. > Version numbers are free, use more of them. If you can?t just issue new > releases quickly, then test your release before you upload it (and then > upload it with twine) and you can even upload it to Test PyPI to test > things earlier than that. Bingo! I'll check out Test PyPi -- maybe that's just what I"ve been looking for. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Oct 15 00:16:22 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 15:16:22 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 1:20 PM, Ionel Cristian M?rie? wrote: > ?This reminds me of Gmail's "unsend" feature where email would be delayed > 10 seconds or something, giving a window to press the unsend button. > FWIW, I am a big fan of that -- and use it with remarkably frequency. -- maybe I'm just twitchier than most... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 15 00:20:43 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 18:20:43 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 6:13:14 PM, Chris Barker (chris.barker at noaa.gov) wrote: > On Wed, Oct 14, 2015 at 1:04 PM, Donald Stufft wrote: > > > > And it just seems pedantic to say: hey -- you've already put that one > > there > > > -- maybe even two minutes ago, so there is NO WAY to fix your mistake. If > > > it happens quickly, then no one has downloaded it, it hasn't made its way > > > to the mirrors, etc? > > > > Generally within 60-120 seconds it?s available in mirrors (most of them > > resync once a minute). > > > interesting -- it doesn't show up in the PyPi search nearly that fast :-) That?s mostly because search is barely being held together right now. When all the new stuff shakes out, that should update within a second or two. For right now it?s on a 15 minute cron job. > > If anyone has downloaded it > > > Honestly, I'm really not worried about people having downloaded it -- but > if it gets mirrored almost instantaneously, then there's a problem. > > > > The original package was NumPy. It had 30,982 downloads in the last day, > > > well, let's face it, most packages are not nearly as popular as numpy... > > But again, if it's been mirrored, we're stuck. > > > Version numbers are free, use more of them. If you can?t just issue new > > releases quickly, then test your release before you upload it (and then > > upload it with twine) and you can even upload it to Test PyPI to test > > things earlier than that. > > > Bingo! I'll check out Test PyPi -- maybe that's just what I"ve been looking > for. > > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Thu Oct 15 00:23:54 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 15:23:54 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 1:59 PM, David Cournapeau wrote: > But I'm talking about the cases of "whoops! I really wish I hadn't >> uploaded that one". We can improve the tooling (some discussion on this in >> this thread right now...), but people are people and some of us are stupid >> and/or careless. So this WILL happen. >> > > I have not done numpy releases for half a decade now, but it was already >> automated enough that putting a new version was not very costly then. >> > yeah, I suppose releases are cheap -- though part of the problem is that your users are likley to think that you actually fixed a bug or something. And maybe wonder why you went from 1.2.3 to 1.3.5 seemingly all at once... another note-- conda has teh concetp of a "build" that's tacked on teh release for conda pacakges. So if I updated somethign about how teh packge is buitl, but am using teh same underllying version of teh package, I update teh build number, get a new "version" of the package, but it's clear that the pacakge itself is the same version. for instance, I'm messing around right now with building libgd for conda, and the latest version I have up on anaconda.org is: libgd-2.1.1 but the actual file is: libgd-2.1.1-1.tar.bz2 (and there is a libgd-2.1.1-0.tar.bz2 there too...) Maybe this would be helpful for PyPi, too? I am sure there are things we can do to improve numpy's release process to > avoid this in the future. > numpy was just an example -- we are all likely to make mistakes in the future - it's human nature. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 15 00:26:23 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 14 Oct 2015 18:26:23 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On October 14, 2015 at 6:24:55 PM, Chris Barker (chris.barker at noaa.gov) wrote: > > another note-- conda has teh concetp of a "build" that's tacked > on teh release for conda pacakges. > > So if I updated somethign about how teh packge is buitl, but am > using teh same underllying version of teh package, I update teh > build number, get a new "version" of the package, but it's clear > that the pacakge itself is the same version. Wheels have a build number concept which is independent of the version and can be used to upload a new build without creating a new release? at least in theory. I?m not sure if anyone has ever tried to actually use it though.? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Thu Oct 15 00:38:59 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Oct 2015 15:38:59 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 3:26 PM, Donald Stufft wrote: > On October 14, 2015 at 6:24:55 PM, Chris Barker (chris.barker at noaa.gov) > wrote: > > > another note-- conda has teh concetp of a "build" that's tacked > > on teh release for conda pacakges. > > > > So if I updated somethign about how teh packge is buitl, but am > > using teh same underllying version of teh package, I update teh > > build number, get a new "version" of the package, but it's clear > > that the pacakge itself is the same version. > > Wheels have a build number concept which is independent of the version and > can be used to upload a new build without creating a new release? at least > in theory. sounds like exactly what conda is doing (and the same context) > I?m not sure if anyone has ever tried to actually use it though. I expect they will -- it's pretty darn useful in conda. And it's all too easy to build a binary wheel that works on the developer machine, but not everywhere else... (OK, the CI systems are making that less likely..) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Thu Oct 15 06:16:51 2015 From: cournape at gmail.com (David Cournapeau) Date: Thu, 15 Oct 2015 05:16:51 +0100 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 11:23 PM, Chris Barker wrote: > On Wed, Oct 14, 2015 at 1:59 PM, David Cournapeau > wrote: > >> But I'm talking about the cases of "whoops! I really wish I hadn't >>> uploaded that one". We can improve the tooling (some discussion on this in >>> this thread right now...), but people are people and some of us are stupid >>> and/or careless. So this WILL happen. >>> >> > >> I have not done numpy releases for half a decade now, but it was already >>> automated enough that putting a new version was not very costly then. >>> >> > yeah, I suppose releases are cheap -- though part of the problem is that > your users are likley to think that you actually fixed a bug or something. > And maybe wonder why you went from 1.2.3 to 1.3.5 seemingly all at once... > > another note-- conda has teh concetp of a "build" that's tacked on teh > release for conda pacakges. > Note that build is a feature for binaries: different builds refer to the same upstream source. Most linux distributions have the notion of downstream version which is a generalization of that. David > So if I updated somethign about how teh packge is buitl, but am using teh > same underllying version of teh package, I update teh build number, get a > new "version" of the package, but it's clear that the pacakge itself is the > same version. > > for instance, I'm messing around right now with building libgd for conda, > and the latest version I have up on anaconda.org is: > > libgd-2.1.1 > > but the actual file is: > > libgd-2.1.1-1.tar.bz2 > > (and there is a libgd-2.1.1-0.tar.bz2 there too...) > > Maybe this would be helpful for PyPi, too? > > I am sure there are things we can do to improve numpy's release process to >> avoid this in the future. >> > > numpy was just an example -- we are all likely to make mistakes in the > future - it's human nature. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Thu Oct 15 07:02:24 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 15 Oct 2015 16:02:24 +1100 Subject: [Distutils] PyPi not allowing duplicate filenames References: Message-ID: <85oag0sp0f.fsf@benfinney.id.au> Nathaniel Smith writes: > That's under Apache 2, so it can't be used by GPLv2 packages, or any > package that might be used by GPLv2 packages. I'm not sure what the claim is here, but it seems false to me. Apache License 2.0 permits a derived work to be released under a different license (even a non-free license). See Apache License 2.0 ?4. So a work BAR can be released under GNU GPL (e.g. ?version 3 or later?) that derives from work FOO released under Apache License 2.0. The GNU GPL version 3 allows combination with other works with different license conditions, provided those conditions don't impose further restrictions on recipients. See GNU GPL v3 ?7, ?10, ?12. (GPL v2 has different wording with similar effect.) Apache License 2.0 imposes no additional reastrictions on the recipient of a work licensed under GNU GPL v3. So that meets the conditions of both. FOO as a distinct work retains its license conditions, and all recipients of FOO or BAR must comply with its conditions. The work BAR as a whole carries the GNU GPL version (for example) 3 conditions, and all recipients of BAR must comply with its conditions. The two licenses are compatible, as far as I can see. The effective result is a work BAR, licensed as free software under GPL version (e.g.) 3 or later. Does that not match your understanding? Have I misunderstood your claim? -- \ ?Corporation, n. An ingenious device for obtaining individual | `\ profit without individual responsibility.? ?Ambrose Bierce, | _o__) _The Devil's Dictionary_, 1906 | Ben Finney From njs at pobox.com Thu Oct 15 07:13:54 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 15 Oct 2015 05:13:54 +0000 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: <85oag0sp0f.fsf@benfinney.id.au> References: <85oag0sp0f.fsf@benfinney.id.au> Message-ID: On Oct 14, 2015 10:02 PM, "Ben Finney" wrote: > > Nathaniel Smith writes: > > > That's under Apache 2, so it can't be used by GPLv2 packages, or any > > package that might be used by GPLv2 packages. > > I'm not sure what the claim is here, but it seems false to me. > > Apache License 2.0 permits a derived work to be released under a > different license (even a non-free license). See Apache License 2.0 ?4. The FSF believes that Apache 2 is incompatible with GPLv2 (but v3 is OK), IIRC because of some technicalities involving the patent provisions. (I'm sure you can google up infinite detail if you want -- I'm just the messenger.) There are others who disagree and think it's fine, but no one wants murkiness when it comes to licenses, so people with BSD or GPLv2 projects are unlikely to risk incorporating code from a project under Apache 2. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From xav.fernandez at gmail.com Thu Oct 15 10:33:58 2015 From: xav.fernandez at gmail.com (Xavier Fernandez) Date: Thu, 15 Oct 2015 10:33:58 +0200 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> References: <24F85F76-1B70-4EB5-ACA0-DD372F29E263@twistedmatrix.com> Message-ID: FWIW https://github.com/pypa/pip/pull/3146 exists for the wheel cache. Concerning http cache and https://github.com/ionrock/cachecontrol the main issue is that the cached request is not present in the cache. We only have a "hash -> response" mapping. I plan to make a PR asking to also keep the request for introspection and move to a "hash -> (request, response)" mapping. On Wed, Oct 14, 2015 at 11:01 PM, Glyph Lefkowitz wrote: > > On Oct 14, 2015, at 1:04 PM, Donald Stufft wrote: > > Generally within 60-120 seconds it?s available in mirrors (most of them > resync once a minute). If anyone has downloaded it then they will have > pretty much permanently cached the package, first in the download cache and > then again in the wheel cache (assuming it wasn?t a wheel already, and they > had that enabled). The original package was NumPy. It had 30,982 downloads > in the last day, so we can average that out to 1290 downloads an hour or 21 > downloads a minute. If it takes you two minutes to notice it and delete it, > then there are ~40 people who already have the original version cached and > who will not notice the updated version. > > > While I don't think PyPI should allow modification of uploaded packages > necessarily, I do think that Pip's caching is (A) too aggressive and (B) > too opaque. For example: > > https://github.com/pypa/pip/issues/3127 > https://github.com/pypa/pip/issues/3034 > https://github.com/pypa/pip/issues/3025 > https://github.com/pypa/pip/issues/2908 > https://github.com/pypa/pip/issues/2882 > > etc, etc. > > I know there are some directories platform-specific directories I can > delete, but almost once a day I want a command like `pip cache show? which > can show me what is cached and when/where it was built, `pip cache clear? > or `pip cache remove twisted? or `pip cache remove cffi>=1.0?. I don't > want to have to care if it's in the HTTP cache or the wheel cache, or how > it got there; I also don't want to have to bust a ~200 megabyte cache that > saves me hours a day just because there's one bad entry in there. > > -glyph > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Thu Oct 15 12:28:57 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 15 Oct 2015 21:28:57 +1100 Subject: [Distutils] PyPi not allowing duplicate filenames References: <85oag0sp0f.fsf@benfinney.id.au> Message-ID: <85bnc0s9w6.fsf@benfinney.id.au> Ben Finney writes: > Apache License 2.0 imposes no additional reastrictions on the > recipient of a work licensed under GNU GPL v3. So that meets the > conditions of both. Apache License 2.0 does impose further restrictions not found in GPL v2 though. Sorry for the confusion. Nathaniel Smith writes: > The FSF believes that Apache 2 is incompatible with GPLv2 (but v3 is > OK) Yes, I'd neglected that. -- \ ?My house is made out of balsa wood, so when I want to scare | `\ the neighborhood kids I lift it over my head and tell them to | _o__) get out of my yard or I'll throw it at them.? ?Steven Wright | Ben Finney From njs at pobox.com Sat Oct 17 08:06:23 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Oct 2015 23:06:23 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On Wed, Oct 14, 2015 at 3:02 PM, Donald Stufft wrote: > On October 14, 2015 at 5:17:42 PM, Nathaniel Smith (njs at pobox.com) wrote: >> > @Donald: is there somewhere we should be filing these warehouse >> wish list things so they don't get lost? :-) > > https://github.com/pypa/warehouse/issues https://github.com/pypa/warehouse/issues/726 -- Nathaniel J. Smith -- http://vorpus.org From qwcode at gmail.com Sat Oct 17 18:14:15 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 17 Oct 2015 09:14:15 -0700 Subject: [Distutils] tests location: Use case: new comers and docs. In-Reply-To: References: <56137342.8090201@thomas-guettler.de> <20151006095101.031c8a56@fsol> <20151006130018.33a5760f@fsol> <20151006141325.125514c7@fsol> <5614AFF2.1090604@thomas-guettler.de> Message-ID: > > > > I was happy to see this thread, because I thought maybe I"d learn what i > > should teach my students - new to python. > > > > > > Maybe we could come up with a decision tree for this -- some guidance for > > knowing what to do, when? > > Exactly. I think it could even be fun :) > > How could we get started to add something like this to the packaging docs? > in case you don't know the project for packaging.python.org is here: https://github.com/pypa/python-packaging-user-guide in general, we're trying to maintain 2 simple guides for installing and distributing... and then there's a somewhat open "Additional Topics" section for tutorials or more "advanced" topics. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at forgac.com Wed Oct 14 19:47:15 2015 From: dave at forgac.com (Dave Forgac) Date: Wed, 14 Oct 2015 13:47:15 -0400 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: FYI, this was discussed recently here too: https://github.com/pypa/packaging-problems/issues/74 On Wed, Oct 14, 2015 at 1:39 PM, Wes Turner wrote: > > On Oct 14, 2015 12:05 PM, "Ionel Cristian M?rie?" > wrote: > > > > > > On Wed, Oct 14, 2015 at 7:43 PM, Chris Barker > wrote: > >> > >> some packages were unable to work with the postN suffix. > > > > > > ?Can you elaborate a bit more here?? > > AFAIU, the community best practice here is to issue a new release with a > bumped version number/pre/post/releasecode. > > * https://www.python.org/dev/peps/pep-0440/#version-scheme > * https://github.com/peritus/bumpversion > * > http://docs.openstack.org/developer/pbr/semver.html#changes-vs-semver-2-0 > > MAJOR.MINOR.PATCH > > The reasons for this are many: > * HTTP caches > * local pip/buildout pkg caches > * it is actually a different resource, with a different URI > (repo,pkg,version,[pkgtype,platform,buildtags]) because it contains > different files > > > > > > > > > Thanks, > > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Oct 14 20:48:54 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 14 Oct 2015 11:48:54 -0700 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: Hi, On Wed, Oct 14, 2015 at 11:25 AM, Nathaniel Smith wrote: > On Oct 14, 2015 11:12 AM, "Donald Stufft" wrote: >> > [...] >>> Apparently some packages were making assumptions about the format of the >>> numpy.__version__ string, and having .postN in there caused errors when they >>> tried to process it. (It would be helpful if there were a little >>> permissively licensed standalone implementation of PEP 440 comparisons, >>> suitable for the "if pkg.version > ...:" checks that people insist on doing >>> -- I couldn't find one in some quick searches.) >> >> https://github.com/pypa/packaging >> >> It?s what both pip and setuptools use (though we embed it, but it?s fine >> to depend on it too). > > That's under Apache 2, so it can't be used by GPLv2 packages, or any package > that might be used by GPLv2 packages. > >>> >>> IIUC, the specific problems numpy ran into that caused the creation of >>> .postN releases were: >>> - oops, didn't sign the uploads, re-upload identical file with proper >>> signature attached -> not allowed. (I'm not sure if these were embedded or >>> detached signatures. Either way it'd be nice if pypi allowed it, but for >>> embedded signatures in particular I can see how this might be a hassle.) >> >> I don?t think we allow embedded signatures, it would be reasonable to >> allow uploading detached signatures after the fact though. >>> >>> >>> - our OS X maintainer tried to use twine to upload OS X wheels for the >>> existing release; instead it created a new release. Not sure if a bug was >>> filed on twine, but if not then one probably should be. As a workaround our >>> release docs now say "always upload wheels by hand using the web interface, >>> never use setup.py upload or twine". >> >> This shouldn?t create a new release unless you?ve changed the version >> number (including adding post releases). If you can reproduce on Test PyPI I >> can fix it. > > Matthew? Any thoughts? Sorry - I'm not sure whether y'all are surprised by the behavior of twine in this situation. Specifically, the situation was: * There was a release on pypi was labeled 1.10.0 : https://pypi.python.org/pypi/numpy/1.10.0; * The source archives that Chuck uploaded were from code that gave a version string of '1.10.0.post2'; * I built wheels from this code, that had this version encoded in the filename - e.g. http://wheels.scikit-image.org/numpy-1.10.0.post2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl ; * I used `twine upload http://wheels.scikit-image.org/numpy-1.10.0.post2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl` etc to upload these. When I did this, pypi created a new release called https://pypi.python.org/pypi/numpy/1.10.0.post2 and put the wheels there. I'm guessing that this is what y'all expected? Best, Matthew From p.f.moore at gmail.com Sun Oct 18 11:36:41 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 18 Oct 2015 10:36:41 +0100 Subject: [Distutils] PyPi not allowing duplicate filenames In-Reply-To: References: Message-ID: On 14 October 2015 at 19:48, Matthew Brett wrote: > I'm guessing that this is what y'all expected? Yes, that sounds right. 1.10.0.post2 is a new (different) release number, and so a new release would be created. Paul From guettliml at thomas-guettler.de Wed Oct 21 09:23:25 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 21 Oct 2015 09:23:25 +0200 Subject: [Distutils] red, green, refactor OR red, green, new feature? Message-ID: <56273D6D.9080506@thomas-guettler.de> Hi distutils friends, which workflow do you prefer? 1. red (test fails) 2. green (test and code works) 3. refactor (clean up, deprecate old stuff) OR 1. red 2. green 3. new feature. I, like most other programmers, love new features and often do workflow2 :-) But thinks get messy in the long run. This is the birds eye view of a particular problem I have. But the particular problem is not that important. Are you willing to clean up and deprecate old stuff? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From ben+python at benfinney.id.au Wed Oct 21 10:00:48 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 21 Oct 2015 19:00:48 +1100 Subject: [Distutils] red, green, refactor OR red, green, new feature? References: <56273D6D.9080506@thomas-guettler.de> Message-ID: <85h9lkps5r.fsf@benfinney.id.au> Thomas G?ttler writes: > Hi distutils friends, > > which workflow do you prefer? > > 1. red (test fails) > 2. green (test and code works) > 3. refactor (clean up, deprecate old stuff) More accurately: * Write a failing test for the new behaviour. (red) * Change the system-under-test until the test passes. (green) * Clean up, remove redundancies, change no behaviour (refactor) > OR I disagree with that dichotomy. > 1. red > 2. green > 3. new feature. The latter isn't a workflow I recognise. It appears to confuse the levels of working. Making a new feature is much higher level than the rest of those steps; it entails many iterations of ?red, green, refactor?. > I, like most other programmers, love new features You're thinking that is incompatible with test-driven development. Actually, from the TDD point of view, adding new features is like any other change to the system under test: follow the TDD workflow (red, green, refactor) until the new feature works. > Are you willing to clean up and deprecate old stuff? Everyone who works with code should be *willing* to do it. What TDD does is allow the programer to do that with confidence. -- \ ?Intellectual property is to the 21st century what the slave | `\ trade was to the 16th.? ?David Mertz | _o__) | Ben Finney From contact at ionelmc.ro Wed Oct 21 10:58:42 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 21 Oct 2015 11:58:42 +0300 Subject: [Distutils] red, green, refactor OR red, green, new feature? In-Reply-To: <85h9lkps5r.fsf@benfinney.id.au> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> Message-ID: On Wed, Oct 21, 2015 at 11:00 AM, Ben Finney wrote: > > 1. red > > 2. green > > 3. new feature. > > The latter isn't a workflow I recognise. It appears to confuse the > levels of working. > ?I think Thomas is making a subtle joke on the state some of the packaging tools? are in. Especially the older ones. Then again, sometimes refactoring is very hard. Is rewriting from scratch better? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Wed Oct 21 13:52:14 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 21 Oct 2015 13:52:14 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> Message-ID: <56277C6E.2030605@thomas-guettler.de> Am 21.10.2015 um 13:28 schrieb Ionel Cristian M?rie?: > > On Wed, Oct 21, 2015 at 1:42 PM, Thomas G?ttler > wrote: > > > Why not deprecate and remove distutils? > > > ?Should we or can we? There was an attempt at that, called distutils2. I'd love to hear what people have learnt from that. > > And if now-days walking around with pants instead of only underwear is the way to go, should we still have underwear > under the pants? I think yes :-) In other words, setuptools uses distutils. ok, at the moment setuptools uses distutils. Why not melt them together into **one** underwear-pants-module? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From p.f.moore at gmail.com Wed Oct 21 14:07:10 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Oct 2015 13:07:10 +0100 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <56277C6E.2030605@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: On 21 October 2015 at 12:52, Thomas G?ttler wrote: > ok, at the moment setuptools uses distutils. > > Why not melt them together into **one** underwear-pants-module? (1) because there's no-one with anything like the time or energy to do such a thing (2) because doing so would almost certainly break vast numbers of existing setup.py scripts Paul From solipsis at pitrou.net Wed Oct 21 14:10:31 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2015 14:10:31 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: <20151021141031.2e25d3a1@fsol> On Wed, 21 Oct 2015 13:07:10 +0100 Paul Moore wrote: > On 21 October 2015 at 12:52, Thomas G?ttler > wrote: > > ok, at the moment setuptools uses distutils. > > > > Why not melt them together into **one** underwear-pants-module? > > (1) because there's no-one with anything like the time or energy to do > such a thing > (2) because doing so would almost certainly break vast numbers of > existing setup.py scripts (3) because distutils is part of the standard library and setuptools is most likely unwelcome there :-) Regards Antoine. From contact at ionelmc.ro Wed Oct 21 14:30:51 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 21 Oct 2015 15:30:51 +0300 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <20151021141031.2e25d3a1@fsol> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <20151021141031.2e25d3a1@fsol> Message-ID: On Wed, Oct 21, 2015 at 3:10 PM, Antoine Pitrou wrote: > (3) because distutils is part of the standard library and setuptools is > most likely unwelcome there :-) > ?Well, it's not dead yet. Sorry, I mean,? good enough to be included in the stdlib :-) But seriously, is it correct to assume that it's not there for largely the same reasons pip ain't in stdlib? (and yes, getpip is in stdlib) Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 21 14:46:25 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2015 14:46:25 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <20151021141031.2e25d3a1@fsol> Message-ID: <20151021144625.668631a4@fsol> On Wed, 21 Oct 2015 15:30:51 +0300 Ionel Cristian M?rie? wrote: > On Wed, Oct 21, 2015 at 3:10 PM, Antoine Pitrou wrote: > > > (3) because distutils is part of the standard library and setuptools is > > most likely unwelcome there :-) > > > > ?Well, it's not dead yet. Sorry, I mean,? good enough to be included in the > stdlib :-) > > But seriously, is it correct to assume that it's not there for largely the > same reasons pip ain't in stdlib? (and yes, getpip is in stdlib) setuptools inclusion was discussed long ago AFAIK, and it was refused. For pip the reasons are different (the much shorter release cycle, the external dependencies required). See e.g.: https://mail.python.org/pipermail/python-dev/2006-April/063966.html https://mail.python.org/pipermail/python-dev/2006-April/063886.html (incidentally, the latter link shows a previous attempt to refactor distutils was shot down because of backwards compatibility; which is also why distutils2 was canned, despite bringing concrete positive enhancements; had those attempts not been shot down, the situation right now would be significantly better than it is...) Regards Antoine. From cournape at gmail.com Wed Oct 21 14:55:40 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 21 Oct 2015 13:55:40 +0100 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <56277C6E.2030605@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > Am 21.10.2015 um 13:28 schrieb Ionel Cristian M?rie?: > >> >> On Wed, Oct 21, 2015 at 1:42 PM, Thomas G?ttler < >> guettliml at thomas-guettler.de > >> wrote: >> >> >> Why not deprecate and remove distutils? >> >> >> ?Should we or can we? There was an attempt at that, called distutils2. >> I'd love to hear what people have learnt from that. >> >> And if now-days walking around with pants instead of only underwear is >> the way to go, should we still have underwear >> under the pants? I think yes :-) In other words, setuptools uses >> distutils. >> > > ok, at the moment setuptools uses distutils. > > Why not melt them together into **one** underwear-pants-module? > What do you hope getting from that ? distutils is in the stdlib, so cannot change easily, and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python. On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 21 17:05:29 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Oct 2015 17:05:29 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: On 21 October 2015 at 14:55, David Cournapeau wrote: > > On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler > wrote: >> ok, at the moment setuptools uses distutils. >> >> Why not melt them together into **one** underwear-pants-module? > > > What do you hope getting from that ? distutils is in the stdlib, so cannot > change easily, and even if putting setuptools in the stdlib were possible, > you would now need to handle different versions of setuptools for different > versions of python. It's more useful to go the other direction and vendor a modern version of distutils inside setuptools: https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils distutils can then optionally be replaced wholesale at runtime, rather than having the internals be monkeypatched. > On top of this, the goal of lots of efforts around packaging is to allow > people to move away from distutils/setuptools, as the underlying design is > fundamentally difficult to extend. We still need a migration path to modern metadata standards for everyone using distutils and setuptools - that's the side of things that caused major problems for both distribute and distutils2. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Wed Oct 21 17:15:39 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2015 17:15:39 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: <20151021171539.2389c1c5@fsol> On Wed, 21 Oct 2015 17:05:29 +0200 Nick Coghlan wrote: > On 21 October 2015 at 14:55, David Cournapeau wrote: > > > > On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler > > wrote: > >> ok, at the moment setuptools uses distutils. > >> > >> Why not melt them together into **one** underwear-pants-module? > > > > > > What do you hope getting from that ? distutils is in the stdlib, so cannot > > change easily, and even if putting setuptools in the stdlib were possible, > > you would now need to handle different versions of setuptools for different > > versions of python. > > It's more useful to go the other direction and vendor a modern version > of distutils inside setuptools: It seems it would only add a bit more craziness to the current landscape. What happens to projects which have a need to monkeypatch distutils? How does it interact with the vendored version? etc. Regards Antoine. From dholth at gmail.com Wed Oct 21 17:29:17 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 21 Oct 2015 15:29:17 +0000 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume. For example we can convert a setuptools egg or a bdist_wininst .exe to a wheel, and in the future we will be able to convert e-mail style PKG-INFO to json. It's much easier to test whether a tool is producing correct metadata than to decide whether the API is compatible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Wed Oct 21 18:34:52 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 21 Oct 2015 18:34:52 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: <5627BEAC.3090808@thomas-guettler.de> Am 21.10.2015 um 17:29 schrieb Daniel Holth: > The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume.... I ask myself: Why a standard? I see that a standard is very important if there will be several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) But here a single implementation for creating and installing packages would be enough. Is a standard really needed? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From chris.barker at noaa.gov Wed Oct 21 18:46:08 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 09:46:08 -0700 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <5627BEAC.3090808@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> Message-ID: On Wed, Oct 21, 2015 at 9:34 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > I ask myself: Why a standard? I see that a standard is very important if > there will be > several implementations (for example TCP/IP, protocols like HTTP, SMTP, > IMAP, ...) > > But here a single implementation for creating and installing packages > would be enough. > > Is a standard really needed? > Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate. If we standardize the format, interfaces and metadata, and properly separate the components, then someone can use some other build system for their package, and still be able to use the same distribution, dependency mangement, and installation tools. or, for that matter swap out one of those other components too. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 21 18:42:02 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 09:42:02 -0700 Subject: [Distutils] Time for a setuptools_lite?? Message-ID: This is kind-of related to the othe thread: "Remove distutils, was: ..." But more specific, so I thought I'd start a new one. Here are my thoughts: We had distutils -- there was a lot it didn't do that the "Masses" needed, so setuptools was born. It proved to be useful to a lot of people, and grew a large userbase.... But a lot was "wrong" with setuptools -- most prominently (in my mind anyway) that it put too many kind-sorta orthogonal stuff into one package: building, installing, distributing, managing version, managing dependencies, managing non-python resources, (and others??). And we didn't like how easy-install installed things :-) So distribute, and pip, and wheel, and now a new backward compatible setuptools was born. But we still have a bunch of should be orthogonal stuff tangled up together. In particular, I find that I often find easy-install getting invoked when I don't want ot to, and I get those darn eggs scattered all over the place, and easy_install.pth, and ???? I think if I am really careful about what I invoke when, this could be avoided, but the reality is that I've been dealing with this for years, and am trying desperately to do things the "right, modern" way, and I still get ugliness. I seriously doubt that I am alone. So -- here's my thought: I think we have it pretty well mapped out what functionality belongs where: one system for building packages (setuptools?) one system for installing packages and managing dependencies (pip) one system (really standard) for metadata and distributing packages (wheel) [I'm just whipping this out off the top of my head, I'm sure we'd need to more clearly define what belongs where] So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc... This seems to me to be a way to go forward -- as it is we'll have people using setuptools features that they "shouldn't" forever, and never be able to move to a cleaner system. Or maybe a flag: import setuptools setuptools.use_only_modern() That would make the dependencies easier -- i.e. pip depends on some of setuptools being there -- hard to say that it needs either setuptools OR setuptools_lite. Maybe I'm missing something here, but if the goal is for there to be one way to do things, let's have a tool chain that only does things one way..... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From opensource at ronnypfannschmidt.de Wed Oct 21 19:04:35 2015 From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Wed, 21 Oct 2015 19:04:35 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <5627BEAC.3090808@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> Message-ID: <5627C5A3.6090004@ronnypfannschmidt.de> another example is that without a standard there is no viable way to replace a grown messy legacy tool with a much less messy tool over time creating a new trimmed down standard and supporting it from the legacy tool as well as from the new tool works out much better than trying to trim down the legacy tool over time On 10/21/2015 06:34 PM, Thomas G?ttler wrote: > Am 21.10.2015 um 17:29 schrieb Daniel Holth: >> The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume.... > I ask myself: Why a standard? I see that a standard is very important if there will be > several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) > > But here a single implementation for creating and installing packages would be enough. > > Is a standard really needed? > > Regards, > Thomas G?ttler > > > > From opensource at ronnypfannschmidt.de Wed Oct 21 19:03:02 2015 From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Wed, 21 Oct 2015 19:03:02 +0200 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: Message-ID: <5627C546.40004@ronnypfannschmidt.de> why does that have to be in setuptools ?! if we want a new light system to begin with, shouldn't it be far more sustainable to use just implementations of the new standards rather than leaving all of setuptools there is no viable way in setuptools to get rid of the legacy ina sane and fast manner, it would drag out over years and both distutils and setuptools are very very sub-par from the maintenance perspective On 10/21/2015 06:42 PM, Chris Barker wrote: > This is kind-of related to the othe thread: > > "Remove distutils, was: ..." > > But more specific, so I thought I'd start a new one. > > Here are my thoughts: > > We had distutils -- there was a lot it didn't do that the "Masses" > needed, so setuptools was born. It proved to be useful to a lot of > people, and grew a large userbase.... > > But a lot was "wrong" with setuptools -- most prominently (in my mind > anyway) that it put too many kind-sorta orthogonal stuff into one > package: building, installing, distributing, managing version, > managing dependencies, managing non-python resources, (and others??). > And we didn't like how easy-install installed things :-) > > So distribute, and pip, and wheel, and now a new backward compatible > setuptools was born. > > But we still have a bunch of should be orthogonal stuff tangled up > together. In particular, I find that I often find easy-install getting > invoked when I don't want ot to, and I get those darn eggs scattered > all over the place, and easy_install.pth, and ???? > > I think if I am really careful about what I invoke when, this could be > avoided, but the reality is that I've been dealing with this for > years, and am trying desperately to do things the "right, modern" way, > and I still get ugliness. I seriously doubt that I am alone. > > So -- here's my thought: > > I think we have it pretty well mapped out what functionality belongs > where: > > one system for building packages (setuptools?) > one system for installing packages and managing dependencies (pip) > one system (really standard) for metadata and distributing packages > (wheel) > > [I'm just whipping this out off the top of my head, I'm sure we'd need > to more clearly define what belongs where] > > So why not have a setuptools-lite that only does the building? We need > to keep the full over-burdened setuptools around, because lot sof > folks are using those features. But for those of us that are doing > something fairly new, and don't want to use stuff that setuptools > "shouldn't" be doing, I'd love to have a setuptools-lite that only did > what I really need, and was guaranteed NOT to accidentally introduce > easy_install, etc... > > This seems to me to be a way to go forward -- as it is we'll have > people using setuptools features that they "shouldn't" forever, and > never be able to move to a cleaner system. > > Or maybe a flag: > > import setuptools > setuptools.use_only_modern() > > That would make the dependencies easier -- i.e. pip depends on some of > setuptools being there -- hard to say that it needs either setuptools > OR setuptools_lite. > > Maybe I'm missing something here, but if the goal is for there to be > one way to do things, let's have a tool chain that only does things > one way..... > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 21 19:17:22 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Oct 2015 18:17:22 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: Message-ID: On 21 October 2015 at 17:42, Chris Barker wrote: > So why not have a setuptools-lite that only does the building? We need to > keep the full over-burdened setuptools around, because lot sof folks are > using those features. But for those of us that are doing something fairly > new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd > love to have a setuptools-lite that only did what I really need, and was > guaranteed NOT to accidentally introduce easy_install, etc... In general, this sounds like a reasonable idea. But, there are the usual problems: 1. Someone needs to do the work. 2. Backward compatibility (as in, there's bound to be some projects out there using whatever functionality you can think of that "nobody would ever need"...) And a third issue specific to this idea. 3. You can't call it setuptools, and pip "injects" the real setuptools into projects when building/installing. Pretending to be setuptools was what made distribute such a PITA to deal with, and I don't imagine we'd ever want to go there again. Sorry, I'm being a downbeat grump today. The reality is that far and away the best shot we've had at getting out of this mess is the plan we're currently working to, which is to standardise interfaces, and once we have some standards, we can start decoupling the various bits and *then* replacing them as needed. We need people working on that in the first instance, work is currently stalled by lack of time from the key participants. So anyone with energy for moving this whole area forward would be really welcome to help out with the interoperability standards (metadata 2.0, defining a sdist format, pinning down pip's interfaces to build systems, things like that). But if you want to rework the existing toolset, you're probably going to be on your own (and there's unfortunately a risk that you'll pull attention away from other work via debates on this list etc). I really appreciate the enthusiasm we're seeing from people on the list at the moment, it's a matter of harnessing that into the right direction. As a thought, does anyone feel like working on an "interoperability standards roadmap" that provides a reference for where the PyPA see things going? It would be useful to keep focus, but the information's currently scattered about various mailing lists and discussions - some of the posts around the recent "new source distribution format" threads might be a good place to start. Paul From waynejwerner at gmail.com Wed Oct 21 19:08:45 2015 From: waynejwerner at gmail.com (Wayne Werner) Date: Wed, 21 Oct 2015 12:08:45 -0500 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: Message-ID: On Wed, Oct 21, 2015 at 11:42 AM, Chris Barker wrote: > But a lot was "wrong" with setuptools -- most prominently (in my mind > anyway) that it put too many kind-sorta orthogonal stuff into one package: > building, installing, distributing, managing version, managing > dependencies, managing non-python resources, (and others??). And we didn't > like how easy-install installed things :-) > > So distribute, and pip, and wheel, and now a new backward compatible > setuptools was born. > > But we still have a bunch of should be orthogonal stuff tangled up > together. In particular, I find that I often find easy-install getting > invoked when I don't want ot to, and I get those darn eggs scattered all > over the place, and easy_install.pth, and ???? > > I think if I am really careful about what I invoke when, this could be > avoided, but the reality is that I've been dealing with this for years, and > am trying desperately to do things the "right, modern" way, and I still get > ugliness. I seriously doubt that I am alone. > > So -- here's my thought: > > I think we have it pretty well mapped out what functionality belongs where: > > one system for building packages (setuptools?) > one system for installing packages and managing dependencies (pip) > one system (really standard) for metadata and distributing packages (wheel) > > [I'm just whipping this out off the top of my head, I'm sure we'd need to > more clearly define what belongs where] > > So why not have a setuptools-lite that only does the building? We need to > keep the full over-burdened setuptools around, because lot sof folks are > using those features. But for those of us that are doing something fairly > new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd > love to have a setuptools-lite that only did what I really need, and was > guaranteed NOT to accidentally introduce easy_install, etc... > > This seems to me to be a way to go forward -- as it is we'll have people > using setuptools features that they "shouldn't" forever, and never be able > to move to a cleaner system. > > Or maybe a flag: > > import setuptools > setuptools.use_only_modern() > > That would make the dependencies easier -- i.e. pip depends on some of > setuptools being there -- hard to say that it needs either setuptools OR > setuptools_lite. > > Maybe I'm missing something here, but if the goal is for there to be one > way to do things, let's have a tool chain that only does things one way..... > I've been thinking about packaging in general and some of the complaints that I've seen and heard raised, as well as some that I have harbored personally, like having a venv per app that I've installed, or my own env where I install several things. I actually don't know if I've ran into the issues with easy_install, but I haven't been involved in installing things enough to know :) I do like the idea of having distinct pieces to the toolchain, though. Of course maybe it's also a good idea to have something that integrates all the pieces, too. -Wayne -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Oct 21 19:32:55 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 21 Oct 2015 18:32:55 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: <5627C546.40004@ronnypfannschmidt.de> References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < opensource at ronnypfannschmidt.de> wrote: > why does that have to be in setuptools ?! > > if we want a new light system to begin with, shouldn't it be far more > sustainable to use just implementations of the new standards rather than > leaving all of setuptools > > there is no viable way in setuptools to get rid of the legacy ina sane and > fast manner, it would drag out over years > agreed. I have never met a person who had to deal substantially with distutils code and enjoyed the experience. The whole architecture is fundamentally flawed. I wrote this a long time ago, but I still stand by most arguments: https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ David > > and both distutils and setuptools are very very sub-par from the > maintenance perspective > > > On 10/21/2015 06:42 PM, Chris Barker wrote: > > This is kind-of related to the othe thread: > > "Remove distutils, was: ..." > > But more specific, so I thought I'd start a new one. > > Here are my thoughts: > > We had distutils -- there was a lot it didn't do that the "Masses" needed, > so setuptools was born. It proved to be useful to a lot of people, and grew > a large userbase.... > > But a lot was "wrong" with setuptools -- most prominently (in my mind > anyway) that it put too many kind-sorta orthogonal stuff into one package: > building, installing, distributing, managing version, managing > dependencies, managing non-python resources, (and others??). And we didn't > like how easy-install installed things :-) > > So distribute, and pip, and wheel, and now a new backward compatible > setuptools was born. > > But we still have a bunch of should be orthogonal stuff tangled up > together. In particular, I find that I often find easy-install getting > invoked when I don't want ot to, and I get those darn eggs scattered all > over the place, and easy_install.pth, and ???? > > I think if I am really careful about what I invoke when, this could be > avoided, but the reality is that I've been dealing with this for years, and > am trying desperately to do things the "right, modern" way, and I still get > ugliness. I seriously doubt that I am alone. > > So -- here's my thought: > > I think we have it pretty well mapped out what functionality belongs where: > > one system for building packages (setuptools?) > one system for installing packages and managing dependencies (pip) > one system (really standard) for metadata and distributing packages (wheel) > > [I'm just whipping this out off the top of my head, I'm sure we'd need to > more clearly define what belongs where] > > So why not have a setuptools-lite that only does the building? We need to > keep the full over-burdened setuptools around, because lot sof folks are > using those features. But for those of us that are doing something fairly > new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd > love to have a setuptools-lite that only did what I really need, and was > guaranteed NOT to accidentally introduce easy_install, etc... > > This seems to me to be a way to go forward -- as it is we'll have people > using setuptools features that they "shouldn't" forever, and never be able > to move to a cleaner system. > > Or maybe a flag: > > import setuptools > setuptools.use_only_modern() > > That would make the dependencies easier -- i.e. pip depends on some of > setuptools being there -- hard to say that it needs either setuptools OR > setuptools_lite. > > Maybe I'm missing something here, but if the goal is for there to be one > way to do things, let's have a tool chain that only does things one way..... > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.orghttps://mail.python.org/mailman/listinfo/distutils-sig > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Oct 21 20:02:14 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 21 Oct 2015 19:02:14 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 21, 2015 at 6:44 PM, Wayne Werner wrote: > > On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau > wrote: > >> >> >> On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < >> opensource at ronnypfannschmidt.de> wrote: >> >>> why does that have to be in setuptools ?! >>> >>> if we want a new light system to begin with, shouldn't it be far more >>> sustainable to use just implementations of the new standards rather than >>> leaving all of setuptools >>> >>> there is no viable way in setuptools to get rid of the legacy ina sane >>> and fast manner, it would drag out over years >>> >> >> agreed. I have never met a person who had to deal substantially with >> distutils code and enjoyed the experience. >> >> The whole architecture is fundamentally flawed. I wrote this a long time >> ago, but I still stand by most arguments: >> https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ >> > > I've (luckily?) never had to deal with distutils code... I am definitely > digging the post. First time I've read it, but I am definitely more > pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a > few minutes ago. > > Would pip's freeze format work a la the cabal file, or is it missing too > much information? > the main cabal file is its own DSL that describes the package metadata (so more a replacement of setup.py). I used a similar idea when I wrote bento (https://cournape.github.io/Bento/), which ended up being a mistake. If I were to write a similar tool today, I would just use templated yaml. A big difference compared to 6-7 years ago is the progress made on pip, wheels, etc... A lot of the current efforts in python package (metadata formalization, etc...) are about enabling tools like bento, flit, etc... to coexist: as long as they produced wheels/(wheel)sdist/etc... that pip can consume, you have a clear separate of concern, and an healthy competition. David > > -Wayne > -------------- next part -------------- An HTML attachment was scrubbed... URL: From waynejwerner at gmail.com Wed Oct 21 19:44:38 2015 From: waynejwerner at gmail.com (Wayne Werner) Date: Wed, 21 Oct 2015 12:44:38 -0500 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau wrote: > > > On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < > opensource at ronnypfannschmidt.de> wrote: > >> why does that have to be in setuptools ?! >> >> if we want a new light system to begin with, shouldn't it be far more >> sustainable to use just implementations of the new standards rather than >> leaving all of setuptools >> >> there is no viable way in setuptools to get rid of the legacy ina sane >> and fast manner, it would drag out over years >> > > agreed. I have never met a person who had to deal substantially with > distutils code and enjoyed the experience. > > The whole architecture is fundamentally flawed. I wrote this a long time > ago, but I still stand by most arguments: > https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ > I've (luckily?) never had to deal with distutils code... I am definitely digging the post. First time I've read it, but I am definitely more pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a few minutes ago. Would pip's freeze format work a la the cabal file, or is it missing too much information? -Wayne -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Oct 21 20:29:17 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 21 Oct 2015 18:29:17 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: We're not just grumpy, we have seen The One Build System idea tried several times with bad results. Instead, if you have the inclination, time, and ability, you could try to build a new competing build system like http://flit.readthedocs.org/ did, or you could help try to figure out how to improve support for said competing build systems in pip, or you could even try to make a pip replacement that is better for distributing end-user applications. Ideally one of these new build systems will be compelling enough that new packages will choose to use it instead of distutils in the same way that people choose Django over the cgi module. On Wed, Oct 21, 2015 at 2:08 PM Wayne Werner wrote: > On Wed, Oct 21, 2015 at 12:32 PM, David Cournapeau > wrote: > >> >> >> On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt < >> opensource at ronnypfannschmidt.de> wrote: >> >>> why does that have to be in setuptools ?! >>> >>> if we want a new light system to begin with, shouldn't it be far more >>> sustainable to use just implementations of the new standards rather than >>> leaving all of setuptools >>> >>> there is no viable way in setuptools to get rid of the legacy ina sane >>> and fast manner, it would drag out over years >>> >> >> agreed. I have never met a person who had to deal substantially with >> distutils code and enjoyed the experience. >> >> The whole architecture is fundamentally flawed. I wrote this a long time >> ago, but I still stand by most arguments: >> https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ >> > > I've (luckily?) never had to deal with distutils code... I am definitely > digging the post. First time I've read it, but I am definitely more > pro-standard-interface-and-let-tools-do-with-it-what-they-will than I was a > few minutes ago. > > Would pip's freeze format work a la the cabal file, or is it missing too > much information? > > -Wayne > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Oct 21 20:35:17 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 21 Oct 2015 18:35:17 +0000 Subject: [Distutils] free project idea Message-ID: Take this setup.py that implements setup requirements (requirements needed for setup.py itself to run) https://bitbucket.org/dholth/setup-requires/src/tip/setup.py Require your new build system using that mechanism, by mentioning it in setup.cfg Then instead of calling setup() from setuptools at the bottom, emulate the setup.py build system interface documented at http://pip.readthedocs.org/en/stable/reference/pip_install/#build-system-interface to invoke new build system. Check to make sure whether pip accepts .dist-info when calling the egg_info command. Then write a command for new build system that adds the shim setup.py to an sdist. Now you have a setup.py that can download and install new-build-system and interoperate with pip, without having to change pip at all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ich at ronnypfannschmidt.de Wed Oct 21 21:00:15 2015 From: ich at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Wed, 21 Oct 2015 21:00:15 +0200 Subject: [Distutils] free project idea In-Reply-To: References: Message-ID: <5627E0BF.2000103@ronnypfannschmidt.de> my suggestion would be provide a whl_setup.py that works in a fashion similar to the current ez_setup however instead of using easy_install + eggs, it would use pip install --target into a folder relative to the setup.py that invoked it if there was a mechanism to prevent pip from invoking setup.py with its stub, that system could even be used to install more recent setuptools itself into the "setup site" a setup.py would then just use whl_setup to get at setuptools, and as a nice extra, it could hijack the install_requirements method of a distrbution, so pip would be used to install the requirements into the "setup site" as well that way we could get rid of the need for eggs to make setup_requires nice, and could just have pip deal with all the details in addition it could bail out on too old pip (after all it needs a properly working install target support to provide the "setup site" with packages, and a even newer pip version should the stub-disabling be implemented) -- Ronny On 10/21/2015 08:35 PM, Daniel Holth wrote: > Take this setup.py that implements setup requirements (requirements > needed for setup.py itself to run) > https://bitbucket.org/dholth/setup-requires/src/tip/setup.py > > Require your new build system using that mechanism, by mentioning it > in setup.cfg > > Then instead of calling setup() from setuptools at the bottom, emulate > the setup.py build system interface documented > at http://pip.readthedocs.org/en/stable/reference/pip_install/#build-system-interface to > invoke new build system. Check to make sure whether pip accepts > .dist-info when calling the egg_info command. > > Then write a command for new build system that adds the shim setup.py > to an sdist. > > Now you have a setup.py that can download and install new-build-system > and interoperate with pip, without having to change pip at all. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Wed Oct 21 21:10:54 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Wed, 21 Oct 2015 21:10:54 +0200 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> Message-ID: <5627E33E.8030305@thomas-guettler.de> Am 21.10.2015 um 18:46 schrieb Chris Barker: > On Wed, Oct 21, 2015 at 9:34 AM, Thomas G?ttler > wrote: > > I ask myself: Why a standard? I see that a standard is very important if there will be > several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) > > But here a single implementation for creating and installing packages would be enough. > > Is a standard really needed? > > > Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- > the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate. What happens if the common use cases are inadequate? My guess: re-inventing the same stuff over and over again. Once in numpy, once in scipy ... Why should it be impossible to get all the needs of numpy and scipy into setuptools? I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ... Even for complex packages. This data gets processed by setuptools. I don't see a need for more than one library doing this. Plugins are nice and can solve edge cases. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From chris.barker at noaa.gov Wed Oct 21 21:05:06 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 12:05:06 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: <5627C546.40004@ronnypfannschmidt.de> References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 21, 2015 at 10:03 AM, Ronny Pfannschmidt < opensource at ronnypfannschmidt.de> wrote: > why does that have to be in setuptools ?! > it doesn't have to be in setuptools at all -- I suppose I should have defined more clearly what I meant: setuptools_lite would be a package that does all things that setuptools does that we currently think it *should* do -- and nothing else. whether it's in setuptools as a special mode, forked from setuptools, or written from scratch is all implementation detail. perhaps it would be better with a different name -- maybe "buildtools"? But I thought for acceptance by the community, maybe saying: replace all your "import setuptools" with "import setuptool_lite" would be clear what the intent was -- i.e. not YET ANOTHER brand new build system... Oh and it would use the same API for the parts that it still supported. Given all that, if I were to do it, and I'm not sure I can... I would probably start with setuptools and rip stuff out rather than start from scratch. and both distutils and setuptools are very very sub-par > well, see the other discussion about the role / necessity of distutils... but yes, it's certainly possible to make a new build system completely independent of distutils. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From opensource at ronnypfannschmidt.de Wed Oct 21 21:23:22 2015 From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Wed, 21 Oct 2015 21:23:22 +0200 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: <5627E62A.80302@ronnypfannschmidt.de> i would very strongly suggest to completely avoid distutils as base for a new tool being based on distutils is one of the main reasons why setuptools is so painful to develop/maintain, its always a huge mental cost and pain to work into that huge pile of historic spaghetti and its even more pain to come up with useful tests -- Ronny On 10/21/2015 09:05 PM, Chris Barker wrote: > On Wed, Oct 21, 2015 at 10:03 AM, Ronny Pfannschmidt > > wrote: > > why does that have to be in setuptools ?! > > > it doesn't have to be in setuptools at all -- I suppose I should have > defined more clearly what I meant: > > setuptools_lite would be a package that does all things that > setuptools does that we currently think it *should* do -- and nothing > else. > > whether it's in setuptools as a special mode, forked from setuptools, > or written from scratch is all implementation detail. > > perhaps it would be better with a different name -- maybe "buildtools"? > > But I thought for acceptance by the community, maybe saying: > > replace all your "import setuptools" with "import setuptool_lite" > would be clear what the intent was -- i.e. not YET ANOTHER brand new > build system... > > Oh and it would use the same API for the parts that it still > supported. Given all that, if I were to do it, and I'm not sure I > can... I would probably start with setuptools and rip stuff out rather > than start from scratch. > > and both distutils and setuptools are very very sub-par > > > well, see the other discussion about the role / necessity of > distutils... but yes, it's certainly possible to make a new build > system completely independent of distutils. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 21 21:26:39 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 21 Oct 2015 15:26:39 -0400 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On October 21, 2015 at 3:13:11 PM, Chris Barker (chris.barker at noaa.gov) wrote: > > replace all your "import setuptools" with "import setuptool_lite" would be > clear what the intent was -- i.e. not YET ANOTHER brand new build system... > Moving from one ?one true build system? to another ?one true build system? is unlikely. The problem is that the kind of things people want to do on the build side are basically infinite and trying to make a single tool to rule them all has historically boiled down to either the tool is good at one particular use case and really really terrible for every other use case OR it?s just terrible (but not really really terrible!) at every use case. A far better approach IMO is the one we?re taking. Define standard *formats* and let end users select whatever tool they want to create things that adhere to that format. This lets people create really focused tools for particular use cases, you can have a dead simple one for pure python things that never need to worry about the complexity of building C libraries (or even the complexity of needing to interface with Fortran) but the people who do need the extra complexity can use tools that enable them to do that complexity in a sane way. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Wed Oct 21 21:32:09 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 12:32:09 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: Message-ID: On Wed, Oct 21, 2015 at 10:17 AM, Paul Moore wrote: > On 21 October 2015 at 17:42, Chris Barker wrote: > > So why not have a setuptools-lite that only does the building? > In general, this sounds like a reasonable idea. But, there are the > usual problems: > > 1. Someone needs to do the work. > well, yeah -- that's the always the big problem. > 2. Backward compatibility (as in, there's bound to be some projects > out there using whatever functionality you can think of that "nobody > would ever need"...) > actually, this is exactly the problem I'm trying to address with this idea. Lots of people use setuptools, and you can bet the someone, somewhere, is using every little feature (and counting on a bug) than it has. Which is why distribute merged into setuptools -- I wasn't involved in the conversation at that point, but I think it more or less started as a "new, better, setuptools replacement", and then turned into a "maintained and updated setuptools". What I do remember from those days is that setuptools itself had fallen into a maintenance hole -- very frustrating. But fixing that means that distribute lost its goal of actually making it better. The idea here is to provide an upgrade path -- if you want to do thing the new way (the sane way :-) ), then drop in setuptools_lite, and see what breaks -- then decide to remove or refactor the things that setuptools_lite doesn't support, or just go back to setuptools. 3. You can't call it setuptools, and pip "injects" the real setuptools > into projects when building/installing. Pretending to be setuptools > was what made distribute such a PITA to deal with, and I don't imagine > we'd ever want to go there again. > no -- it would be setuptools_lite -- different name :-) Or it would be a somewhat-disabled setuptools -- which would be the distribute problem again. But I _think_ the problem with distribute was that people WANTED it to have all the functionality of setuptools -- and going back to setuptools wasn't a good option because it was no longer well maintained (see above). Granted, I wasn't in the trenches then, or even lurking -- but I was a user, and trying to teach python newbies what to do -- and yes, it did kind of suck. As I understand it, pip injects setuptools because pip really needs a build tool that does things that distutils doesn't. And there are no other build tools. But that architecture really sucks -- pip should not have to rely on injecting a particular package, but rather be able to rely on a given API to be available. I have no idea how we can get from here to there cleanly, but it would it be that big a deal for pip to look for either setuptools or setuptools_lite, and neither are there, inject setuptools_lite. The whole point is that setuptools_lite would have everything that pip needs. Sorry, I'm being a downbeat grump today. not at all -- this has been an ugly mess for a long time -- we really, really need the folks with experience to keep things in check :-) > The reality is that far and > away the best shot we've had at getting out of this mess is the plan > we're currently working to, which is to standardise interfaces, and > once we have some standards, we can start decoupling the various bits > and *then* replacing them as needed. well, yes. But my idea here is to have something for users to do in the interim -- right now, I HAVE to use setuptools -- it's the only thing that knows how to find the compiler on Windows, and pip needs it to install. So there is no way for me not to find it really easy to accidentally use setuptools features that I don't want to use, and "shouldn't" use. So while we are waiting for the grand future of clear APIs and well-defined components, we can start getting people ready for that.... As I write this, I'm starting to think that despite my post a half hour ago, probably the best way to do this is to add a __future__ feature to setuptools: import setuptools setuptools.__future__() or some such -- all it would do is remove and/or turn off stuff setuptools should no longer be doing. And this might actually help with the "standardise interfaces" part of the project. Essentially a working prototype of what the build tool needs to support. If turn off a feature, we may find that it's needed -- then we can decide where that feature needs to live. But if you want to > rework the existing toolset, you're probably going to be on your own > (and there's unfortunately a risk that you'll pull attention away from > other work via debates on this list etc). > Fair enough -- though I"m thinking of this as both a way to get something useful sooner than later, AND a way to maybe test out some of the API ideas. After all, there have been efforts to just build something new -- distutils2, etc -- maybe we need a more transitional, try it as you go approach. > As a thought, does anyone feel like working on an > "interoperability standards roadmap" that provides a reference for > where the PyPA see things going? That would be great, yes! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 21 21:41:07 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 12:41:07 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 21, 2015 at 12:26 PM, Donald Stufft wrote: > On October 21, 2015 at 3:13:11 PM, Chris Barker (chris.barker at noaa.gov) > wrote: > > > > replace all your "import setuptools" with "import setuptool_lite" would > be > > clear what the intent was -- i.e. not YET ANOTHER brand new build > system... > > Moving from one ?one true build system? to another ?one true build system? > is unlikely. sure -- but I'm not even talking about another special purpose build system. A far better approach IMO is the one we?re taking. Define standard > *formats* and let end users select whatever tool they want to create things > that adhere to that format. yes -- fabulous -- but you need API as well as format, yes? i.e if pip is going to be able to find a source package and build and install it, it needs to know how to do that, ideally without knowing what the build tool is. So that's API, yes? (maybe plain old setup.py build" is a fine API... As I understand it, the current "API" that pip knows about is: "some subset of what setuptools provides" So my proposal is here is to provide a way for users to easily use jsut that subset. This would: - give folks like me a way to avoid easy-install, etc... - give all of us a way to start moving our packages to a structure that will be better suited to the upcoming orthogonal build-install toolchains. - give those folks involved in the planning a concrete way to know, for sure what people really need (or really use, anyway), both for features and API. In short -- aside from the distraction issue, this is a way to help us get to the future we want, not an alternative to that future. This lets people create really focused tools for particular use cases, you > can have a dead simple one for pure python things that never need to worry > about the complexity of building C libraries which distutils isn't so bad at, eh? :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 21 22:36:52 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Oct 2015 21:36:52 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: Message-ID: On 21 October 2015 at 20:32, Chris Barker wrote: > I have no idea how we can get from here to there cleanly, but it would it be > that big a deal for pip to look for either setuptools or setuptools_lite, > and neither are there, inject setuptools_lite. Yes it would, unfortunately. The problem is that setup.py is not introspectable - pip simply can't "detect" what is imported (short of fragile approaches such as searching the source for "import\s*setuptools$" or something) without running it. So pip just unilaterally injects setuptools at the moment, even if it's already there. One of the points about having a setup_requires element in static metadata is precisely so that pip *can* introspect what build system is in use and not have to make assumptions/inject. Paul From p.f.moore at gmail.com Wed Oct 21 22:41:35 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Oct 2015 21:41:35 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On 21 October 2015 at 20:41, Chris Barker wrote: > As I understand it, the current "API" that pip knows about is: > > "some subset of what setuptools provides" > > So my proposal is here is to provide a way for users to easily use jsut that > subset. https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface All you need to do is write a setup.py, which can do *anything you like*, doesn't need to be based on distutils, setuptools, or whatever (but it needs to survive the setuptools injection pip currently does, but that's not a big deal). As long as it provides this API, that's fine. The only proviso is that no-one has ever tried this in anger, so it's largely untested so far. Feel free to put it through its paces and provide PRs for the docs or for pip to improve it :-) Paul From marques at displague.com Wed Oct 21 22:26:46 2015 From: marques at displague.com (Marques Johansson) Date: Wed, 21 Oct 2015 20:26:46 +0000 Subject: [Distutils] py2dsc --with-python3=True not producing python3- packages Message-ID: I assume I am doing something wrong, but the man page seems a bit light on the matter. Does it have something to do with "python setup.py" being called instead of "python3 setup.py". I have python3-stdeb=0.8.2-4 installed. wget https://pypi.python.org/packages/source/F/Flask-Login/Flask-Login-0.3.2.tar.gz py2dsc --with-python3=True --no-python2-scripts=True --no-python3-scripts=False --with-python2=False Flask-Login-0.3.2.tar.gz -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- running the following command in directory: deb_dist/tmp_py2dsc/Flask-Login-0.3.2 /usr/bin/python setup.py --command-packages stdeb.command sdist_dsc --dist-dir=/home/marques/src/deb_dist --use-premade-distfile=/home/marques/src/Flask-Login-0.3.2.tar.gz --with-python2=False --with-python3=True --no-python2-scripts=True --no-python3-scripts=False -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- running sdist_dsc running egg_info writing requirements to Flask_Login.egg-info/requires.txt writing Flask_Login.egg-info/PKG-INFO writing top-level names to Flask_Login.egg-info/top_level.txt writing dependency_links to Flask_Login.egg-info/dependency_links.txt reading manifest file 'Flask_Login.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'Flask_Login.egg-info/SOURCES.txt' CALLING dpkg-source -b flask-login-0.3.2 flask-login_0.3.2.orig.tar.gz (in dir /home/marques/src/deb_dist) dpkg-source: info: using source format `3.0 (quilt)' dpkg-source: info: building flask-login using existing ./flask-login_0.3.2.orig.tar.gz dpkg-source: info: building flask-login in flask-login_0.3.2-1.debian.tar.xz dpkg-source: info: building flask-login in flask-login_0.3.2-1.dsc dpkg-buildpackage: source package flask-login dpkg-buildpackage: source version 0.3.2-1 dpkg-buildpackage: source distribution unstable dpkg-buildpackage: source changed by Matthew Frazier < leafstormrush at gmail.com> dpkg-source --before-build flask-login-0.3.2 fakeroot debian/rules clean dh clean --with python3 --buildsystem=pybuild dh_testdir -O--buildsystem=pybuild dh_auto_clean -O--buildsystem=pybuild I: pybuild base:170: python3.4 setup.py clean running clean removing '/home/marques/src/deb_dist/flask-login-0.3.2/.pybuild/pythonX.Y_3.4/build' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.4' does not exist -- can't clean it dh_clean -O--buildsystem=pybuild dpkg-source -b flask-login-0.3.2 dpkg-source: info: using source format `3.0 (quilt)' dpkg-source: info: building flask-login using existing ./flask-login_0.3.2.orig.tar.gz dpkg-source: warning: ignoring deletion of directory Flask_Login.egg-info dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/dependency_links.txt, use --include-removal to override dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/not-zip-safe, use --include-removal to override dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/PKG-INFO, use --include-removal to override dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/requires.txt, use --include-removal to override dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/SOURCES.txt, use --include-removal to override dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/top_level.txt, use --include-removal to override dpkg-source: warning: ignoring deletion of file Flask_Login.egg-info/version_info.json, use --include-removal to override dpkg-source: info: building flask-login in flask-login_0.3.2-1.debian.tar.xz dpkg-source: info: building flask-login in flask-login_0.3.2-1.dsc dpkg-genchanges -S -sa >../flask-login_0.3.2-1_source.changes dpkg-genchanges: including full source code in upload dpkg-source --after-build flask-login-0.3.2 dpkg-buildpackage: full upload (original source is included) dpkg-source: warning: extracting unsigned source package (flask-login_0.3.2-1.dsc) dpkg-source: info: extracting flask-login in flask-login-0.3.2 dpkg-source: info: unpacking flask-login_0.3.2.orig.tar.gz dpkg-source: info: unpacking flask-login_0.3.2-1.debian.tar.xz -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 21 23:09:19 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 14:09:19 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: > > > "some subset of what setuptools provides" > > > > So my proposal is here is to provide a way for users to easily use jsut > that > > subset. > > https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface note above: *easily* use just that subset. All you need to do is write a setup.py, which can do *anything you > like*, doesn't need to be based on distutils, setuptools, or whatever > (but it needs to survive the setuptools injection pip currently does, > but that's not a big deal). not if you don't use setuptools at all-- sure. But this is the big project -- make a whole new build system. I'm suggesting that we could prototype that interface without having to create a fully functional alternative to setuptool+distutils first. Maybe I'll look into the pip-injecting-setuptools code and see if I can come up with anyway to hijack that. But If we take the "cripple setuptools" approach, rather than the "replace setuptools", than pip can still inject it... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Oct 21 23:19:57 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2015 23:19:57 +0200 Subject: [Distutils] Time for a setuptools_lite?? References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: <20151021231957.5508ff48@fsol> On Wed, 21 Oct 2015 18:29:17 +0000 Daniel Holth wrote: > We're not just grumpy, we have seen The One Build System idea tried several > times with bad results. Instead, if you have the inclination, time, and > ability, you could try to build a new competing build system like > http://flit.readthedocs.org/ did, or you could help try to figure out how > to improve support for said competing build systems in pip, or you could > even try to make a pip replacement that is better for distributing end-user > applications. That doesn't seem to follow at all. First sentence you're talking about a build system, second you're talking about distributing end-user applications. Building and distributing are different things. "flit" certainly does nothing for non-trivial build chains. Regards Antoine. From solipsis at pitrou.net Wed Oct 21 23:25:20 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 21 Oct 2015 23:25:20 +0200 Subject: [Distutils] Time for a setuptools_lite?? References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: <20151021232520.63a34414@fsol> On Wed, 21 Oct 2015 21:41:35 +0100 Paul Moore wrote: > On 21 October 2015 at 20:41, Chris Barker wrote: > > As I understand it, the current "API" that pip knows about is: > > > > "some subset of what setuptools provides" > > > > So my proposal is here is to provide a way for users to easily use jsut that > > subset. > > https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface > > All you need to do is write a setup.py, which can do *anything you > like*, Reading this, the CLI options which have to be implemented are completely tied to setuptools' own view of the world. `--single-version-externally-managed`? `--install-headers`? Why should a random build system care about that gunk? What should it do with it? I think Nathaniel's PEP, for all its shortcomings, looked much saner than that piece of ad-hoc specification (a.k.a. "here's the random set of things we're currently doing, let's make a spec out of it"). This is like the Microsoft OOXML of Python packages distribution. Regards Antoine. From cournape at gmail.com Wed Oct 21 23:52:31 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 21 Oct 2015 22:52:31 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: <20151021232520.63a34414@fsol> References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou wrote: > On Wed, 21 Oct 2015 21:41:35 +0100 > Paul Moore wrote: > > On 21 October 2015 at 20:41, Chris Barker wrote: > > > As I understand it, the current "API" that pip knows about is: > > > > > > "some subset of what setuptools provides" > > > > > > So my proposal is here is to provide a way for users to easily use > jsut that > > > subset. > > > > > https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface > > > > All you need to do is write a setup.py, which can do *anything you > > like*, > > Reading this, the CLI options which have to be implemented are > completely tied to setuptools' own view of the world. > `--single-version-externally-managed`? `--install-headers`? Why should > a random build system care about that gunk? What should it do with it? > I agree it is not ideal, but it is not *that* hard. Regarding `--single-version-externally-managed`, if you don't care about setuptools interoperability, I believe it is safe to just ignore the flag (or more exactly do like what distutils does, that is installed directly in site-packages, and make `--single-version-externally-managed` a no-op). install_headers, I am not sure how many projects depend on that. The only projects I know that install headers (numpy, etc...) are not using the setuptools mechanism. If that's deemed useful, I could write a simple implementation that does not use distutils/setuptools but support that interface as an example (I have something similar enough times that I unfortunately know the ropes :) ). > I think Nathaniel's PEP, for all its shortcomings, looked much saner > than that piece of ad-hoc specification (a.k.a. "here's the random set > of things we're currently doing, let's make a spec out of it"). This is > like the Microsoft OOXML of Python packages distribution. > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 21 23:53:47 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Oct 2015 22:53:47 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: <20151021232520.63a34414@fsol> References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On 21 October 2015 at 22:25, Antoine Pitrou wrote: > Reading this, the CLI options which have to be implemented are > completely tied to setuptools' own view of the world. > `--single-version-externally-managed`? `--install-headers`? Why should > a random build system care about that gunk? What should it do with it? Sorry, maybe I didn't explain properly. That's what you can do right now. It's certainly completely tied to setuptools' view of the world, because that's what it was written for. But just because we supply setuptools' options doesn't mean you have to use them (you can just ignore most of the junk). Changing pip to work based on a more tool-neutral API is a bigger piece of work, which has backward compatibility issues. It's doable, but then we're back to the whole process of moving forward with a better solution. As I've already said, we have a plan for that, it's just happening slowly (apparently too slowly for some people, but rather than helping with it they seem happier to propose alternative approaches that we've typically considered in the past and discarded, but hey, it's their free time and they can spend it how they like...) > I think Nathaniel's PEP, for all its shortcomings, looked much saner > than that piece of ad-hoc specification (a.k.a. "here's the random set > of things we're currently doing, let's make a spec out of it"). This is > like the Microsoft OOXML of Python packages distribution. Absolutely 100%. That spec is just an attempt to document what's there, because people keep saying "why do we have to use setuptools?" and the answer is "you don't, you can ignore it as long as you emulate this tiny portion of its API (and most of *that* you can ignore)". Nathaniel's PEP is much closer to what we need - people coming up with a concrete way forward that builds on what we have. And the pushback he got was where his proposal didn't take into account issues we knew needed solving (and the real-time discussion they had seems to have helped bring people's understanding a lot closer together, even though progress seems to have stalled since then. Paul From robertc at robertcollins.net Thu Oct 22 00:00:03 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 22 Oct 2015 11:00:03 +1300 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On 22 October 2015 at 10:53, Paul Moore wrote: > On 21 October 2015 at 22:25, Antoine Pitrou wrote: >> Reading this, the CLI options which have to be implemented are >> completely tied to setuptools' own view of the world. >> `--single-version-externally-managed`? `--install-headers`? Why should >> a random build system care about that gunk? What should it do with it? > > Sorry, maybe I didn't explain properly. That's what you can do right > now. It's certainly completely tied to setuptools' view of the world, > because that's what it was written for. But just because we supply > setuptools' options doesn't mean you have to use them (you can just > ignore most of the junk). > > Changing pip to work based on a more tool-neutral API is a bigger > piece of work, which has backward compatibility issues. It's doable, > but then we're back to the whole process of moving forward with a > better solution. As I've already said, we have a plan for that, it's > just happening slowly (apparently too slowly for some people, but > rather than helping with it they seem happier to propose alternative > approaches that we've typically considered in the past and discarded, > but hey, it's their free time and they can spend it how they like...) > >> I think Nathaniel's PEP, for all its shortcomings, looked much saner >> than that piece of ad-hoc specification (a.k.a. "here's the random set >> of things we're currently doing, let's make a spec out of it"). This is >> like the Microsoft OOXML of Python packages distribution. > > Absolutely 100%. That spec is just an attempt to document what's > there, because people keep saying "why do we have to use setuptools?" > and the answer is "you don't, you can ignore it as long as you emulate > this tiny portion of its API (and most of *that* you can ignore)". > > Nathaniel's PEP is much closer to what we need - people coming up with > a concrete way forward that builds on what we have. And the pushback > he got was where his proposal didn't take into account issues we knew > needed solving (and the real-time discussion they had seems to have > helped bring people's understanding a lot closer together, even though > progress seems to have stalled since then. Nathaniel said he was going to follow up on it with the improvements from the discussion. If he can't, someone else certainly can. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From dholth at gmail.com Thu Oct 22 00:15:17 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 21 Oct 2015 22:15:17 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On Wed, Oct 21, 2015 at 5:52 PM David Cournapeau wrote: > On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou > wrote: > >> On Wed, 21 Oct 2015 21:41:35 +0100 >> Paul Moore wrote: >> > On 21 October 2015 at 20:41, Chris Barker >> wrote: >> > > As I understand it, the current "API" that pip knows about is: >> > > >> > > "some subset of what setuptools provides" >> > > >> > > So my proposal is here is to provide a way for users to easily use >> jsut that >> > > subset. >> > >> > >> https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface >> > >> > All you need to do is write a setup.py, which can do *anything you >> > like*, >> >> Reading this, the CLI options which have to be implemented are >> completely tied to setuptools' own view of the world. >> `--single-version-externally-managed`? `--install-headers`? Why should >> a random build system care about that gunk? What should it do with it? >> > > I agree it is not ideal, but it is not *that* hard. Regarding > `--single-version-externally-managed`, if you don't care about setuptools > interoperability, I believe it is safe to just ignore the flag (or more > exactly do like what distutils does, that is installed directly in > site-packages, and make `--single-version-externally-managed` a no-op). > install_headers, I am not sure how many projects depend on that. The only > projects I know that install headers (numpy, etc...) are not using the > setuptools mechanism. > > If that's deemed useful, I could write a simple implementation that does > not use distutils/setuptools but support that interface as an example (I > have something similar enough times that I unfortunately know the ropes :) ) > Also, it's no longer necessary to implement 'setup.py install' at all; you can just implement bdist_wheel and pip will go from there. Of course if pip can beat us to a better interface that would be great too, but a setup.py shim would be a fun little project that someone could do alone. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 22 00:23:15 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 21 Oct 2015 18:23:15 -0400 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On October 21, 2015 at 6:15:50 PM, Daniel Holth (dholth at gmail.com) wrote: > On Wed, Oct 21, 2015 at 5:52 PM David Cournapeau wrote: > > > On Wed, Oct 21, 2015 at 10:25 PM, Antoine Pitrou > > wrote: > > > >> On Wed, 21 Oct 2015 21:41:35 +0100 > >> Paul Moore wrote: > >> > On 21 October 2015 at 20:41, Chris Barker > >> wrote: > >> > > As I understand it, the current "API" that pip knows about is: > >> > > > >> > > "some subset of what setuptools provides" > >> > > > >> > > So my proposal is here is to provide a way for users to easily use > >> jsut that > >> > > subset. > >> > > >> > > >> https://pip.pypa.io/en/stable/reference/pip_install/#build-system-interface > >> > > >> > All you need to do is write a setup.py, which can do *anything you > >> > like*, > >> > >> Reading this, the CLI options which have to be implemented are > >> completely tied to setuptools' own view of the world. > >> `--single-version-externally-managed`? `--install-headers`? Why should > >> a random build system care about that gunk? What should it do with it? > >> > > > > I agree it is not ideal, but it is not *that* hard. Regarding > > `--single-version-externally-managed`, if you don't care about setuptools > > interoperability, I believe it is safe to just ignore the flag (or more > > exactly do like what distutils does, that is installed directly in > > site-packages, and make `--single-version-externally-managed` a no-op). > > install_headers, I am not sure how many projects depend on that. The only > > projects I know that install headers (numpy, etc...) are not using the > > setuptools mechanism. > > > > If that's deemed useful, I could write a simple implementation that does > > not use distutils/setuptools but support that interface as an example (I > > have something similar enough times that I unfortunately know the ropes :) ) > > > > Also, it's no longer necessary to implement 'setup.py install' at all; you > can just implement bdist_wheel and pip will go from there. > Well, it is if you want it to work with older pips or pip when wheel isn?t also installed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.barker at noaa.gov Thu Oct 22 00:33:49 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 15:33:49 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On Wed, Oct 21, 2015 at 2:53 PM, Paul Moore wrote: > As I've already said, we have a plan for that, it's > just happening slowly (apparently too slowly for some people, but > rather than helping with it they seem happier to propose alternative > approaches that we've typically considered in the past and discarded, > I'm not exactly sure who (or which proposals) that was aimed at, but my idea, at least is not an alternative approach at all -- it's small idea for how to help get there, sooner than later. It seems the plan is to: finish hammering out the spec. Wait for _someone_ to make a new build system. Then wait how long? decades? for the entire user community to go to some new build system(s). Then we can finally deprecate the cruft of setuptools. Actually -- I like that plan -- and I'll probably be one of the early folks to jump on the bandwagon of the new build system. But in the meantime, a way to get my project to build without the setuptools cruft would be a nice option. BTW -- this is not just about letting pip work with more than one build system -- it's also about letting more than one package manage work with the existing (and other future) build systems. In conda, for instance, the convention is to call: python setup.py intstall. It turns out that now, we _should_ be calling: pip install ./ to actually get the right thing done. Really? A setuptools lite would do the right thing with "setup.py install" Hmm, now that I think about it, maybe I can just banish setuptools from my setup.py, stick with distutils, and then make sure to use pip to actually invoke anything -- so it will inject setuptools, but only use the bits we need.... I'll have to try that..... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Oct 22 00:50:51 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 21 Oct 2015 22:50:51 +0000 (UTC) Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: Message-ID: <1807122548.1819446.1445467851327.JavaMail.yahoo@mail.yahoo.com> From: Nick Coghlan > We still need a migration path to modern metadata standards for > everyone using distutils and setuptools - that's the side of things > that caused major problems for both distribute and distutils2. I've thought about the migration path with distil, which uses a declarative metadata format (superset of PEP-426 as it was a while ago) where the actual metadata is extracted from setup.py files in releases already on PyPI. This works well for simple distributions which don't do anything fancy in setup.py, such as extending build commands to create files on the fly which are then installed etc. In my early testing, distil (which uses no distutils or setuptools code) could install and package distributions which were pure-Python as well as package, build and install ones with C extensions and even Fortran extensions. I haven't done much work on it for a while due to other commitments, but when I had the time to work on it, I noted precious little interest from distutils-sig (yes, I know we're all volunteers, but one person can only do so much). IMO the distil approach comes the closest to a least-work migration path: with distutils2 / packaging that was pulled from 3.3, almost everyone would have to convert their releases to setup.cfg, with limited support for automating this and no support for existing stuff on PyPI, which made it a non-starter. Plus, setup.cfg never supported build-side metadata IIRC, only installation-side. With distil, the metadata for build and installation is *automatically* extracted from setup.py with no work required from packagers e.g. for existing releases, and the extraction only fails if the setup.py code does certain operations which can't be inferred from the actual call to setup() in setup.py (I know that those hard-to-infer operations occur in a lot of releases - but that's the nature of setup.py, where anything goes - I still wanted to see how much progress could be made for the simple cases). The metadata extracted has been useful in other contexts, e.g. Brett Cannon's caniusepython3 uses it to work out package dependencies. With distil, the entire dependency tree is calculated before any package is downloaded at all - something people seem to want - but, it seems, not to the extent of actually doing anything concrete, even to try out code and report problems ... distil isn't perfect and I've said it's more like a proof of concept, but it's quite usable for the "non-fancy" distributions. As long as people are free to put whatever code they want in setup.py, we're not really going to have a simple migration path, because a sizeable number of packagers will carry on doing so, and that'll be hard to convert to declarative metadata automagically (not to mention the situation with existing releases). And as long as there is so much inertia stopping interested parties from trying out new approaches, it seems like the pace of development of better tools will continue to be glacial. Regards, Vinay Sajip From contact at ionelmc.ro Thu Oct 22 01:05:12 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 22 Oct 2015 02:05:12 +0300 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <1807122548.1819446.1445467851327.JavaMail.yahoo@mail.yahoo.com> References: <1807122548.1819446.1445467851327.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Thu, Oct 22, 2015 at 1:50 AM, Vinay Sajip wrote: > > I've thought about the migration path with distil, which uses a > declarative metadata format (superset of PEP-426 as it was a while ago) > where the actual metadata is extracted from setup.py files in releases > already on PyPI. This works well for simple distributions which don't do > anything fancy in setup.py, such as extending build commands to create > files on the fly which are then installed etc. In my early testing, distil > (which uses no distutils or setuptools code) could install and package > distributions which were pure-Python as well as package, build and install > ones with C extensions and even Fortran extensions. I haven't done much > work on it for a while due to other commitments, but when I had the time to > work on it, I noted precious little interest from distutils-sig (yes, I > know we're all volunteers, but one person can only do so much). ?What does distil do to achieve this? A setuptools polyfill? Just parsing ?setup.py and inspecting file layout? Is there any code for this or you're just talking about an idea? I couldn't find any distil project in https://bitbucket.org/vinay.sajip/ ... Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 22 01:10:27 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 21 Oct 2015 19:10:27 -0400 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <1807122548.1819446.1445467851327.JavaMail.yahoo@mail.yahoo.com> Message-ID: On October 21, 2015 at 7:05:59 PM, Ionel Cristian M?rie? (contact at ionelmc.ro) wrote: > On Thu, Oct 22, 2015 at 1:50 AM, Vinay Sajip > wrote: > > > > > I've thought about the migration path with distil, which uses a > > declarative metadata format (superset of PEP-426 as it was a while ago) > > where the actual metadata is extracted from setup.py files in releases > > already on PyPI. This works well for simple distributions which don't do > > anything fancy in setup.py, such as extending build commands to create > > files on the fly which are then installed etc. In my early testing, distil > > (which uses no distutils or setuptools code) could install and package > > distributions which were pure-Python as well as package, build and install > > ones with C extensions and even Fortran extensions. I haven't done much > > work on it for a while due to other commitments, but when I had the time to > > work on it, I noted precious little interest from distutils-sig (yes, I > > know we're all volunteers, but one person can only do so much). > > > What does distil do to achieve this? A setuptools polyfill? Just parsing > setup.py and inspecting file layout? > > Is there any code for this or you're just talking about an idea? I couldn't > find any distil project in https://bitbucket.org/vinay.sajip/ ? If I remember correctly distil was not OSS at all, it was distributed as a runnable zip file and if you unzipped it there wasn?t a license at all (or perhaps it was all rights reserved?). I forget the exact details but I recall unzipping it to take a look, seeing it wasn?t OSS and closing it after that. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From robertc at robertcollins.net Thu Oct 22 01:52:16 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 22 Oct 2015 12:52:16 +1300 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On 22 October 2015 at 11:33, Chris Barker wrote: > On Wed, Oct 21, 2015 at 2:53 PM, Paul Moore wrote: >> >> As I've already said, we have a plan for that, it's >> just happening slowly (apparently too slowly for some people, but >> rather than helping with it they seem happier to propose alternative >> approaches that we've typically considered in the past and discarded, > > > I'm not exactly sure who (or which proposals) that was aimed at, but my > idea, at least is not an alternative approach at all -- it's small idea for > how to help get there, sooner than later. > > It seems the plan is to: > > finish hammering out the spec. > > Wait for _someone_ to make a new build system. > > Then wait how long? decades? for the entire user community to go to some new > build system(s). > > Then we can finally deprecate the cruft of setuptools. > > Actually -- I like that plan -- and I'll probably be one of the early folks > to jump on the bandwagon of the new build system. > > But in the meantime, a way to get my project to build without the setuptools > cruft would be a nice option. > > BTW -- this is not just about letting pip work with more than one build > system -- it's also about letting more than one package manage work with the > existing (and other future) build systems. > > In conda, for instance, the convention is to call: > > python setup.py intstall. > > It turns out that now, we _should_ be calling: > > pip install ./ > > to actually get the right thing done. Really? > > A setuptools lite would do the right thing with "setup.py install" > > Hmm, now that I think about it, maybe I can just banish setuptools from my > setup.py, stick with distutils, and then make sure to use pip to actually > invoke anything -- so it will inject setuptools, but only use the bits we > need.... > > I'll have to try that..... You won't gain anything from that. distutils doesn't track installed files either. You *must* use 'pip install .' or 'setup.py bdist_wheel' + 'pip install .' with the current ecosystem state, -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From chris.barker at noaa.gov Thu Oct 22 01:59:56 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Oct 2015 16:59:56 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On Wed, Oct 21, 2015 at 4:52 PM, Robert Collins wrote: > > Hmm, now that I think about it, maybe I can just banish setuptools from > my > > setup.py, stick with distutils, and then make sure to use pip to actually > > invoke anything -- so it will inject setuptools, but only use the bits we > > need.... > > > > I'll have to try that..... > > You won't gain anything from that. distutils doesn't track installed > files either. You *must* use 'pip install .' or 'setup.py bdist_wheel' > + 'pip install .' with the current ecosystem state, > I guess I wasn't clear -- the idea was to force myself to use pip install, rather than ever doing a plain setup.py install or setup.py develop so: pip install ./ or pip install -e ./ this way, pip will inject the parts of setuptools I really need, but hopefully not any other cruft. We'll see. -CHB > > -Rob > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Thu Oct 22 04:27:04 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 22 Oct 2015 15:27:04 +1300 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On 22 October 2015 at 12:59, Chris Barker wrote: > On Wed, Oct 21, 2015 at 4:52 PM, Robert Collins > wrote: >> >> > Hmm, now that I think about it, maybe I can just banish setuptools from >> > my >> > setup.py, stick with distutils, and then make sure to use pip to >> > actually >> > invoke anything -- so it will inject setuptools, but only use the bits >> > we >> > need.... >> > >> > I'll have to try that..... >> >> You won't gain anything from that. distutils doesn't track installed >> files either. You *must* use 'pip install .' or 'setup.py bdist_wheel' >> + 'pip install .' with the current ecosystem state, > > > I guess I wasn't clear -- the idea was to force myself to use pip install, > rather than ever doing a plain > > setup.py install or > setup.py develop > > so: > pip install ./ > or > pip install -e ./ > > this way, pip will inject the parts of setuptools I really need, but > hopefully not any other cruft. Its precisely the same as doing """ import setuptools from distutils import setup ... setup() """ Except that other people may still run setup.py directly, and then you'll be picking up the pieces :). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From guettliml at thomas-guettler.de Thu Oct 22 17:12:41 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 17:12:41 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <1807122548.1819446.1445467851327.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5628FCE9.2040501@thomas-guettler.de> Am 22.10.2015 um 01:10 schrieb Donald Stufft: > On October 21, 2015 at 7:05:59 PM, Ionel Cristian M?rie? (contact at ionelmc.ro) wrote: >> On Thu, Oct 22, 2015 at 1:50 AM, Vinay Sajip >> wrote: >> >>> >>> I've thought about the migration path with distil, which uses a >>> declarative metadata format (superset of PEP-426 as it was a while ago) >>> where the actual metadata is extracted from setup.py files in releases >>> already on PyPI. This works well for simple distributions which don't do >>> anything fancy in setup.py, such as extending build commands to create >>> files on the fly which are then installed etc. In my early testing, distil >>> (which uses no distutils or setuptools code) could install and package >>> distributions which were pure-Python as well as package, build and install >>> ones with C extensions and even Fortran extensions. I haven't done much >>> work on it for a while due to other commitments, but when I had the time to >>> work on it, I noted precious little interest from distutils-sig (yes, I >>> know we're all volunteers, but one person can only do so much). >> >> >> What does distil do to achieve this? A setuptools polyfill? Just parsing >> setup.py and inspecting file layout? >> >> Is there any code for this or you're just talking about an idea? I couldn't >> find any distil project in https://bitbucket.org/vinay.sajip/ ? > > If I remember correctly distil was not OSS at all, it was distributed as a runnable zip file and if you unzipped it there wasn?t a license at all (or perhaps it was all rights reserved?). I forget the exact details but I recall unzipping it to take a look, seeing it wasn?t OSS and closing it after that. Hi Donald, If distil is good software, it's sad that there is no licence. Maybe there is an mail adress to contact the orignal author? Most authors feel proud if they get positive feedback. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 17:46:08 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 17:46:08 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: <5627C5A3.6090004@ronnypfannschmidt.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627C5A3.6090004@ronnypfannschmidt.de> Message-ID: <562904C0.10000@thomas-guettler.de> Am 21.10.2015 um 19:04 schrieb Ronny Pfannschmidt: > another example is that without a standard there is no viable way to > replace a grown messy legacy tool with a much less messy tool over time > creating a new trimmed down standard and supporting it from the legacy > tool as well as from the new tool works out much better than trying to > trim down the legacy tool over time I prefer the code of unittests to the not testable sentences in a standard. But maybe I am missing something. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 17:47:57 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 17:47:57 +0200 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> Message-ID: <5629052D.2070201@thomas-guettler.de> Am 22.10.2015 um 09:28 schrieb anatoly techtonik: > On Wed, Oct 21, 2015 at 10:10 PM, Thomas G?ttler > wrote: > > Am 21.10.2015 um 18:46 schrieb Chris Barker: > > On Wed, Oct 21, 2015 at 9:34 AM, Thomas G?ttler >> wrote: > > > > I ask myself: Why a standard? I see that a standard is very important if there will be > > several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) > > > > But here a single implementation for creating and installing packages would be enough. > > > > Is a standard really needed? > > > > > > Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- > > the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate. > > What happens if the common use cases are inadequate? > > My guess: re-inventing the same stuff over and over again. Once in numpy, once in scipy ... > > Why should it be impossible to get all the needs of numpy and scipy into setuptools? > > I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ... > > Even for complex packages. > > This data gets processed by setuptools. I don't see a need for more than one library doing this. Plugins are nice and can solve edge cases. > > > setuptools is one bloated piece of code that contains too much features. I'd prefer more lean and mean package. Yes, your are rigth. I'd like a lean and mean package, too. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 17:52:50 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 17:52:50 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: <56290652.2000900@thomas-guettler.de> Am 21.10.2015 um 17:29 schrieb Daniel Holth: > The strategy that we are successfully pursuing is to standardize around file formats that the build tools generate and the installers and runtime can consume. > For example we can convert a setuptools egg or a bdist_wininst .exe to a wheel, and in the future we will be able to convert e-mail style PKG-INFO to json. > It's much easier to test whether a tool is producing correct metadata than to decide whether the API is compatible. I don't see any reason to convert a setuptools egg or a bdist_wininst .exe to a wheel. Why should this be done? The egg is something created from sources. Why not create the wheel from the sources? It is like SRPM (source RPMs). I never understood why they exist. Why not create the target from the sources? Regards, Thomas -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 18:01:45 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 18:01:45 +0200 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> Message-ID: <56290869.8070807@thomas-guettler.de> Am 21.10.2015 um 18:46 schrieb Chris Barker: > On Wed, Oct 21, 2015 at 9:34 AM, Thomas G?ttler > wrote: > > I ask myself: Why a standard? I see that a standard is very important if there will be > several implementations (for example TCP/IP, protocols like HTTP, SMTP, IMAP, ...) > > But here a single implementation for creating and installing packages would be enough. > > Is a standard really needed? > > > Yes -- because of exactly what you say above -- we really can't have a SINGLE build system that will well support everything -- > the common use-caes, sure (distutils already does that), but when it comes to bulding complex packages like numpy, sciPy, etc, it's really inadequate. Yes, you are right: "we really can't have a SINGLE build system that will well support everything". But maybe a single build systems which has a well documented plugin API can solve all needs? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 18:04:31 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 18:04:31 +0200 Subject: [Distutils] Monkey patching distutils was: Remove distutils, was: red, green, refactor ... In-Reply-To: <20151021171539.2389c1c5@fsol> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <20151021171539.2389c1c5@fsol> Message-ID: <5629090F.6060808@thomas-guettler.de> Am 21.10.2015 um 17:15 schrieb Antoine Pitrou: > On Wed, 21 Oct 2015 17:05:29 +0200 > Nick Coghlan wrote: >> On 21 October 2015 at 14:55, David Cournapeau wrote: >>> >>> On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler >>> wrote: >>>> ok, at the moment setuptools uses distutils. >>>> >>>> Why not melt them together into **one** underwear-pants-module? >>> >>> >>> What do you hope getting from that ? distutils is in the stdlib, so cannot >>> change easily, and even if putting setuptools in the stdlib were possible, >>> you would now need to handle different versions of setuptools for different >>> versions of python. >> >> It's more useful to go the other direction and vendor a modern version >> of distutils inside setuptools: > > It seems it would only add a bit more craziness to the current > landscape. What happens to projects which have a need to monkeypatch > distutils? How does it interact with the vendored version? etc. What are the needs to monkeypatch distutils? I guess there are good reasons to do so. So why not implement this directly there, so that no monkey patching is needed any more? What are "vendored version" in this context? Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 18:07:19 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 18:07:19 +0200 Subject: [Distutils] Migration Path to metadata was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: <562909B7.90003@thomas-guettler.de> Am 21.10.2015 um 17:05 schrieb Nick Coghlan: > On 21 October 2015 at 14:55, David Cournapeau wrote: >> >> On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler >> wrote: >>> ok, at the moment setuptools uses distutils. >>> >>> Why not melt them together into **one** underwear-pants-module? >> >> >> What do you hope getting from that ? distutils is in the stdlib, so cannot >> change easily, and even if putting setuptools in the stdlib were possible, >> you would now need to handle different versions of setuptools for different >> versions of python. > > It's more useful to go the other direction and vendor a modern version > of distutils inside setuptools: > https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils > > distutils can then optionally be replaced wholesale at runtime, rather > than having the internals be monkeypatched. > >> On top of this, the goal of lots of efforts around packaging is to allow >> people to move away from distutils/setuptools, as the underlying design is >> fundamentally difficult to extend. > > We still need a migration path to modern metadata standards for > everyone using distutils and setuptools - that's the side of things > that caused major problems for both distribute and distutils2. I guess you have a rough migration path in your mind? I guess some people here are interested. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Thu Oct 22 18:15:37 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 22 Oct 2015 18:15:37 +0200 Subject: [Distutils] Undelying design is fundamentally difficult to extend was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> Message-ID: <56290BA9.7060602@thomas-guettler.de> Am 21.10.2015 um 14:55 schrieb David Cournapeau: > > > On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler > wrote: > > Am 21.10.2015 um 13:28 schrieb Ionel Cristian M?rie?: > > > On Wed, Oct 21, 2015 at 1:42 PM, Thomas G?ttler >> wrote: > > > Why not deprecate and remove distutils? > > > ?Should we or can we? There was an attempt at that, called distutils2. I'd love to hear what people have learnt from that. > > And if now-days walking around with pants instead of only underwear is the way to go, should we still have underwear > under the pants? I think yes :-) In other words, setuptools uses distutils. > > > ok, at the moment setuptools uses distutils. > > Why not melt them together into **one** underwear-pants-module? > > > What do you hope getting from that ? distutils is in the stdlib, so cannot change easily > , and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python. I hope to be getting this: Packaging Python will be KISS. If setuptools in inside Python, then different versions of Python each have their matching version included. If a package has the need for newer version of setuptool, it provides the hint somewhere: required_setuptools_version>=N.M Then the matching version from pypi gets downloaded first. Looks straight forward for me. > On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend. If the underlying design is fundamentally difficult to extend, what should be done? Build tools on top of it? I guess this is not a good idea. Or start from scratch and deprecate the current underlying design in the future? What do you think? Regards, Thomas -- http://www.thomas-guettler.de/ From cournape at gmail.com Thu Oct 22 18:15:46 2015 From: cournape at gmail.com (David Cournapeau) Date: Thu, 22 Oct 2015 17:15:46 +0100 Subject: [Distutils] Monkey patching distutils was: Remove distutils, was: red, green, refactor ... In-Reply-To: <5629090F.6060808@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <20151021171539.2389c1c5@fsol> <5629090F.6060808@thomas-guettler.de> Message-ID: On Thu, Oct 22, 2015 at 5:04 PM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > Am 21.10.2015 um 17:15 schrieb Antoine Pitrou: > > On Wed, 21 Oct 2015 17:05:29 +0200 > > Nick Coghlan wrote: > >> On 21 October 2015 at 14:55, David Cournapeau > wrote: > >>> > >>> On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler > >>> wrote: > >>>> ok, at the moment setuptools uses distutils. > >>>> > >>>> Why not melt them together into **one** underwear-pants-module? > >>> > >>> > >>> What do you hope getting from that ? distutils is in the stdlib, so > cannot > >>> change easily, and even if putting setuptools in the stdlib were > possible, > >>> you would now need to handle different versions of setuptools for > different > >>> versions of python. > >> > >> It's more useful to go the other direction and vendor a modern version > >> of distutils inside setuptools: > > > > It seems it would only add a bit more craziness to the current > > landscape. What happens to projects which have a need to monkeypatch > > distutils? How does it interact with the vendored version? etc. > > What are the needs to monkeypatch distutils? > > That's the only way to extend distutils in a meaningful way, because of the broken command concept (e.g. the command dependency graph is defined inside the classes instead of being defined by an external scheduler). As anybody who has dealt with that codebase will tell you, It is impossible to fix distutils without breaking most non trivial uses of it. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Oct 22 18:15:36 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 22 Oct 2015 09:15:36 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> <20151021232520.63a34414@fsol> Message-ID: On Wed, Oct 21, 2015 at 7:27 PM, Robert Collins wrote: > > I guess I wasn't clear -- the idea was to force myself to use pip > install, > > rather than ever doing a plain > > > > setup.py install or > > setup.py develop > > > > so: > > pip install ./ > > or > > pip install -e ./ > > > > this way, pip will inject the parts of setuptools I really need, but > > hopefully not any other cruft. > > Its precisely the same as doing > > """ > import setuptools > from distutils import setup > > ... > setup() > > """ > > Except that other people may still run setup.py directly, and then > you'll be picking up the pieces :). exactly! That is the point -- pip simply injects the full-on setuptools, but it calls it in specific ways -- i.e. --single-version-externally-managed and all that. But, you also wouldn't get things like setuptools find_packages and the like -- though I"m not sure I've ever liked that... The trick is to make it clear to your users that this particular setup.py is not expected to work right unless run from pip.... - CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Oct 22 18:24:07 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 22 Oct 2015 18:24:07 +0200 Subject: [Distutils] Undelying design is fundamentally difficult to extend was: Remove distutils, was: red, green, refactor ... References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <56290BA9.7060602@thomas-guettler.de> Message-ID: <20151022182407.0d296208@fsol> On Thu, 22 Oct 2015 18:15:37 +0200 Thomas G?ttler wrote: > > > On top of this, the goal of lots of efforts around packaging is to allow people to move away from distutils/setuptools, as the underlying design is fundamentally difficult to extend. > > If the underlying design is fundamentally difficult to extend, what should be done? > > Build tools on top of it? I guess this is not a good idea. > > Or start from scratch and deprecate the current underlying design in the future? Providing official hooks would be a way to provide extension facilities. Of course this assumes someone would actually maintain distutils, and "the community" would stop making a ruckus everytime an extremely minor behaviour change is considered (otherwise you get what happened to distutils2). Regards Antoine. From chris.barker at noaa.gov Thu Oct 22 18:27:42 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 22 Oct 2015 09:27:42 -0700 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: <5629052D.2070201@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> <5629052D.2070201@thomas-guettler.de> Message-ID: On Thu, Oct 22, 2015 at 8:47 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > > I have a dream: For packaging and building package provides only > **data**. Data is condition-less: No single "if", "def" or method call. > Just data: json or yaml ... > > > > Even for complex packages. > It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front. But we do have a bad situtation now where you can't know the metadata until you've run setup.py -- not good. Maybe we need to add a configure step: a configure_package.py that would return the full set of metadata -- or maybe a dict that could be passed right on to: setup(**configuration) In the simple cases, this would simply return a static dict, but it would be a hook where people could do whatever they need. Perhaps sdists would have had the configure step already run -- not sure about that one off the bat, but it would be kind of like distributing a source tarball with autotools already run. I think this may be a bit where the sdist conversation was going (at least on one side-track...): you might have sdists that are customized to the platform: "this sdist is for numpy running on Windows with OpenBLAS" that sort of thing. I THINK, once you'd broken it down to that level, you could have static configuration. -Chris > > This data gets processed by setuptools. I don't see a need for more > than one library doing this. Plugins are nice and can solve edge cases. > > > > > > setuptools is one bloated piece of code that contains too much features. > I'd prefer more lean and mean package. > > Yes, your are rigth. I'd like a lean and mean package, too. > > Regards, > Thomas G?ttler > > > > -- > http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Oct 22 18:31:26 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 22 Oct 2015 09:31:26 -0700 Subject: [Distutils] Undelying design is fundamentally difficult to extend was: Remove distutils, was: red, green, refactor ... In-Reply-To: <56290BA9.7060602@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <56290BA9.7060602@thomas-guettler.de> Message-ID: On Thu, Oct 22, 2015 at 9:15 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > If setuptools in inside Python, then different versions of Python each > have their matching version included. > > If a package has the need for newer version of setuptool, it provides the > hint somewhere: required_setuptools_version>=N.M > > Then the matching version from pypi gets downloaded first. > Isn't this what ensurepip already does??? Sure, it's called ensure *pip* but pip requires setuptools, so you get that out of the box. If in the future we have some nifty replacement for both setuptools and distutils, pip can just grab that, too... I think we're at the "someone needs to actually write the code" stage.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Oct 22 18:24:53 2015 From: brett at python.org (Brett Cannon) Date: Thu, 22 Oct 2015 16:24:53 +0000 Subject: [Distutils] py2dsc --with-python3=True not producing python3- packages In-Reply-To: References: Message-ID: Distutils-sig has nothing to do with py2dsc so I doubt anyone can really help here unfortunately. I would try reaching out to py2dsc or the flask-login team. On Wed, 21 Oct 2015 at 13:51 Marques Johansson wrote: > I assume I am doing something wrong, but the man page seems a bit light on > the matter. Does it have something to do with "python setup.py" being > called instead of "python3 setup.py". I have python3-stdeb=0.8.2-4 > installed. > > wget > https://pypi.python.org/packages/source/F/Flask-Login/Flask-Login-0.3.2.tar.gz > py2dsc --with-python3=True --no-python2-scripts=True > --no-python3-scripts=False --with-python2=False Flask-Login-0.3.2.tar.gz > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > running the following command in directory: > deb_dist/tmp_py2dsc/Flask-Login-0.3.2 > /usr/bin/python setup.py --command-packages stdeb.command sdist_dsc > --dist-dir=/home/marques/src/deb_dist > --use-premade-distfile=/home/marques/src/Flask-Login-0.3.2.tar.gz > --with-python2=False --with-python3=True --no-python2-scripts=True > --no-python3-scripts=False > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > running sdist_dsc > running egg_info > writing requirements to Flask_Login.egg-info/requires.txt > writing Flask_Login.egg-info/PKG-INFO > writing top-level names to Flask_Login.egg-info/top_level.txt > writing dependency_links to Flask_Login.egg-info/dependency_links.txt > reading manifest file 'Flask_Login.egg-info/SOURCES.txt' > reading manifest template 'MANIFEST.in' > writing manifest file 'Flask_Login.egg-info/SOURCES.txt' > CALLING dpkg-source -b flask-login-0.3.2 flask-login_0.3.2.orig.tar.gz (in > dir /home/marques/src/deb_dist) > dpkg-source: info: using source format `3.0 (quilt)' > dpkg-source: info: building flask-login using existing > ./flask-login_0.3.2.orig.tar.gz > dpkg-source: info: building flask-login in > flask-login_0.3.2-1.debian.tar.xz > dpkg-source: info: building flask-login in flask-login_0.3.2-1.dsc > dpkg-buildpackage: source package flask-login > dpkg-buildpackage: source version 0.3.2-1 > dpkg-buildpackage: source distribution unstable > dpkg-buildpackage: source changed by Matthew Frazier < > leafstormrush at gmail.com> > dpkg-source --before-build flask-login-0.3.2 > fakeroot debian/rules clean > dh clean --with python3 --buildsystem=pybuild > dh_testdir -O--buildsystem=pybuild > dh_auto_clean -O--buildsystem=pybuild > I: pybuild base:170: python3.4 setup.py clean > running clean > removing > '/home/marques/src/deb_dist/flask-login-0.3.2/.pybuild/pythonX.Y_3.4/build' > (and everything under it) > 'build/bdist.linux-x86_64' does not exist -- can't clean it > 'build/scripts-3.4' does not exist -- can't clean it > dh_clean -O--buildsystem=pybuild > dpkg-source -b flask-login-0.3.2 > dpkg-source: info: using source format `3.0 (quilt)' > dpkg-source: info: building flask-login using existing > ./flask-login_0.3.2.orig.tar.gz > dpkg-source: warning: ignoring deletion of directory Flask_Login.egg-info > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/dependency_links.txt, use --include-removal to override > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/not-zip-safe, use --include-removal to override > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/PKG-INFO, use --include-removal to override > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/requires.txt, use --include-removal to override > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/SOURCES.txt, use --include-removal to override > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/top_level.txt, use --include-removal to override > dpkg-source: warning: ignoring deletion of file > Flask_Login.egg-info/version_info.json, use --include-removal to override > dpkg-source: info: building flask-login in > flask-login_0.3.2-1.debian.tar.xz > dpkg-source: info: building flask-login in flask-login_0.3.2-1.dsc > dpkg-genchanges -S -sa >../flask-login_0.3.2-1_source.changes > dpkg-genchanges: including full source code in upload > dpkg-source --after-build flask-login-0.3.2 > dpkg-buildpackage: full upload (original source is included) > dpkg-source: warning: extracting unsigned source package > (flask-login_0.3.2-1.dsc) > dpkg-source: info: extracting flask-login in flask-login-0.3.2 > dpkg-source: info: unpacking flask-login_0.3.2.orig.tar.gz > dpkg-source: info: unpacking flask-login_0.3.2-1.debian.tar.xz > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Oct 22 18:53:52 2015 From: brett at python.org (Brett Cannon) Date: Thu, 22 Oct 2015 16:53:52 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: Message-ID: On Wed, 21 Oct 2015 at 10:17 Paul Moore wrote: > On 21 October 2015 at 17:42, Chris Barker wrote: > > So why not have a setuptools-lite that only does the building? We need to > > keep the full over-burdened setuptools around, because lot sof folks are > > using those features. But for those of us that are doing something fairly > > new, and don't want to use stuff that setuptools "shouldn't" be doing, > I'd > > love to have a setuptools-lite that only did what I really need, and was > > guaranteed NOT to accidentally introduce easy_install, etc... > > In general, this sounds like a reasonable idea. But, there are the > usual problems: > > 1. Someone needs to do the work. > 2. Backward compatibility (as in, there's bound to be some projects > out there using whatever functionality you can think of that "nobody > would ever need"...) > > And a third issue specific to this idea. > > 3. You can't call it setuptools, and pip "injects" the real setuptools > into projects when building/installing. Pretending to be setuptools > was what made distribute such a PITA to deal with, and I don't imagine > we'd ever want to go there again. > > Sorry, I'm being a downbeat grump today. The reality is that far and > away the best shot we've had at getting out of this mess is the plan > we're currently working to, which is to standardise interfaces, and > once we have some standards, we can start decoupling the various bits > and *then* replacing them as needed. We need people working on that in > the first instance, work is currently stalled by lack of time from the > key participants. So anyone with energy for moving this whole area > forward would be really welcome to help out with the interoperability > standards (metadata 2.0, defining a sdist format, pinning down pip's > interfaces to build systems, things like that). But if you want to > rework the existing toolset, you're probably going to be on your own > (and there's unfortunately a risk that you'll pull attention away from > other work via debates on this list etc). > > I really appreciate the enthusiasm we're seeing from people on the > list at the moment, it's a matter of harnessing that into the right > direction. As a thought, does anyone feel like working on an > "interoperability standards roadmap" that provides a reference for > where the PyPA see things going? It would be useful to keep focus, but > the information's currently scattered about various mailing lists and > discussions - some of the posts around the recent "new source > distribution format" threads might be a good place to start. > Would it makes sense to start a roadmap doc/repo under the PyPA account so the current grand vision is written down in a very high-level overview and then what the issues needing solving for each bit are along with any in-progress work are kept in a single place (I don't care if this is one big doc or there is the overview doc and then an individual doc per part of the grand vision)? Putting it up on GitHub allows for quick PRs on some Markdown doc that people can do quickly after an email thread reaches some form of a conclusion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Sat Oct 24 08:02:46 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Sat, 24 Oct 2015 14:02:46 +0200 Subject: [Distutils] Migration Path to metadata was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <562909B7.90003@thomas-guettler.de> Message-ID: <562B7366.8010506@thomas-guettler.de> Am 23.10.2015 um 21:12 schrieb Nick Coghlan: > On 22 October 2015 at 18:07, Thomas G?ttler > wrote: >> Am 21.10.2015 um 17:05 schrieb Nick Coghlan: >>> On 21 October 2015 at 14:55, David Cournapeau wrote: >>>> >>>> On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler >>>> wrote: >>>>> ok, at the moment setuptools uses distutils. >>>>> >>>>> Why not melt them together into **one** underwear-pants-module? >>>> >>>> >>>> What do you hope getting from that ? distutils is in the stdlib, so cannot >>>> change easily, and even if putting setuptools in the stdlib were possible, >>>> you would now need to handle different versions of setuptools for different >>>> versions of python. >>> >>> It's more useful to go the other direction and vendor a modern version >>> of distutils inside setuptools: >>> https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils >>> >>> distutils can then optionally be replaced wholesale at runtime, rather >>> than having the internals be monkeypatched. >>> >>>> On top of this, the goal of lots of efforts around packaging is to allow >>>> people to move away from distutils/setuptools, as the underlying design is >>>> fundamentally difficult to extend. >>> >>> We still need a migration path to modern metadata standards for >>> everyone using distutils and setuptools - that's the side of things >>> that caused major problems for both distribute and distutils2. >> >> I guess you have a rough migration path in your mind? I guess some >> people here are interested. > > The people here are the ones already building it: ensuring setuptools > can generate any required metadata, using pip to inject setuptools > into the build process for projects using plain distutils. > > Providing the option for folks that don't want to use distutils or > setuptools to be able to use something else and have pip handle that > reliably would be a nice addition, especially if it's done in a > beginner friendly way that can become the new default build tool > recommendation, but ideas for improvement that start with expecting > tens of thousands of project owners to change the build system they > use aren't going to work. yes, you are right. Several small steps are better than one big. -- http://www.thomas-guettler.de/ From p.f.moore at gmail.com Sat Oct 24 17:57:27 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 24 Oct 2015 22:57:27 +0100 Subject: [Distutils] New PyPUG Tutorials In-Reply-To: References: Message-ID: On 24 October 2015 at 20:25, Marcus Smith wrote: > As someone who has slowly chipped away on the PyPUG for the last 2 years, > I'm excited about the prospect of being able to make a big jump in quality > very quickly. Agreed - this is great news :-) Paul From qwcode at gmail.com Sat Oct 24 15:25:13 2015 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 24 Oct 2015 12:25:13 -0700 Subject: [Distutils] New PyPUG Tutorials Message-ID: As some of you may already know, Nicole from the Warehouse team has initiated an effort to improve our packaging tutorials, and have them linked in the new Warehouse UI (https://github.com/pypa/warehouse/issues/729 ). Although originally the idea was to create additional tutorials (but still maintain consistency with the "Python Packaging User Guide", aka "PyPUG"), after further discussions, we've decided to just improve the 2 existing PyPUG sections that we have (for installation and distribution), with the goal being to improve the accessibilty for new users, and in general to improve the quality and overall style and experience. Nicole has already gathered a group of volunteer writers, and there's a plan in place to manage the process from planning to release. To have the most success, the writers will certainly need feedback from subject matter experts, so the process will include 2 stages where we specifically ask for feedback from PyPA-Dev and Distutils-Sig: 1) To validate the initial proposal that covers the scope of the changes, and 2) to review the actual PRs to PyPUG for accuracy, when it's time for merging. I'll post again with more details as those stages occur. As someone who has slowly chipped away on the PyPUG for the last 2 years, I'm excited about the prospect of being able to make a big jump in quality very quickly. --Marcus P.S. for anyone here who wants to be on the core writing team, see https://github.com/pypa/warehouse/issues/729 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 23 17:32:17 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2015 23:32:17 +0200 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On 23 October 2015 at 23:12, Nick Coghlan wrote: > The core problem is that the status quo is merely clumsy and > inelegant, rather than completely unusable. For all their flaws, both > hobbyists and professional developers *can* use the current tools to > distribute software (and they're still easier to deal with than > writing directly for the native installer toolsets on each target > platform), while in any organisation that uses Python heavily, setup > scripts for new projects are likely to be either generated from > templates or copied from those for old projects, rather than needing > to be written from scratch each time. And now that I've got my "Mr grumpy pants" reaction to the earlier parts of the thread out of my system (sorry about that), I'd like to revisit Chris's idea of having a *simple* replacement for setuptools as a build system. In particular, what if there was a build system aimed specifically at beginners that, at least initially, *only* supported the creation of sdist's and wheels for pure Python source publication (no "install" or "upload" commands), without support for binary extensions, and defaulted to publishing everything in the current directory and below, while relying on twine to actually do the uploads to PyPI? Folks dealing with C extensions (including via cffi or Cython), using pkg_resources for plugins, installing command line wrappers, or otherwise doing something more than publishing a pure Python library would still need to "graduate" to dealing with the complexities of setuptools or one of its competitors, but we could potentially get it out of the initial learning curve for students publishing list-of-list printers :) One particularly nice aspect of that is anyone interested in working on it could potentially pitch it to the PSF's Education Working Group as a project worth investigating as part of the Education bundle. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Oct 25 19:22:12 2015 From: donald at stufft.io (Donald Stufft) Date: Sun, 25 Oct 2015 19:22:12 -0400 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On October 25, 2015 at 7:17:12 PM, Nick Coghlan (ncoghlan at gmail.com) wrote: > > And now that I've got my "Mr grumpy pants" reaction to the earlier > parts of the thread out of my system (sorry about that), I'd like to > revisit Chris's idea of having a *simple* replacement for setuptools > as a build system. > I?m not *against* it, but it has some problems: * Older versions of pip and setuptools will not be able to install it at all. This includes anyplace setuptools is used implicitly such as ``setup.py install`` from another package that depends on it or setup_requires or test_requires etc. * A lot of downstream folks will probably be mad at us for introducing a second way of doing things (particularly since it won?t come with the agile build system support that we have on the road map that will make this a lot better for non setuptools build systems). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From njs at pobox.com Sun Oct 25 19:23:42 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Oct 2015 16:23:42 -0700 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Fri, Oct 23, 2015 at 2:32 PM, Nick Coghlan wrote: > On 23 October 2015 at 23:12, Nick Coghlan wrote: >> The core problem is that the status quo is merely clumsy and >> inelegant, rather than completely unusable. For all their flaws, both >> hobbyists and professional developers *can* use the current tools to >> distribute software (and they're still easier to deal with than >> writing directly for the native installer toolsets on each target >> platform), while in any organisation that uses Python heavily, setup >> scripts for new projects are likely to be either generated from >> templates or copied from those for old projects, rather than needing >> to be written from scratch each time. > > And now that I've got my "Mr grumpy pants" reaction to the earlier > parts of the thread out of my system (sorry about that), I'd like to > revisit Chris's idea of having a *simple* replacement for setuptools > as a build system. > > In particular, what if there was a build system aimed specifically at > beginners that, at least initially, *only* supported the creation of > sdist's and wheels for pure Python source publication (no "install" or > "upload" commands), without support for binary extensions, and > defaulted to publishing everything in the current directory and below, > while relying on twine to actually do the uploads to PyPI? This isn't terribly far from what flit already is. -n -- Nathaniel J. Smith -- http://vorpus.org From gandalf at shopzeus.com Thu Oct 22 01:40:03 2015 From: gandalf at shopzeus.com (=?UTF-8?Q?Nagy_L=c3=a1szl=c3=b3_Zsolt?=) Date: Thu, 22 Oct 2015 07:40:03 +0200 Subject: [Distutils] Is this a bug? Message-ID: <562876B3.1010607@shopzeus.com> Today I have tried to register and upload a new package by executing setup.py register I was asked if I want to save the creditentials in a .pypirc file and I answered yes. Next I wanted to run setup.py upload and I got this error: Traceback (most recent call last): File "C:\Python\Projects\some_package\setup.py", line 15, in classifiers=['Topic :: Security', 'Topic :: Internet :: WWW/HTTP'], File "C:\Python35\lib\distutils\core.py", line 148, in setup dist.run_commands() File "C:\Python35\lib\distutils\dist.py", line 955, in run_commands self.run_command(cmd) File "C:\Python35\lib\distutils\dist.py", line 973, in run_command cmd_obj.ensure_finalized() File "C:\Python35\lib\distutils\cmd.py", line 107, in ensure_finalized self.finalize_options() File "C:\Python35\lib\distutils\command\upload.py", line 46, in finalize_options config = self._read_pypirc() File "C:\Python35\lib\distutils\config.py", line 83, in _read_pypirc current[key] = config.get(server, key) File "C:\Python35\lib\configparser.py", line 798, in get d) File "C:\Python35\lib\configparser.py", line 396, in before_get self._interpolate_some(parser, option, L, value, section, defaults, 1) File "C:\Python35\lib\configparser.py", line 445, in _interpolate_some "found: %r" % (rest,)) configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found : *****************' Instead of the many stars, of course there is the password. The problem might be that the password contains a % character, and it was incorrectly saved by distutils. Can somebody please confirm that this is a bug in distutils? Thanks, Laszlo From techtonik at gmail.com Thu Oct 22 03:28:57 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 22 Oct 2015 10:28:57 +0300 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: <5627E33E.8030305@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> Message-ID: On Wed, Oct 21, 2015 at 10:10 PM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: > Am 21.10.2015 um 18:46 schrieb Chris Barker: > > On Wed, Oct 21, 2015 at 9:34 AM, Thomas G?ttler < > guettliml at thomas-guettler.de > wrote: > > > > I ask myself: Why a standard? I see that a standard is very > important if there will be > > several implementations (for example TCP/IP, protocols like HTTP, > SMTP, IMAP, ...) > > > > But here a single implementation for creating and installing > packages would be enough. > > > > Is a standard really needed? > > > > > > Yes -- because of exactly what you say above -- we really can't have a > SINGLE build system that will well support everything -- > > the common use-caes, sure (distutils already does that), but when it > comes to bulding complex packages like numpy, sciPy, etc, it's really > inadequate. > > What happens if the common use cases are inadequate? > > My guess: re-inventing the same stuff over and over again. Once in numpy, > once in scipy ... > > Why should it be impossible to get all the needs of numpy and scipy into > setuptools? > > I have a dream: For packaging and building package provides only **data**. > Data is condition-less: No single "if", "def" or method call. Just data: > json or yaml ... > > Even for complex packages. > > This data gets processed by setuptools. I don't see a need for more than > one library doing this. Plugins are nice and can solve edge cases. > setuptools is one bloated piece of code that contains too much features. I'd prefer more lean and mean package. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Thu Oct 22 05:46:04 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 22 Oct 2015 12:46:04 +0300 Subject: [Distutils] Data centric standard for distributing Python modules Message-ID: I see a lot of talks about setuptools/no setuptools/distutils and let me (as usual) state my opinion that all of that is wrong. Instead of discussing who will write the code (which is essentially the root problem) how about just setting the data processing requirements and cover them with tests and stories? Ship supporting infrastructure that allows people from Fedora, Debian, Anaconda see HOW DATA about packages is managed and adopt their tools to support Python style. This will save us all a lot of effort in maintaining the stuff between different systems. As you may see - PEPs are good, but not effective. The testsuite that you can download and test your tools against - that would be awesome thing both for tool development and for LEARNING Python. Why learning is here? Just a simple data definition is not enough - you need to understand why the data structure is there in the first place - for what story it is made for? And when you have the story and the data - it is everything that is needed for a practical Python lesson about how to process files and data formats. Stories are scenarios. My usual scenario is: - install files from a package into local python When you think in data centric way, do you need to know the files that you need to copy from a package into local Python. Yes. But do you need a data format for that into? If you're using wheels - no. Because there are no needless flies. If you're using .rpm? Yes, you need to parse and exclude .spec files etc. Every author may choose their own input - we just define the common output. - uninstall files from local python Here you need the list of files. Where do you get it from? >From some custom specific format located in a directory named after some complicated calculated that is separated over separate PEPs that requires a separate module to calculate. You see. It is where everybody in a trap of using setuptools, distutils etc. I just want to query static data - "give me all files for this package" or "give me all packages with this name so that I can choose specific versions". What should be the data layout to allow that? - create isolated virtualenv from existing files I am pretty sure it is not possible now and everybody is redownloading the stuff again and again to create new environment. - check all my installed modules for security issues - check all my virtualenvs for security issues The last part requires a place for virtualenvs to register their location, so that you can read their modules and give security warning about them too. With test coverage (and beautiful and easy runner) this will allow people to write final tools in a fun way. And if people have fun, they usually create better things than those who are forced to do their thing for a living. And it will be more fun if they can easily extend tests for new stories. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 23 15:12:41 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2015 21:12:41 +0200 Subject: [Distutils] Migration Path to metadata was: Remove distutils, was: red, green, refactor ... In-Reply-To: <562909B7.90003@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <562909B7.90003@thomas-guettler.de> Message-ID: On 22 October 2015 at 18:07, Thomas G?ttler wrote: > Am 21.10.2015 um 17:05 schrieb Nick Coghlan: >> On 21 October 2015 at 14:55, David Cournapeau wrote: >>> >>> On Wed, Oct 21, 2015 at 12:52 PM, Thomas G?ttler >>> wrote: >>>> ok, at the moment setuptools uses distutils. >>>> >>>> Why not melt them together into **one** underwear-pants-module? >>> >>> >>> What do you hope getting from that ? distutils is in the stdlib, so cannot >>> change easily, and even if putting setuptools in the stdlib were possible, >>> you would now need to handle different versions of setuptools for different >>> versions of python. >> >> It's more useful to go the other direction and vendor a modern version >> of distutils inside setuptools: >> https://bitbucket.org/pypa/setuptools/issues/417/adopt-distutils >> >> distutils can then optionally be replaced wholesale at runtime, rather >> than having the internals be monkeypatched. >> >>> On top of this, the goal of lots of efforts around packaging is to allow >>> people to move away from distutils/setuptools, as the underlying design is >>> fundamentally difficult to extend. >> >> We still need a migration path to modern metadata standards for >> everyone using distutils and setuptools - that's the side of things >> that caused major problems for both distribute and distutils2. > > I guess you have a rough migration path in your mind? I guess some > people here are interested. The people here are the ones already building it: ensuring setuptools can generate any required metadata, using pip to inject setuptools into the build process for projects using plain distutils. Providing the option for folks that don't want to use distutils or setuptools to be able to use something else and have pip handle that reliably would be a nice addition, especially if it's done in a beginner friendly way that can become the new default build tool recommendation, but ideas for improvement that start with expecting tens of thousands of project owners to change the build system they use aren't going to work. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Oct 23 15:19:36 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2015 21:19:36 +0200 Subject: [Distutils] Undelying design is fundamentally difficult to extend was: Remove distutils, was: red, green, refactor ... In-Reply-To: <56290BA9.7060602@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <56290BA9.7060602@thomas-guettler.de> Message-ID: On 22 October 2015 at 18:15, Thomas G?ttler wrote: > Am 21.10.2015 um 14:55 schrieb David Cournapeau: >> What do you hope getting from that ? distutils is in the stdlib, so cannot change easily >> , and even if putting setuptools in the stdlib were possible, you would now need to handle different versions of setuptools for different versions of python. > > I hope to be getting this: Packaging Python will be KISS. > > If setuptools in inside Python, then different versions of Python each have their matching version included. No, this is what got us into the problem with distutils: we want the packaging system to be *version independent*, as many projects want to support multiple versions of Python. Having the build system be runtime version dependent turns out to be a huge pain. This is why Microsoft versions Visual Studio separately from Windows, Apple versions Xcode separately from Mac OS X, and Red Hat versions the RHEL Developer Toolset separately from RHEL itself. However, we also have more than a decade's worth of existing projects on PyPI that we can't force to switch to a new build system, so we need to ensure any improvements we make benefit *existing* projects, even if they never get updated again. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From robertc at robertcollins.net Sun Oct 25 20:21:25 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 26 Oct 2015 13:21:25 +1300 Subject: [Distutils] Is this a bug? In-Reply-To: <562876B3.1010607@shopzeus.com> References: <562876B3.1010607@shopzeus.com> Message-ID: I can't confirm or deny that with the data given:) Have a look in your .pypirc file - don't share it, it has your password in it - but see if it looks sane. If you can't tell, mask out your username and password and share the content. If you do that, please be sure to note exactly how the % in your password was encoded in the file. -Rob On 22 October 2015 at 18:40, Nagy L?szl? Zsolt wrote: > Today I have tried to register and upload a new package by executing > > setup.py register > > > I was asked if I want to save the creditentials in a .pypirc file and I answered yes. > > Next I wanted to run > > setup.py upload > > and I got this error: > > > Traceback (most recent call last): > File "C:\Python\Projects\some_package\setup.py", line 15, in > classifiers=['Topic :: Security', 'Topic :: Internet :: WWW/HTTP'], > File "C:\Python35\lib\distutils\core.py", line 148, in setup > dist.run_commands() > File "C:\Python35\lib\distutils\dist.py", line 955, in run_commands > self.run_command(cmd) > File "C:\Python35\lib\distutils\dist.py", line 973, in run_command > cmd_obj.ensure_finalized() > File "C:\Python35\lib\distutils\cmd.py", line 107, in ensure_finalized > self.finalize_options() > File "C:\Python35\lib\distutils\command\upload.py", line 46, in > finalize_options > config = self._read_pypirc() > File "C:\Python35\lib\distutils\config.py", line 83, in _read_pypirc > current[key] = config.get(server, key) > File "C:\Python35\lib\configparser.py", line 798, in get > d) > File "C:\Python35\lib\configparser.py", line 396, in before_get > self._interpolate_some(parser, option, L, value, section, defaults, 1) > File "C:\Python35\lib\configparser.py", line 445, in _interpolate_some > "found: %r" % (rest,)) > configparser.InterpolationSyntaxError: '%' must be followed by '%' or > '(', found > : *****************' > > Instead of the many stars, of course there is the password. The problem > might be that the password contains a % character, and it was > incorrectly saved by distutils. > > Can somebody please confirm that this is a bug in distutils? > > Thanks, > > Laszlo > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -- Robert Collins Distinguished Technologist HP Converged Cloud From ncoghlan at gmail.com Fri Oct 23 17:12:02 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Oct 2015 23:12:02 +0200 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On 21 October 2015 at 19:32, David Cournapeau wrote: > > > On Wed, Oct 21, 2015 at 6:03 PM, Ronny Pfannschmidt > wrote: >> >> why does that have to be in setuptools ?! >> >> if we want a new light system to begin with, shouldn't it be far more >> sustainable to use just implementations of the new standards rather than >> leaving all of setuptools >> >> there is no viable way in setuptools to get rid of the legacy ina sane and >> fast manner, it would drag out over years > > agreed. I have never met a person who had to deal substantially with > distutils code and enjoyed the experience. > > The whole architecture is fundamentally flawed. I wrote this a long time > ago, but I still stand by most arguments: > https://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/ It's also worth reading Greg Ward's post on how distutils came to be: http://gerg.ca/blog/post/2013/distutils-history/ Something else not a lot of folks realise is that setuptools was originally built to handle development for the Open Source Application Foundation's Chandler project: https://en.wikipedia.org/wiki/Chandler_%28software%29 Neither setuptools nor distutils has particularly robust test suties, and nor do they have nicely structured public APIs, so refactoring either of them heavily is almost guaranteed to break something. OSAF's and Chandler's original needs are also far removed from the way we're using setuptools today (although the greatest of the differences relate to the differences between pip and easy_install moreso than anything on the build side). Despite those limitations, tens of thousands of software projects have been successfully published with both of them, so anyone trying to push for improvements needs to account for the popularity of pkg_resources for plugin management, the widespread use of setuptools based executable script wrappers, and the intertwined combination of distutils and setuptools that needs to be managed in order to successfully rebuild the whole of PyPI from source. Cheers, Nick. P.S. Anyone that decides to work on improving the packaging ecosystem also needs to account for the fact that they will hear relatively frequent complaints from *everybody*, as newcomers will complain "Why isn't it fixed yet?", veterans will complain "What have you broken today?", but almost nobody will offer to actually pay anyone to fix anything other than their own specific corner case, rather than investing in properly overhauling the entire system (which is as formidable a political challenge as it as a technical one). The core problem is that the status quo is merely clumsy and inelegant, rather than completely unusable. For all their flaws, both hobbyists and professional developers *can* use the current tools to distribute software (and they're still easier to deal with than writing directly for the native installer toolsets on each target platform), while in any organisation that uses Python heavily, setup scripts for new projects are likely to be either generated from templates or copied from those for old projects, rather than needing to be written from scratch each time. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Mon Oct 26 02:04:18 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Oct 2015 23:04:18 -0700 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format Message-ID: Hi all, Here's a second round of text towards making a build-system independent interface between pip and source trees/sdists. My idea this time is to take a divide-and-conquer approach: this text tries to summarize all the stuff that it seemed like we had mostly reached consensus on in the previous thread + call, with blank chunks marked "TBD" where there are specific points that still need To Be Determined. So my hope is that everyone will read what's here and agree that it's great as far as it goes, and then we can go through and fill in each missing piece one at a time. ------ PEP: ?? Title: A build-system independent format for source trees and source distributions Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: 1 Oct 2015, 25 Oct 2015 Discussions-To: Abstract ======== Distutils delenda est. Extended abstract ================= While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like autoconfiguration and usable build-time dependency declaration, (b) extending them is quirky, complicated, and fragile, (c) it's very difficult to use anything else, because they provide the standard interface for installing python packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). We propose to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit `_ or `bento `_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. In addition, we propose a wheel-inspired static metadata format for sdists, suitable for tools like PyPI and pip's resolver. Terminology and goals ===================== A *source tree* is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like ``pip install some-directory/``. A *source distribution* is a static snapshot representing a particular release of some source code, like ``lxml-3.4.4.zip``. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/..., and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like ``pip`` are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution ``foo.whl`` which declares a dependency on ``bar``, then we need to support the case where ``pip install bar`` or ``pip install foo`` automatically locates the sdist for ``bar``, downloads it, builds it, and installs the resulting package. Source distributions are also known as "sdists" for short. Source trees ============ We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code and documentation of ``distutils``, ``setuptools``, ``pip``, and other tools. A "version 1" (or greater) source tree is any directory which contains a file named ``pypackage.cfg``, which will -- in some manner whose details are TBD -- describe the package's build dependencies and how to invoke the build system. This mechanism: - Will allow for both static and dynamic specification of build dependencies - Will have some degree of isolation of different builds from each other, so that it will be possible for a single run of pip to install one package that build-depends on ``foo = 1.1`` and another package that build-depends on ``foo = 1.2``. - Will leave the actual installation of the package in the hands of the build/installation tool (i.e. individual package build systems will not need to know about things like --user versus --global or make decisions about when and how to modify .pth files) [TBD: the exact set of operations to be supported and their detailed semantics] [TBD: should builds be performed in a fully isolated environment, or should they get access to packages that are already installed in the target install environment? The former simplifies a number of things, but Robert was skeptical it would be possible.] [TBD: the form of the communication channel between an installation tool like ``pip`` and the build system, over which these operations are requested] [TBD: the syntactic details of the configuration file format itself. We can change the name too if we want, I just think it's useful to have a single name to refer to it for now, and this is the last and least interesting thing to figure out.] Source distributions ==================== [possibly this should get split off into a separate PEP, but I'll keep it together for now for ease of discussion] A "version 1" (or greater) source distribution is a file meeting the following criteria: - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant version number, and {EXT} is a compliant archive format. The set of compliant archive formats is: zip, [TBD] [QUESTION: should we continue to allow .tar.gz and friends? In practice by "allow" I mean something like "accept new-style sdists on PyPI in this format". I'm inclined not to -- zip is the most universally supported format around, it allows file-based random access (unlike tar-based things) which is useful for pulling out metadata without decompressing the whole thing, and standardizing on one format dodges distracting and pointless discussions about which format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to continue to support other archive formats when passed explicitly on the command line. Any objections?] Similar to wheels, the archive is Unicode, and the filenames inside the archive are encoded in UTF-8. - When unpacked, it MUST contain a single directory directory tree named ``{PACKAGE}-{VERSION}``. - This directory tree MUST be a valid version 1 (or greater) source tree as defined above. - It MUST additionally contain a directory named ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the following contents: - ``SDIST``: Mandatory. Same record-oriented format as a wheel's ``WHEEL`` file, but with different fields:: SDist-Version: 1.0 Generator: setuptools sdist 20.1 ``SDist-Version`` is the version number of this specification. Software that processes sdists should warn if ``SDist-Version`` is greater than the version it supports, and must fail if ``SDist-Version`` has a greater major version than the version it supports. ``Generator`` is the name and optionally the version of the software that produced the archive. - ``RECORD``: Mandatory. A list of all files contained in the sdist (except for the RECORD file itself and any signature files) together with their hashes, as specified in PEP 427. - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as specified in PEP 427. - ``METADATA``: Mandatory. Metadata version 1.1 or greater format metadata, with an additional rule that fields may contain the special sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value of this field cannot be determined until build time. If a "multiple use field" is present with the value ``__SDIST_DYNAMIC__``, then this field MUST occur exactly once, e.g.:: # Okay: Requires-Dist: lxml (> 3.3) Requires-Dist: requests # no Requires-Dist lines at all is okay # (meaning: this package's requirements are the empty set) # Okay, requirements will be determined at build time: Requires-Dist: __SDIST_DYNAMIC__ # NOT okay: Requires-Dist: lxml (> 3.3) Requires-Dist: __SDIST_DYNAMIC__ (The use of a special token allows us to distinguish between multiple use fields whose value is statically the empty list versus one whose value is dynamic; it also allows us to distinguish between optional fields which are statically not present versus ones whose value is dynamic.) When this sdist is built, the resulting wheel MUST have metadata which is identical to the metadata present in this file, except that any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have arbitrary values in the wheel. A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for the package name or version (i.e., these must be given statically), and these MUST match the {PACKAGE} and {VERSION} of the sdist as described above. [TBD: do we want to forbid the use of dynamic metadata for any other fields? I assume PyPI will enforce some stricter rules at least, but I don't know if we want to make that part of the spec, or just part of PyPI's administrative rules.] This is intentionally a close analogue of a wheel's ``.dist-info`` directory; intention is that as future metadata standards are defined, the specifications for the ``.sdist-info`` and ``.dist-info`` directories will evolve in synchrony. Evolutionary notes ================== A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static pypackage.cfg that could be dropped into any "version 0" VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it's important to keep track of how close we are... hence this section. A rough plan would be: Create a build system package (``setuptools_pypackage`` or whatever) that knows how to speak whatever hook language we come up with, and convert them into setuptools calls. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the ``setup_requires=`` argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we'll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won't be able to automatically upgrade packages to the new format: 1) There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via ``setup_requires=`` or via static declaration in ``pypackage.cfg``). 2) There currently exist packages which do not declare consistent metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different ``install_requires=``). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to either stop doing that, or else add ``__SDIST_DYNAMIC__`` annotations at appropriate places. We'll also presumably need some API for packages to describe which parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for the packages that need it (a new argument to ``setup()`` or some setting in ``setup.cfg`` or something). -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Mon Oct 26 02:04:43 2015 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 25 Oct 2015 23:04:43 -0700 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: Wow, uh, that's some timing we have. I'll, uh, take a look :-) On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote: > Since Nathaniel seems busy, I've taken the liberty of drafting a > narrow PEP based on the conversations that arose from the prior > discussion. > > It (naturally) has my unique flavor, but builds on the work Nathaniel > had put together, so I've put his name as a co-author even though he > hasn't seen a word of it until now :) - all errors and mistakes are > therefore mine... > > Current draft text in rendered form at: > https://gist.github.com/rbtcollins/666c12aec869237f7cf7 > > I've run it past Donald and he has a number of concerns - I think > we'll need to discuss them here, and possibly in another hangout, to > get a path forward. > > Cheers, > Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -- Nathaniel J. Smith -- http://vorpus.org From robertc at robertcollins.net Mon Oct 26 02:20:34 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 26 Oct 2015 19:20:34 +1300 Subject: [Distutils] Is this a bug? In-Reply-To: <562DC3C2.40101@shopzeus.com> References: <562876B3.1010607@shopzeus.com> <562DC3C2.40101@shopzeus.com> Message-ID: Ok, so I'm not sure where the bug is, but I'd say in the save-password code that should have escaped it. Not sure if that is in the disutils core code or setuptools. -Rob On 26 October 2015 at 19:10, Nagy L?szl? Zsolt wrote: > > > >> I can't confirm or deny that with the data given:) >> >> Have a look in your .pypirc file - don't share it, it has your >> password in it - but see if it looks sane. If you can't tell, mask out >> your username and password and share the content. If you do that, >> please be sure to note exactly how the % in your password was encoded >> in the file. > > My .pypirc is very simple: > > [distutils] > index-servers = > pypi > > [pypi] > repository = https://pypi.python.org/pypi > username:nagylzs > password:****************** > > > The password of course is different, and it has a *single* percent sign > in it. I had no .pypirc file at all. Here are the steps I have taken: > > 1. register online at pypi.python.org > 2. create setup file for my package > 3. run "setup.py register" (python version 3.5) > > The registration was successful, and setup.py asked me if I want to save > the password into the pypirc file. I said "YES" and the .pypirc file was > created. Then I wanted to do "setup.py sdist upload" and I got the > aforementioned traceback. Then I have deleted the "password:" line from > the pypirc file and the upload succeeded (but of course, I had to retype > the password.) > > Finally - as a workaround - I have changed my password so it does not > contain % sign. -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 26 02:22:44 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 26 Oct 2015 19:22:44 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 26 October 2015 at 19:04, Nathaniel Smith wrote: > Wow, uh, that's some timing we have. I'll, uh, take a look :-) Hah! Jinx. Thanks. I'll read through yours too :) -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 26 03:36:25 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Oct 2015 00:36:25 -0700 Subject: [Distutils] wacky idea about reifying extras Message-ID: Hi all, I had a wacky idea and I can't tell if it's brilliant, or ridiculous, or both. Which makes sense given that I had a temperature of 39.3 when I thought of it, but even after getting better and turning it over in my mind for a while I still can't tell, so, I figured I'd put it out there and see what y'all thought :-). The initial motivation was to try and clean up an apparent infelicity in how extras work. After poking at it for a bit, I realized that it might accidentally solve a bunch of other problems, including removing the need for dynamic metadata in sdists (!). If it can be made to work at all... Motivating problem ------------------ "Extras" are pretty handy: e.g., if you just want the basic ipython REPL, you can do ``pip install ipython``, but if you want the fancy HTML notebook interface that's implemented in the same source code base but has lots of extra heavy dependencies, then you can do ``pip install ipython[notebook]`` instead. Currently, extras are implemented via dedicated metadata fields inside each package -- when you ``pip install ipython[notebook]``, it downloads ipython-XX.whl, and when it reads the metadata about this package's dependencies and entry points, the ``notebook`` tag flips on some extra dependencies and extra entry points. This worked pretty well for ``easy_install``'s purposes, since ``easy_install`` mostly cared about installing things; there was no concern for upgrading. But people do want to upgrade, and as ``pip`` gets better at this, extras are going to start causing problems. Hopefully soon, pip will get a proper resolver, and then an ``upgrade-all`` command like other package managers have. Once this happens, it's entirely possible -- indeed, a certainty in the long run -- that if you do:: $ pip install ipython[notebook] # wait a week $ pip upgrade-all you will no longer have the notebook installed, because the new version of ipython added a new dependency to the "notebook" extra, and since there's no record that you ever installed that extra, this new dependency won't be installed when you upgrade. I'm not sure what happens to any entry points or scripts that were part of ipython[notebook] but not ipython -- I'm guessing they'd still be present, but broken? If you want to upgrade while keeping the notebook around, then ``upgrade-all`` is useless to you; you have to manually keep a list of all packages-with-extras you have installed and explicitly pass them to the upgrade command every time. Which is terrible UX. Supporting extras in this manner also ramifies complexity through the system: e.g., the specification of entry points becomes more complex because you need a way to make them conditional on particular extras, PEP 426 proposes special mechanisms to allow package A to declare a dependency on extra B of package C, etc. And extras also have minor but annoying limitations, e.g. there's no mechanism provided to store a proper description of what an extra provides and why you might want it. Solution (?): promoting extras to first-class packages ------------------------------------------------------ There's an obvious possible solution, inspired by how other systems (e.g. Debian) handle this situation: promote ``ipython[notebook]`` to a full-fledged package, that happens to contain no files of its own, but which gets its own dependencies and other metadata. What would this package look like? Something like:: Name: ipython[notebook] Version: 4.0.0 Requires-Dist: ipython (= 4.0.0) Requires-Dist: extra_dependency_1 Requires-Dist: extra_dependency_2 Requires-Dist: ... The ``notebook`` extra extends IPython with an HTML interface to... Installing it needs to automatically trigger the installation of ipython, so it should depend on ipython. It needs to be upgraded in sync with ``ipython``, so this dependency should be an exact version dependency -- that way, upgrading ``ipython`` will (once we have a real resolver!) force an upgrade of ``ipython[notebook]`` and vice-versa. Then of course we also need to include the extra's unique dependencies, and whatever else we want (e.g. a description). What would need to happen to get there from here? AFAICT a relatively small set of changes would actually suffice: **PyPI:** starts allowing the upload of wheels named like ``BASE[EXTRA]-VERSION-COMPAT.whl``. They get special handling, though: who-ever owns ``BASE`` gets to do whatever they like with names like ``BASE[EXTRA]``, that's an established rule, so wheels following this naming scheme would be treated like other artifacts associated with the (BASE, VERSION) release. In particular, the uploader would need to have write permission to the ``BASE`` name, and it would remain impossible to register top-level distribution names containing square brackets. **setuptools:** Continues to provide "extra" metadata inside the METADATA file just as it does now (for backwards compatibility with old versions of pip that encounter new packages). In addition, though, the egg-info command would starts generating .egg-info directories for each defined extra (according to the schema described above), the bdist_wheel command would start generating a wheel file for each defined extra, etc. **pip:** Uses a new and different mechanism for looking up packages with extras: - when asked to fulfill a requirement for ``BASE[EXTRA1,EXTRA2,...] (> X)``, it should expand this to ``BASE[EXTRA1] (> X), BASE[EXTRA2] (> X), ...``, and then attempt to find wheels with those actual names - backcompat case: if we fail to find a BASE[EXTRA] wheel, then fall back to fetching a wheel named BASE and attempt to use the "extra" metadata inside it to generate BASE[EXTRA], and install this (this is morally similar to the fallback logic where if it can't find foo.whl it tries to generate it from the foo.zip sdist) - Optionally, PyPI itself could auto-generate these wheels for legacy versions (since they can be generated automatically from static wheel metadata), thus guaranteeing that this path would never be needed, and then pip could disable this fallback path... but I guess it would still need it to handle non-PyPI indexes. - if this fails, then it falls back to fetching an sdist named BASE (*not* BASE[EXTRA]) and attempting to build it (while making sure to inject a version of setuptools that's recent enough to include the above changes). **PEP 426:** can delete all the special case stuff for extras, because they are no longer special cases, and there's no backcompat needed for a format that is not yet in use. **twine and other workflows:** ``twine upload dist/*`` continues to do the right thing (now including the new extra wheels). Other workflows might need the obvious tweaking to include the new wheel files. So this seems surprisingly achievable (except for the obvious glaring problem that I missed but someone is about to point out?), would improve correctness in the face of upgrades, and simplifies our conceptual models, and provides a more solid basis for future improvements (e.g. if in the future we add better tracking of which packages were manually installed, then this will automatically apply to extras as well, since they are just packages). But wait there's more --------------------- **Non-trivial plugins:** Once we've done this, suddenly extras become much more powerful. Right now it's a hard constraint that extras can only add new dependencies and entry points, not contain any code. But this is rather artificial -- from the user's point of view, 'foo[bar]' just means 'I want a version of foo that has the bar feature', they don't care whether this requires installing some extra foo-to-bar shim code. With extras as first-class packages, it becomes possible to use this naming scheme for things like plugins or compiled extensions that add actual code. **Build variants:** People keep asking for us to provide numpy builds against Intel's super-fancy closed-source (but freely redistributable) math library, MKL. It will never be the case that 'pip install numpy' will automatically give you the MKL-ified version, because see above re: "closed source". But we could provide a numpy[mkl]-{VERSION}-{COMPAT}.whl with metadata like:: Name: numpy[mkl] Conflicts: numpy Provides: numpy which acts as a drop-in replacement for the regular numpy for those who explicitly request it via ``pip install numpy[mkl]``. This involves two new concepts on top of the ones above: Conflicts: is missing from the current metadata standards but (I think?) trivial to implement in any real resolver. It means "I can't be installed at the same time as something else which matches this requirement". In a sense, it's actually an even more primitive concept than a versioned requirement -- Requires: foo (> 2.0) is equivalent to Requires: foo + Conflicts: foo (<= 2.0), but there's no way to expand an arbitrary Conflicts in terms of Requires. (A minor but important wrinkle: the word "else" is important there; you need a special case saying that a package never conflicts with itself. But I think that's the only tricky bit.) Provides: is trickier -- there's some vestigial support in the current standards and even in pip, but AFAICT it hasn't really been worked out properly. The semantics are obvious enough (Provides: numpy means that this package counts as being numpy; there's some subtleties around what version of numpy it should count as but I think that can be worked out), but it opens a can of worms, because you don't want to allow things like:: Name: numpy Provides: django But once you have the concept of a namespace for multiple distributions from the same project, then you can limit Provides: so that it's only legal if the provider distribution and the provided distribution have the same BASE. This solves the social problem (PyPI knows that numpy[mkl] and numpy are 'owned' by the same people, so this Provides: is OK), and provides algorithmic benefits (if you're trying to find some package that provides foo[extra] out of a flat directory of random distributions, then you only have to examine wheels and sdists that have BASE=foo). The other advantage to having the package be ``numpy[mkl]`` instead of ``numpy-mkl`` is that it correctly encodes that the sdist is ``numpy.zip``, not ``numpy-mkl.zip`` -- the rules we derived to match how extras work now are actually exactly what we want here too. **ABI tracking:** This also solves another use case entirely: the numpy ABI tracking problem (which is probably the single #1 problem the numerical crowd has with current packaging, because it actually prevents us making basic improvements to our code -- the reason I've been making a fuss about other things first is that until now I couldn't figure out any tractable way to solve this problem, but now I have hope). Once you have Provides: and a namespace to use with it, then you can immediately start using "pure virtual" packages to keep track of which ABIs are provided by a single distribution, and determine that these two packages are consistent: Name: numpy Version: 1.9.2 Provides: numpy[abi-2] Provides: numpy[abi-3] Name: scipy Depends: numpy Depends: numpy[abi-2] (AFAICT this would actually make pip *better* than conda as far as numpy's needs are concerned.) The build variants and virtual packages bits also work neatly together. If SciPy wants to provide builds against multiple versions of numpy during the transition period between two ABIs, then these are build variants exactly like numpy[mkl]. For their 0.17.0 release they can upload:: scipy-0.17.0.zip scipy[numpy-abi-2]-0.17.0.whl scipy[numpy-abi-3]-0.17.0.whl (And again, it would be ridiculous to have to register scipy-numpy-abi-2, scipy-numpy-abi-3, etc. on PyPI, and upload separate sdists for each of them. Note that there's nothing magical about the names -- those are just arbitrary tags chosen by the project; what pip would care about is that one of the wheels' metadata says Requires-Dist: numpy[abi-2] and the other says Requires-Dist: numpy[abi-3].) So far as I can tell, these build variant cases therefore cover *all of the situations that were discussed in the previous thread* as reasons why we can't necessarily provide static metadata for an sdist. The numpy sdist can't statically declare a single set of install dependencies for the resulting wheel... but it could give you a menu, and say that it knows how to build numpy.whl, numpy[mkl].whl, or numpy[external-blas].whl, and tell you what the dependencies will be in each case. (And maybe it's also possible to make numpy[custom] by manually editing some configuration file or whatever, but pip would never be called upon to do this so it doesn't need the static metadata.) So I think this would be sufficient to let us start providing full static metadata inside sdists? (Concretely, I imagine that the way this would work is that when we define the new sdist hooks, one of the arguments that pip would pass in when running the build system would be a list of the extras that it's hoping to see, e.g. "the user asked for numpy[mkl], please configure yourself accordingly". For legacy setuptools builds that just use traditional extras, this could safely be ignored.) TL;DR ----- If we: - implement a real resolver, and - add a notion of a per-project namespace of distribution names, that are collected under the same PyPI registration and come from the same sdist, and - add Conflicts:, and Provides:, then we can elegantly solve a collection of important and difficult problems, and we can retroactively pun the old extras system onto the new system in a way that preserves 100% compatibility with all existing packages. I think? What do you think? -n -- Nathaniel J. Smith -- http://vorpus.org From guettliml at thomas-guettler.de Mon Oct 26 03:53:03 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Mon, 26 Oct 2015 08:53:03 +0100 Subject: [Distutils] Undelying design is fundamentally difficult to extend was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <56290BA9.7060602@thomas-guettler.de> Message-ID: <562DDBDF.6000302@thomas-guettler.de> Am 22.10.2015 um 18:31 schrieb Chris Barker: > On Thu, Oct 22, 2015 at 9:15 AM, Thomas G?ttler > wrote: > > If setuptools in inside Python, then different versions of Python each have their matching version included. > > If a package has the need for newer version of setuptool, it provides the hint somewhere: > required_setuptools_version>=N.M > > Then the matching version from pypi gets downloaded first. > > > Isn't this what ensurepip already does??? > Sure, it's called ensure *pip* but pip requires setuptools, so you get that out of the box. ok, nice. I was thinking about a new key-value pair in setup.py or setup.cfg. ensurepip is a package. Is there already a way to pull the matching pip or setuptools version via a setup.py/setup.cfg? Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From guettliml at thomas-guettler.de Mon Oct 26 03:56:55 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Mon, 26 Oct 2015 08:56:55 +0100 Subject: [Distutils] Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627C5A3.6090004@ronnypfannschmidt.de> <562904C0.10000@thomas-guettler.de> Message-ID: <562DDCC7.9080901@thomas-guettler.de> Am 22.10.2015 um 18:34 schrieb Chris Barker: > On Thu, Oct 22, 2015 at 8:46 AM, Thomas G?ttler > wrote: > > I prefer the code of unittests to the not testable sentences in a standard. > > > sure -- but unittests are essentially an implementation of a standard -- you need a standard to write the unit tests for. You need a standard to write the unit test for .... If I look at the all the unittests in python itself. What is your guess: How many (percentage) exist to ensure the compatibility with a standard, and how many exist to ensure things work and bugs don't appear again? Unittest can be written to ensure a road map or "rough consensus and working code". Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From robertc at robertcollins.net Mon Oct 26 02:01:17 2015 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 26 Oct 2015 19:01:17 +1300 Subject: [Distutils] build system abstraction PEP Message-ID: Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion. It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine... Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7 I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward. Cheers, Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From donald at stufft.io Mon Oct 26 07:41:27 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 26 Oct 2015 07:41:27 -0400 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On October 26, 2015 at 3:36:47 AM, Nathaniel Smith (njs at pobox.com) wrote: > > TL;DR > ----- > > If we: > > - implement a real resolver, and > - add a notion of a per-project namespace of distribution names, > that > are collected under the same PyPI registration and come from > the same > sdist, and > - add Conflicts:, and Provides:, > > then we can elegantly solve a collection of important and difficult > problems, and we can retroactively pun the old extras system > onto the > new system in a way that preserves 100% compatibility with all > existing packages. > > I think? > > What do you think? My initial reaction when I started reading your idea was that I didn't see a point in having something like foo[bar] be a "real" package when you could just as easily have foo-bar. However, as I continued to read through the idea it started to grow on me. I think I need to let it percolate in my brain a little bit, but there may be a non-crazy (or at least, crazy in a good way) idea here that could push things forward in a nice way. Some random thoughts: * Reusing the extra syntax is nice because it doesn't require end users to ? learn any new concepts, however we shouldn't take a new syntax off the table ? either if it makes the feature easier to implement with regards to backwards ? compatability. Something like numpy{mkl,some-other-thing} could work just as ? well too. We'll need to make sure that whatever symbols we choose can be ? represented on all the major FS we care about and that they are ideally non ? ugly in an URL too. Of course, the filename and user interface symbols don't ? *need* to match. It could just as easily example numpy[mkl] out to numpy#mkl ? or whatever which should make it easier to come up with a nice scheme. * Provides is a bit of an odd duck, I think in my head I've mostly come to ? terms with allowing unrestricted Provides when you've already installed the ? package doing the Providing but completely ignoring the field when pulling ? data from a repository. Our threat model assumes that once you've selected to ? install something then it's generally safe to trust (though we still do try ? to limit that). The problem with Provides mostly comes into play when you ? will respect the Provides: field for any random package on PyPI (or any other ? repo). * The upgrade mess around extras as they stand today could also be solved just ? by recording what extras (if any) were selected to be installed so that we ? keep a consistent view of the world. Your proposal is essentially doing that, ? just by (ab)using the fact that by installing a package we essentially get ? that aspect of it for "free". * Would this help at all with differentiating between SSE2 and SSE3 builds and ? things like that? Or does that need something more automatic to be really ? usable? * PEP 426 (I think it was?) has some extra syntax for extras which could ? probably be really nice here, things like numpy[*] to get *all* of the extras ? (though if they are real packages, what even is "all"?). It also included ? (though this might have been only in my head) default to installed packages ? which meant you could do something like split numpy into numpy[abi2] and ? numpy[abi3] packages and have the different ABIs actually contained within ? those other packages. Then you could have your top level package default to ? installing abi3 and abi2 so that ``pip install numpy`` is equivilant to ? ``pip install numpy[abi2,abi3]``. The real power there, is that people can ? trim down their install a bit by then doing ``pip install numpy[-abi2]`` if ? they don't want to have that on-by-default feature. I'm going to roll through around in my head more, and hopefully more people with other interesting problems to solve can chime in and say whether they think this would solve their problems or not. My less-than-immediate reaction now that I'm through the entire thing is that it seems like it could be good and I'm struggling to think of a major downside. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From waynejwerner at gmail.com Mon Oct 26 08:53:21 2015 From: waynejwerner at gmail.com (Wayne Werner) Date: Mon, 26 Oct 2015 07:53:21 -0500 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> <5629052D.2070201@thomas-guettler.de> Message-ID: On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker wrote: > On Thu, Oct 22, 2015 at 8:47 AM, Thomas G?ttler < > guettliml at thomas-guettler.de> wrote: > > >> > I have a dream: For packaging and building package provides only >> **data**. Data is condition-less: No single "if", "def" or method call. >> Just data: json or yaml ... >> > >> > Even for complex packages. >> > > It's a nice dream -- but I think there is more or less a consensus on this > list that you really can't do everything completely declaratively. I know I > would find it very frustrating to have to hard-code everything up front. > I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively? -W -------------- next part -------------- An HTML attachment was scrubbed... URL: From gandalf at shopzeus.com Mon Oct 26 02:10:10 2015 From: gandalf at shopzeus.com (=?UTF-8?Q?Nagy_L=c3=a1szl=c3=b3_Zsolt?=) Date: Mon, 26 Oct 2015 07:10:10 +0100 Subject: [Distutils] Is this a bug? In-Reply-To: References: <562876B3.1010607@shopzeus.com> Message-ID: <562DC3C2.40101@shopzeus.com> > I can't confirm or deny that with the data given:) > > Have a look in your .pypirc file - don't share it, it has your > password in it - but see if it looks sane. If you can't tell, mask out > your username and password and share the content. If you do that, > please be sure to note exactly how the % in your password was encoded > in the file. My .pypirc is very simple: [distutils] index-servers = pypi [pypi] repository = https://pypi.python.org/pypi username:nagylzs password:****************** The password of course is different, and it has a *single* percent sign in it. I had no .pypirc file at all. Here are the steps I have taken: 1. register online at pypi.python.org 2. create setup file for my package 3. run "setup.py register" (python version 3.5) The registration was successful, and setup.py asked me if I want to save the password into the pypirc file. I said "YES" and the .pypirc file was created. Then I wanted to do "setup.py sdist upload" and I got the aforementioned traceback. Then I have deleted the "password:" line from the pypirc file and the upload succeeded (but of course, I had to retype the password.) Finally - as a workaround - I have changed my password so it does not contain % sign. From njs at pobox.com Mon Oct 26 15:07:21 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Oct 2015 12:07:21 -0700 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> <5629052D.2070201@thomas-guettler.de> Message-ID: On Oct 26, 2015 6:59 AM, "Wayne Werner" wrote: > > On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker wrote: >> >> On Thu, Oct 22, 2015 at 8:47 AM, Thomas G?ttler < guettliml at thomas-guettler.de> wrote: >> >>> >>> > I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ... >>> > >>> > Even for complex packages. >> >> >> It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front. > > > I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively? Test the system's implementation of the math.h 'tanh' function to decide whether it correctly implements C99 annex F style handling of edge cases and thus can be used safely, or whether we need to fall back to our internal version of tanh instead, and define some appropriate C preprocessor macros depending on the result. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Mon Oct 26 16:36:57 2015 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Mon, 26 Oct 2015 21:36:57 +0100 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> <5629052D.2070201@thomas-guettler.de> Message-ID: Test if the provided system C/C++ compiler supports OpenMP and do not use the openmp compiler flag to fallback to a sequential build if not. -- Olivier Grisel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Mon Oct 26 17:32:28 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 26 Oct 2015 22:32:28 +0100 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On Mon, Oct 26, 2015 at 7:01 AM, Robert Collins wrote: > Since Nathaniel seems busy, I've taken the liberty of drafting a > narrow PEP based on the conversations that arose from the prior > discussion. > > It (naturally) has my unique flavor, but builds on the work Nathaniel > had put together, so I've put his name as a co-author even though he > hasn't seen a word of it until now :) - all errors and mistakes are > therefore mine... > > Current draft text in rendered form at: > https://gist.github.com/rbtcollins/666c12aec869237f7cf7 I have the feeling that I like where this PEP is going, but it's quite difficult to read. It would be very helpful to add one or two examples. Suggestions: (1) Super simple example: for a pure Python package with one dependency and a build tool which has no dependencies itself. (2) Complex example: to build a Scipy wheel on Windows with MinGW the command is ``python setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wheel``. I tried to do (1) with flit as the build tool, the pypa.yaml should include: bootstrap-requires: - requests - docutils - requirement:python_version>="3" So you have to encode flit's dependencies in your pypa.yaml, which will break as soon as flit grows a new dependency. Or did I misunderstand? Maybe (2) simply is not possible / out of scope, but I have the feeling that there'll be a need for users to be able pass stuff (via pip or some other mechanism) to the build tool. If it is out of scope, I'd be interested to see what you think are use-cases with complex requirements that are enabled by this PEP. Cheers, Ralf > > I've run it past Donald and he has a number of concerns - I think > we'll need to discuss them here, and possibly in another hangout, to > get a path forward. > > Cheers, > Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Mon Oct 26 19:23:16 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 27 Oct 2015 12:23:16 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 27 October 2015 at 10:32, Ralf Gommers wrote: > > > On Mon, Oct 26, 2015 at 7:01 AM, Robert Collins > wrote: >> >> Since Nathaniel seems busy, I've taken the liberty of drafting a >> narrow PEP based on the conversations that arose from the prior >> discussion. >> >> It (naturally) has my unique flavor, but builds on the work Nathaniel >> had put together, so I've put his name as a co-author even though he >> hasn't seen a word of it until now :) - all errors and mistakes are >> therefore mine... >> >> Current draft text in rendered form at: >> https://gist.github.com/rbtcollins/666c12aec869237f7cf7 > > > I have the feeling that I like where this PEP is going, but it's quite > difficult to read. It would be very helpful to add one or two examples. Ok, need to make it easier to read :). > Suggestions: > > (1) Super simple example: for a pure Python package with one dependency and > a build tool which has no dependencies itself. pypy.yaml === bootstrap-requires: - flit build-tool: flit --dump-build-description === flit --dump-build-description would output something like: === build-requires: flit --dump-build-requires dist-info: flit --dump-dist-info provided-by: flit wheel: flit wheel -d $OUTPUT_DIR === The --dump- switches in this example would be needed to be added to flit (or someone else could write an adapter). Because flit doesn't have setup-requires in flit.ini today, --dump-build-requires would just output a constant string: {"schema_version": "2.0", "build_requires": []} --dump-dist-info interrogate flit.ini and marshal the metadata into a PEP 426 JSON document and output that on stdout. flit wheel would need a -d parameter that tells it where to output the wheel(pip needs this). > (2) Complex example: to build a Scipy wheel on Windows with MinGW the > command is ``python setup.py config --compiler=mingw32 build > --compiler=mingw32 bdist_wheel``. So in this case the build tool needs to know about the compiler stuff itself- pip doesn't know. We have a way in pip to tunnel stuff down to setuptools today; thats incompatible with dynamically building wheels on the fly for 'pip install' - so I'm not sure it needs to be reflected here. We'll need some more input on that I think. ... > mechanism) to the build tool. If it is out of scope, I'd be interested to > see what you think are use-cases with complex requirements that are enabled > by this PEP. The PEP is aimed at enabling additional build-tools to be on parity with setuptools in *pip install* invocations that go through the wheel-autobuild-path in pip. The complex examples of passing arbitrary options to setup.py currently bypasses wheel building in pip, and so can't be tackled at all :(. But we can work on including that with some thought. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Mon Oct 26 19:41:21 2015 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 26 Oct 2015 16:41:21 -0700 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Mon, Oct 26, 2015 at 4:41 AM, Donald Stufft wrote: > On October 26, 2015 at 3:36:47 AM, Nathaniel Smith (njs at pobox.com) wrote: >> > TL;DR >> ----- >> >> If we: >> >> - implement a real resolver, and >> - add a notion of a per-project namespace of distribution names, >> that >> are collected under the same PyPI registration and come from >> the same >> sdist, and >> - add Conflicts:, and Provides:, >> >> then we can elegantly solve a collection of important and difficult >> problems, and we can retroactively pun the old extras system >> onto the >> new system in a way that preserves 100% compatibility with all >> existing packages. >> >> I think? >> >> What do you think? > > My initial reaction when I started reading your idea was that I didn't see a > point in having something like foo[bar] be a "real" package when you could just > as easily have foo-bar. However, as I continued to read through the idea it > started to grow on me. I think I need to let it percolate in my brain a little > bit, but there may be a non-crazy (or at least, crazy in a good way) idea here > that could push things forward in a nice way. Oh good, at least I'm not the only one :-). I'd particularly like to hear Robert's thoughts when he has time, since the details depend strongly on some assumptions about how a real resolver would work. > Some random thoughts: > > * Reusing the extra syntax is nice because it doesn't require end users to > learn any new concepts, however we shouldn't take a new syntax off the table > either if it makes the feature easier to implement with regards to backwards > compatability. Something like numpy{mkl,some-other-thing} could work just as > well too. We'll need to make sure that whatever symbols we choose can be > represented on all the major FS we care about and that they are ideally non > ugly in an URL too. Of course, the filename and user interface symbols don't > *need* to match. It could just as easily example numpy[mkl] out to numpy#mkl > or whatever which should make it easier to come up with a nice scheme. Right -- obviously it would be *nice* to keep the number of concepts down, and to avoid skew between filenames and user interface (because users do see filenames), but if these turn out to be impossible then there are still options that would let us save the per-project-package-namespace idea. > * Provides is a bit of an odd duck, I think in my head I've mostly come to > terms with allowing unrestricted Provides when you've already installed the > package doing the Providing but completely ignoring the field when pulling > data from a repository. Our threat model assumes that once you've selected to > install something then it's generally safe to trust (though we still do try > to limit that). The problem with Provides mostly comes into play when you > will respect the Provides: field for any random package on PyPI (or any other > repo). Yeah, I'm actually not too worried about malicious use either in practice, for the reason you say. But even so I can think of two good reasons we might want to be careful about stating exactly when "Provides:" can be trusted: 1) if you have neither scipy nor numpy installed, and you do 'pip install scipy', and scipy depends on the pure virtual package 'numpy[abi-2]' which is only available as a Provides: on the concrete package 'numpy', then in this case the resolver has to take Provides: into account when pulling data from the repo -- if it doesn't, then it'll ignore the Provides: on 'numpy' and say that scipy's dependencies can't be satisfied. So for this use case to work, we actually do need to be able to sometimes trust Provides: fields. 2) the basic idea of a resolver is that it considers a whole bunch of possible configurations for your environment, and picks the configuration that seems best. But if we pay attention to different metadata when installing as compared to after installation, then this skew makes it possible for the algorithm to pick a configuration that looks good a priori but is broken after installation. E.g. for a simple case: Name: a Conflicts: some-virtual-package Name: b Provides: some-virtual-package 'pip install a b' will work, because the resolver ignores the Provides: and treats the packages as non-conflicting -- but then once installed we have a broken system. This is obviously an artificial example, but creating the possibility of such messes just seems like the kind of headache we don't need. So I think whatever we do with Provides:, we should do the same thing both before and after installation. A simple safe rule is to say that Provides: is always legal iff a package's Name: and Provides: have a matching BASE, and always illegal otherwise, making the package just invalid, like if the METADATA were written in Shift-JIS or something. This rule is trivial to statically check/enforce, and could always be relaxed more later. > * The upgrade mess around extras as they stand today could also be solved just > by recording what extras (if any) were selected to be installed so that we > keep a consistent view of the world. Your proposal is essentially doing that, > just by (ab)using the fact that by installing a package we essentially get > that aspect of it for "free". Right -- you certainly could implement a database of installed extras to go alongside the database of installed packages, but it seems like it just makes things more complicated with minimal benefit. E.g., you have to add special case code to the resolver to check both databases, and then you have to add more special case code to 'pip freeze' to make sure *it* checks both databases... this kind of stuff adds up. > * Would this help at all with differentiating between SSE2 and SSE3 builds and > things like that? Or does that need something more automatic to be really > usable? I'm not convinced that SSE2 versus SSE3 is really worth trying to handle automatically, just because we have more urgent issues and everyone else in the world seems to get by okay without special support for this in their package system (even if it's not always optimal). But if we did want to do this then my intuition is that it'd be more elegant to do it via the wheel platform/architecture field, since this actually is a difference in architectures? So you could have one wheel for the "win32" platform and another wheel for the "win32sse3" platform, and the code in the installer that figures out which wheels are compatible would know that both of these are compatible with the machine it was running on (or not), and that win32sse3 is preferable to plain win32. > * PEP 426 (I think it was?) has some extra syntax for extras which could > probably be really nice here, things like numpy[*] to get *all* of the extras > (though if they are real packages, what even is "all"?). It also included > (though this might have been only in my head) default to installed packages > which meant you could do something like split numpy into numpy[abi2] and > numpy[abi3] packages and have the different ABIs actually contained within > those other packages. Then you could have your top level package default to > installing abi3 and abi2 so that ``pip install numpy`` is equivilant to > ``pip install numpy[abi2,abi3]``. The real power there, is that people can > trim down their install a bit by then doing ``pip install numpy[-abi2]`` if > they don't want to have that on-by-default feature. Hmm, right, I'm not thinking of a way to *quite* duplicate this. One option would be to have a numpy[all] package that just depends on all the other extras packages -- for the traditional 'extra' cases this could be autogenerated by setuptools at build time and then be a regular package after that, and for next-generation build systems that had first-class support for these [] packages, it would be up to the build system / project whether to generate such an [all] package and what to include in it if they did. But that doesn't give you the special all-except-for-one behavior. The other option that jumps to mind is what Debian calls "recommends", which act like a soft-dependency: in debian, if numpy recommends: numpy[abi-2] and numpy[abi-3], then 'apt-get install numpy' would give you all three of them by default, just like if numpy required them -- but for recommends: you can also say something like 'apt-get install numpy -numpy[abi-3]' if you want numpy without the abi-3 package, or 'apt-get install --no-recommends numpy' if you want a fully minimal install, and this is okay because these are only *recommendations*, not an actual requirements. I don't see any fundamental reasons why we couldn't add something like this to pip, though it's probably not that urgent. My guess is that these two solutions together would pretty much cover the relevant use cases? -n -- Nathaniel J. Smith -- http://vorpus.org From robertc at robertcollins.net Mon Oct 26 21:06:47 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 27 Oct 2015 14:06:47 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 27 October 2015 at 13:07, Ionel Cristian M?rie? wrote: > > On Tue, Oct 27, 2015 at 1:23 AM, Robert Collins > wrote: >> >> wheel: flit wheel -d $ >> OUTPUT_DIR > > > What's expected to happen if there are multiple files produced in > $OUTPUT_DIR? Is it an error? https://github.com/pypa/pip/blob/develop/pip/wheel.py#L665 is the calling function in pip today. It picks a single output, so if a wheel build generates multiple outputs, the results will be nondeterministic. Don't do that :). > Also, doesn't this pep overlap with PEP 426? Shouldn't these ideas be > incorporated there? It seems that only the command metadata is missing from > PEP 426. It does. I believe Nick wants to see PEP-426 broken up into small bits. The meta build system described in PEP 426 is aspirational and not reflective of what pip *actually does today*. The intent I have with this PEP is to have something that preserves the current behaviour while adding the abstraction - but there is no PEP that describes the current markers-inclusive metadata on disk (345 misses setup-requires and markers). So we may need a PEP to just bless the metadata from 426, without any of the aspirational stuff. > Another thing, I've noticed there's a "develop" command in this draft. I > think this is confusing, I'd expect it to be a "install" command (and that's > what it is now in setuptools). It should be renamed to something more clear, > like "inplace-build" and leave the installing to pip. The current thing pip calls - and all existing released pips is 'setup.py develop'. I agree that doing an in place build and having pip do the install into an environment would make sense, but it makes this more aspirational. And the value here is that this is doable *now* without e.g. an additional PEP to describe how 'develop' mode in pip and related tools should behave and interoperate. We can rev this in schema version 2. There's no prose in the PEP about how that should work, so I'll add that now. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Mon Oct 26 21:39:09 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 27 Oct 2015 14:39:09 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 27 October 2015 at 14:06, Robert Collins wrote: > We can rev this in schema version 2. There's no prose in the PEP about > how that should work, so I'll add that now. Done: https://gist.github.com/rbtcollins/666c12aec869237f7cf7#upgrades -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From contact at ionelmc.ro Mon Oct 26 20:07:07 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Tue, 27 Oct 2015 02:07:07 +0200 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 1:23 AM, Robert Collins wrote: > wheel: flit wheel -d $ > ?? > OUTPUT_DIR What's expected to happen if there are multiple files produced in $?OUTPUT_DIR? Is it an error? Also, doesn't this pep overlap with PEP 426? Shouldn't these ideas be incorporated there? It seems that only the command metadata is missing from PEP 426. Another thing, I've noticed there's a "develop" command in this draft. I think this is confusing, I'd expect it to be a "install" command (and that's what it is now in setuptools). It should be renamed to something more clear, like "inplace-build" and leave the installing to pip. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From guettliml at thomas-guettler.de Tue Oct 27 03:13:03 2015 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Tue, 27 Oct 2015 08:13:03 +0100 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> <5629052D.2070201@thomas-guettler.de> Message-ID: <562F23FF.1030306@thomas-guettler.de> Am 26.10.2015 um 20:07 schrieb Nathaniel Smith: > On Oct 26, 2015 6:59 AM, "Wayne Werner" > wrote: > > > > On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker > wrote: > >> > >> On Thu, Oct 22, 2015 at 8:47 AM, Thomas G?ttler > wrote: > >> > >>> > >>> > I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ... > >>> > > >>> > Even for complex packages. > >> > >> > >> It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front. > > > > > > I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively? > > Test the system's implementation of the math.h 'tanh' function to decide whether it correctly implements C99 annex F style handling of edge cases and thus can be used safely, or whether we need to fall back to our internal version of tanh instead, and define some appropriate C preprocessor macros depending on the result. With a plugin-system it would be easy to handle this declaratively. We need a way to extend the supported key-value pairs. This would be done in check_tanh plugin package. Then you need to build-depend on the new check_tanh package. I guess that most developers feel that this is very complicated. I feel the same, but I **think** it is not. The structure is straight forward. It just feels complicated. The check_tanh package can be create in just a few minutes. Regards, Thomas G?ttler -- http://www.thomas-guettler.de/ From njs at pobox.com Tue Oct 27 03:29:44 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Oct 2015 00:29:44 -0700 Subject: [Distutils] What's special about numpy, scipy, ...was: Remove distutils, was: red, green, refactor ... In-Reply-To: <562F23FF.1030306@thomas-guettler.de> References: <56273D6D.9080506@thomas-guettler.de> <85h9lkps5r.fsf@benfinney.id.au> <56276BFE.8010508@thomas-guettler.de> <56277C6E.2030605@thomas-guettler.de> <5627BEAC.3090808@thomas-guettler.de> <5627E33E.8030305@thomas-guettler.de> <5629052D.2070201@thomas-guettler.de> <562F23FF.1030306@thomas-guettler.de> Message-ID: On Tue, Oct 27, 2015 at 12:13 AM, Thomas G?ttler wrote: > Am 26.10.2015 um 20:07 schrieb Nathaniel Smith: >> On Oct 26, 2015 6:59 AM, "Wayne Werner" > wrote: >> > >> > On Thu, Oct 22, 2015 at 11:27 AM, Chris Barker > wrote: >> >> >> >> On Thu, Oct 22, 2015 at 8:47 AM, Thomas G?ttler > wrote: >> >> >> >>> >> >>> > I have a dream: For packaging and building package provides only **data**. Data is condition-less: No single "if", "def" or method call. Just data: json or yaml ... >> >>> > >> >>> > Even for complex packages. >> >> >> >> >> >> It's a nice dream -- but I think there is more or less a consensus on this list that you really can't do everything completely declaratively. I know I would find it very frustrating to have to hard-code everything up front. >> > >> > >> > I've seen this sentiment mentioned several times... Having only had experience with sdist-style packages which are dead simple to define, do you have any examples of some specific thing that's a pain to do declaratively? >> >> Test the system's implementation of the math.h 'tanh' function to decide whether it correctly implements C99 annex F style handling of edge cases and thus can be used safely, or whether we need to fall back to our internal version of tanh instead, and define some appropriate C preprocessor macros depending on the result. > > > With a plugin-system it would be easy to handle this declaratively. > > We need a way to extend the supported key-value pairs. This would be done in check_tanh plugin package. > > Then you need to build-depend on the new check_tanh package. > > I guess that most developers feel that this is very complicated. I feel the same, but I **think** it is not. > The structure is straight forward. It just feels complicated. The check_tanh package > can be create in just a few minutes. Plugins are arbitrary code, that contain plenty of ifs, defs, and method calls... In any case, the next thread over is discussing exactly the question of how we can make it easy for you to write your own build system and see how declarative you can make it, with plugins or without :-). -n -- Nathaniel J. Smith -- http://vorpus.org From cournape at gmail.com Tue Oct 27 04:47:24 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 27 Oct 2015 08:47:24 +0000 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Mon, Oct 26, 2015 at 11:41 PM, Nathaniel Smith wrote: > On Mon, Oct 26, 2015 at 4:41 AM, Donald Stufft wrote: > > On October 26, 2015 at 3:36:47 AM, Nathaniel Smith (njs at pobox.com) > wrote: > >> > TL;DR > >> ----- > >> > >> If we: > >> > >> - implement a real resolver, and > >> - add a notion of a per-project namespace of distribution names, > >> that > >> are collected under the same PyPI registration and come from > >> the same > >> sdist, and > >> - add Conflicts:, and Provides:, > >> > >> then we can elegantly solve a collection of important and difficult > >> problems, and we can retroactively pun the old extras system > >> onto the > >> new system in a way that preserves 100% compatibility with all > >> existing packages. > >> > >> I think? > >> > >> What do you think? > > > > My initial reaction when I started reading your idea was that I didn't > see a > > point in having something like foo[bar] be a "real" package when you > could just > > as easily have foo-bar. However, as I continued to read through the idea > it > > started to grow on me. I think I need to let it percolate in my brain a > little > > bit, but there may be a non-crazy (or at least, crazy in a good way) > idea here > > that could push things forward in a nice way. > > Oh good, at least I'm not the only one :-). > > I'd particularly like to hear Robert's thoughts when he has time, > since the details depend strongly on some assumptions about how a real > resolver would work. > > > Some random thoughts: > > > > * Reusing the extra syntax is nice because it doesn't require end users > to > > learn any new concepts, however we shouldn't take a new syntax off the > table > > either if it makes the feature easier to implement with regards to > backwards > > compatability. Something like numpy{mkl,some-other-thing} could work > just as > > well too. We'll need to make sure that whatever symbols we choose can > be > > represented on all the major FS we care about and that they are > ideally non > > ugly in an URL too. Of course, the filename and user interface symbols > don't > > *need* to match. It could just as easily example numpy[mkl] out to > numpy#mkl > > or whatever which should make it easier to come up with a nice scheme. > > Right -- obviously it would be *nice* to keep the number of concepts > down, and to avoid skew between filenames and user interface (because > users do see filenames), but if these turn out to be impossible then > there are still options that would let us save the > per-project-package-namespace idea. > > > * Provides is a bit of an odd duck, I think in my head I've mostly come > to > > terms with allowing unrestricted Provides when you've already > installed the > > package doing the Providing but completely ignoring the field when > pulling > > data from a repository. Our threat model assumes that once you've > selected to > > install something then it's generally safe to trust (though we still > do try > > to limit that). The problem with Provides mostly comes into play when > you > > will respect the Provides: field for any random package on PyPI (or > any other > > repo). > > Yeah, I'm actually not too worried about malicious use either in > practice, for the reason you say. But even so I can think of two good > reasons we might want to be careful about stating exactly when > "Provides:" can be trusted: > > 1) if you have neither scipy nor numpy installed, and you do 'pip > install scipy', and scipy depends on the pure virtual package > 'numpy[abi-2]' which is only available as a Provides: on the concrete > package 'numpy', then in this case the resolver has to take Provides: > into account when pulling data from the repo -- if it doesn't, then > it'll ignore the Provides: on 'numpy' and say that scipy's > dependencies can't be satisfied. So for this use case to work, we > actually do need to be able to sometimes trust Provides: fields. > > 2) the basic idea of a resolver is that it considers a whole bunch of > possible configurations for your environment, and picks the > configuration that seems best. But if we pay attention to different > metadata when installing as compared to after installation, then this > skew makes it possible for the algorithm to pick a configuration that > looks good a priori but is broken after installation. E.g. for a > simple case: > > Name: a > Conflicts: some-virtual-package > > Name: b > Provides: some-virtual-package > > 'pip install a b' will work, because the resolver ignores the > Provides: and treats the packages as non-conflicting -- but then once > installed we have a broken system. This is obviously an artificial > example, but creating the possibility of such messes just seems like > the kind of headache we don't need. So I think whatever we do with > Provides:, we should do the same thing both before and after > installation. > Another simple solution for this particular case is to add conflict rules between packages that provide the same requirement (that's what php's composer do IIRC). The case of safety against malicious forks is handled quite explicitly in composer, we may want to look at how they do it when considering solutions (e.g. https://github.com/composer/composer/issues/2690, though it has changed a bit since then) Adding the provides/conflict concepts to pip resolver will complexify it quite significantly, both in terms of running time complexity (since at that point you are solving a NP-complete problem) and in terms of implementation. But we also know for real cases this is doable, even in pure python (composer handles all the cases you are mentioning, and is in pure php). David > A simple safe rule is to say that Provides: is always legal iff a > package's Name: and Provides: have a matching BASE, and always illegal > otherwise, making the package just invalid, like if the METADATA were > written in Shift-JIS or something. This rule is trivial to statically > check/enforce, and could always be relaxed more later. > > > * The upgrade mess around extras as they stand today could also be > solved just > > by recording what extras (if any) were selected to be installed so > that we > > keep a consistent view of the world. Your proposal is essentially > doing that, > > just by (ab)using the fact that by installing a package we essentially > get > > that aspect of it for "free". > > Right -- you certainly could implement a database of installed extras > to go alongside the database of installed packages, but it seems like > it just makes things more complicated with minimal benefit. E.g., you > have to add special case code to the resolver to check both databases, > and then you have to add more special case code to 'pip freeze' to > make sure *it* checks both databases... this kind of stuff adds up. > > > * Would this help at all with differentiating between SSE2 and SSE3 > builds and > > things like that? Or does that need something more automatic to be > really > > usable? > > I'm not convinced that SSE2 versus SSE3 is really worth trying to > handle automatically, just because we have more urgent issues and > everyone else in the world seems to get by okay without special > support for this in their package system (even if it's not always > optimal). But if we did want to do this then my intuition is that it'd > be more elegant to do it via the wheel platform/architecture field, > since this actually is a difference in architectures? So you could > have one wheel for the "win32" platform and another wheel for the > "win32sse3" platform, and the code in the installer that figures out > which wheels are compatible would know that both of these are > compatible with the machine it was running on (or not), and that > win32sse3 is preferable to plain win32. > > > * PEP 426 (I think it was?) has some extra syntax for extras which could > > probably be really nice here, things like numpy[*] to get *all* of the > extras > > (though if they are real packages, what even is "all"?). It also > included > > (though this might have been only in my head) default to installed > packages > > which meant you could do something like split numpy into numpy[abi2] > and > > numpy[abi3] packages and have the different ABIs actually contained > within > > those other packages. Then you could have your top level package > default to > > installing abi3 and abi2 so that ``pip install numpy`` is equivilant to > > ``pip install numpy[abi2,abi3]``. The real power there, is that people > can > > trim down their install a bit by then doing ``pip install > numpy[-abi2]`` if > > they don't want to have that on-by-default feature. > > Hmm, right, I'm not thinking of a way to *quite* duplicate this. > > One option would be to have a numpy[all] package that just depends on > all the other extras packages -- for the traditional 'extra' cases > this could be autogenerated by setuptools at build time and then be a > regular package after that, and for next-generation build systems that > had first-class support for these [] packages, it would be up to the > build system / project whether to generate such an [all] package and > what to include in it if they did. But that doesn't give you the > special all-except-for-one behavior. > > The other option that jumps to mind is what Debian calls "recommends", > which act like a soft-dependency: in debian, if numpy recommends: > numpy[abi-2] and numpy[abi-3], then 'apt-get install numpy' would give > you all three of them by default, just like if numpy required them -- > but for recommends: you can also say something like 'apt-get install > numpy -numpy[abi-3]' if you want numpy without the abi-3 package, or > 'apt-get install --no-recommends numpy' if you want a fully minimal > install, and this is okay because these are only *recommendations*, > not an actual requirements. I don't see any fundamental reasons why we > couldn't add something like this to pip, though it's probably not that > urgent. > > My guess is that these two solutions together would pretty much cover > the relevant use cases? > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Tue Oct 27 04:50:20 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 27 Oct 2015 21:50:20 +1300 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On 27 October 2015 at 21:47, David Cournapeau wrote: > Another simple solution for this particular case is to add conflict rules > between packages that provide the same requirement (that's what php's > composer do IIRC). > > The case of safety against malicious forks is handled quite explicitly in > composer, we may want to look at how they do it when considering solutions > (e.g. https://github.com/composer/composer/issues/2690, though it has > changed a bit since then) > > Adding the provides/conflict concepts to pip resolver will complexify it > quite significantly, both in terms of running time complexity (since at that > point you are solving a NP-complete problem) and in terms of implementation. > But we also know for real cases this is doable, even in pure python > (composer handles all the cases you are mentioning, and is in pure php). We already require a full NP-complete solver because of <, <= and ~ version operators. I haven't adsorbed this proposal enough to comment on the reification aspect yet. I'm worried about provides and conflicts in general, but not from a resolver code perspective - thats a one-ish-time-cost, but instead from a user experience perspective. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Tue Oct 27 06:43:01 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Oct 2015 10:43:01 +0000 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On 26 October 2015 at 06:04, Nathaniel Smith wrote: > Here's a second round of text towards making a build-system > independent interface between pip and source trees/sdists. My idea > this time is to take a divide-and-conquer approach: this text tries to > summarize all the stuff that it seemed like we had mostly reached > consensus on in the previous thread + call, with blank chunks marked > "TBD" where there are specific points that still need To Be > Determined. So my hope is that everyone will read what's here and > agree that it's great as far as it goes, and then we can go through > and fill in each missing piece one at a time. I'll comment on what's here, but ignore the TBD items - I'd rather (as you suggest) leave discussion of those details till the basic idea is agreed. > Abstract > ======== > > Distutils delenda est. While this makes a nice tagline, I'd rather something less negative. Distutils does not "need" to be destroyed. It's perfectly adequate (although hardly user friendly) for a lot of cases - I'd be willing to suggest *most* users can work just fine with distutils. I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric limited - as Nick pointed out this whole area is as much a political issue as a technical one. > Extended abstract > ================= > > While ``distutils`` / ``setuptools`` have taken us a long way, they > suffer from three serious problems: (a) they're missing important > features like autoconfiguration and usable build-time dependency > declaration, (b) extending them is quirky, complicated, and fragile, > (c) it's very difficult to use anything else, because they provide the > standard interface for installing python packages expected by both > users and installation tools like ``pip``. Again, this is overstated. You very nearly lost me right here - people won't read the details of the proposal if they disagree with the abstract(s). Specifically: * The features in (a) are only important to *some* parts of the community. The scientific community is the major one, and is a huge influence over the direction we want to go in, but again, not crucial to many people. And even where they might be useful (e.g., Windows users building pyyaml, lxml, pillow, ...) the description implies "working out what's there" rather than "allowing users to easily manage non-Python dependencies", which gives the wrong impression. * The features in (b) are highly specialised. Very few people extend setuptools/distutils. And those who do, have often invested a lot of effort in doing so. Sure, they'd rather not have needed to, but now that they have, a replacement system simply means that work is lost. Arguably, fixing (b) is only useful for people (like the scientific community) who have needed to extend setuptools and have been unable to achieve their goals that way. That's an even smaller part of the community. > Previous efforts (e.g. distutils2 or setuptools itself) have attempted > to solve problems (a) and/or (b). We propose to solve (c). Agreed - this is a good approach. But it's at odds with your abstract, which says distutils must die. Here you're saying you want to allow people to keep using distutils but allow people with specialised needs to choose an alternative. Or are you offering an alternative to people who use distutils? The whole of the above is confusing on the face of it. The details below clarify a lot, as does knowing how the previous discussions have gone. But it would help a lot if the introduction to this PEP were clearer. > The goal of this PEP is get distutils-sig out of the business of being > a gatekeeper for Python build systems. If you want to use distutils, > great; if you want to use something else, then that should be easy to > do using standardized methods. The difficulty of interfacing with > distutils means that there aren't many such systems right now, but to > give a sense of what we're thinking about see `flit > `_ or `bento > `_. Fortunately, wheels have now > solved many of the hard problems here -- e.g. it's no longer necessary > that a build system also know about every possible installation > configuration -- so pretty much all we really need from a build system > is that it have some way to spit out standard-compliant wheels. OK. Although I see a risk here that if I want to build package FOO, I now have to worry whether FOO's build system supports Windows, as well as worrying whether FOO itself supports Windows. There's still a role for some "gatekeeper" (not a good word IMO, maybe "coordinator") to provide a certain level of support or review of build systems, and a point of contact for users with build issues (the point of this proposal is to some extent that people don't need to *know* what build system a project uses, so suggesting everyone has to direct issues to the correct build system support forum isn't necessarily practical). > We therefore propose a new, relatively minimal interface for > installation tools like ``pip`` to interact with package source trees > and source distributions. > > In addition, we propose a wheel-inspired static metadata format for > sdists, suitable for tools like PyPI and pip's resolver. > > > Terminology and goals > ===================== > > A *source tree* is something like a VCS checkout. We need a standard > interface for installing from this format, to support usages like > ``pip install some-directory/``. > > A *source distribution* is a static snapshot representing a particular > release of some source code, like ``lxml-3.4.4.zip``. Source > distributions serve many purposes: they form an archival record of > releases, they provide a stupid-simple de facto standard for tools > that want to ingest and process large corpora of code, possibly > written in many languages (e.g. code search), they act as the input to > downstream packaging systems like Debian/Fedora/Conda/..., and so > forth. In the Python ecosystem they additionally have a particularly > important role to play, because packaging tools like ``pip`` are able > to use source distributions to fulfill binary dependencies, e.g. if > there is a distribution ``foo.whl`` which declares a dependency on > ``bar``, then we need to support the case where ``pip install bar`` or > ``pip install foo`` automatically locates the sdist for ``bar``, > downloads it, builds it, and installs the resulting package. This is somewhat misleading, given that you go on to specify the format below, but maybe that's only an issue for someone like me who saw the previous debate over "source distribution" (as a bundled up source tree) vs "sdist" as a specified format. If I understand, you've now discarded the former sense of source distribution, and are sticking with the latter (specified format) definition. > Source distributions are also known as "sdists" for short. > > > Source trees > ============ > > We retroactively declare the legacy source tree format involving > ``setup.py`` to be "version 0". We don't try to specify it further; > its de facto specification is encoded in the source code and > documentation of ``distutils``, ``setuptools``, ``pip``, and other > tools. > > A "version 1" (or greater) source tree is any directory which contains > a file named ``pypackage.cfg``, which will -- in some manner whose > details are TBD -- describe the package's build dependencies and how > to invoke the build system. This mechanism: > > - Will allow for both static and dynamic specification of build dependencies > > - Will have some degree of isolation of different builds from each > other, so that it will be possible for a single run of pip to install > one package that build-depends on ``foo = 1.1`` and another package > that build-depends on ``foo = 1.2``. All good so far. > - Will leave the actual installation of the package in the hands of > the build/installation tool (i.e. individual package build systems > will not need to know about things like --user versus --global or make > decisions about when and how to modify .pth files) This seems completely backwards to me. It's pip's job to do the actual install. The build tool should *only* focus on generating standard conforming binary wheels - otherwise what's the point of the separation of concerns that wheels provide? Or maybe I'm confused by the term "build/installation tool" - by that did you actually mean pip, rather than the build system? (TBDs omitted) > Source distributions > ==================== > > [possibly this should get split off into a separate PEP, but I'll keep > it together for now for ease of discussion] > > A "version 1" (or greater) source distribution is a file meeting the > following criteria: > > - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where > {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant > version number, and {EXT} is a compliant archive format. > > The set of compliant archive formats is: zip, [TBD] > > [QUESTION: should we continue to allow .tar.gz and friends? In > practice by "allow" I mean something like "accept new-style sdists on > PyPI in this format". I'm inclined not to -- zip is the most > universally supported format around, it allows file-based random > access (unlike tar-based things) which is useful for pulling out > metadata without decompressing the whole thing, and standardizing on > one format dodges distracting and pointless discussions about which > format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to > continue to support other archive formats when passed explicitly on > the command line. Any objections?] +1 on having a single archive format, and zip seems like the best choice. > Similar to wheels, the archive is Unicode, and the filenames inside > the archive are encoded in UTF-8. This isn't the job of the sdist format to specify. It should be implicit in the choice of archive format. Having said that, I'd go with 1. The sdist filename MUST support the full range of package names as specified in PEP 426 (https://www.python.org/dev/peps/pep-0426/#name) and versions as in PEP 440 (https://www.python.org/dev/peps/pep-0440/). That's actually far less than full Unicode. 2. The archive format MUST support arbitrary Unicode filenames. That means zip is OK, but tar.gz isn't unless you specify UTF-8 is used (the tar format doesn't allow for an encoding declaration - see https://docs.python.org/3.5/library/tarfile.html#tar-unicode for details on Unicode issues in the tar format). Having said that I'd also go with "filenames in the archive SHOULD be limited to ASCII" - because we have had issues with pip where test files have Unicode filenames, and builds break because they get mangled on systems with weird encoding setups... IIRC, these are typically related to .tar.gz sdists, which (due to the lack of encoding support) result in files being unpacked with the wrong names. So maybe if we enforce zip format we don't need to add this limitation. > - When unpacked, it MUST contain a single directory directory tree > named ``{PACKAGE}-{VERSION}``. > > - This directory tree MUST be a valid version 1 (or greater) source > tree as defined above. > > - It MUST additionally contain a directory named > ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the > following contents: > > - ``SDIST``: Mandatory. Same record-oriented format as a wheel's > ``WHEEL`` file, but with different fields:: > > SDist-Version: 1.0 > Generator: setuptools sdist 20.1 > > ``SDist-Version`` is the version number of this specification. > Software that processes sdists should warn if ``SDist-Version`` is > greater than the version it supports, and must fail if > ``SDist-Version`` has a greater major version than the version it > supports. > > ``Generator`` is the name and optionally the version of the > software that produced the archive. > > - ``RECORD``: Mandatory. A list of all files contained in the sdist > (except for the RECORD file itself and any signature files) together > with their hashes, as specified in PEP 427. > > - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as > specified in PEP 427. > > - ``METADATA``: Mandatory. Metadata version 1.1 or greater format > metadata, with an additional rule that fields may contain the special > sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value > of this field cannot be determined until build time. If a "multiple > use field" is present with the value ``__SDIST_DYNAMIC__``, then this > field MUST occur exactly once, e.g.:: > > # Okay: > Requires-Dist: lxml (> 3.3) > Requires-Dist: requests > > # no Requires-Dist lines at all is okay > # (meaning: this package's requirements are the empty set) > > # Okay, requirements will be determined at build time: > Requires-Dist: __SDIST_DYNAMIC__ > > # NOT okay: > Requires-Dist: lxml (> 3.3) > Requires-Dist: __SDIST_DYNAMIC__ > > (The use of a special token allows us to distinguish between > multiple use fields whose value is statically the empty list versus > one whose value is dynamic; it also allows us to distinguish between > optional fields which are statically not present versus ones whose > value is dynamic.) > > When this sdist is built, the resulting wheel MUST have metadata > which is identical to the metadata present in this file, except that > any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have > arbitrary values in the wheel. > > A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for > the package name or version (i.e., these must be given statically), > and these MUST match the {PACKAGE} and {VERSION} of the sdist as > described above. This seems pretty good at first reading. > [TBD: do we want to forbid the use of dynamic metadata for any > other fields? I assume PyPI will enforce some stricter rules at least, > but I don't know if we want to make that part of the spec, or just > part of PyPI's administrative rules.] This covers the main point of contention. It would be bad if build systems started using __SDIST_DYNAMIC__ just because "it's easier". Maybe add * A valid sdist SHOULD NOT use the __SDIST_DYNAMIC__ mechanism any more than necessary (i.e., if the metadata is the same in all generated wheels, it does not need to use the __SDIST_DYNAMIC__ mechanism, and so should not do so). > This is intentionally a close analogue of a wheel's ``.dist-info`` > directory; intention is that as future metadata standards are defined, > the specifications for the ``.sdist-info`` and ``.dist-info`` > directories will evolve in synchrony. > > > Evolutionary notes > ================== > > A goal here is to make it as simple as possible to convert old-style > sdists to new-style sdists. (E.g., this is one motivation for > supporting dynamic build requirements.) The ideal would be that there > would be a single static pypackage.cfg that could be dropped into any > "version 0" VCS checkout to convert it to the new shiny. This is > probably not 100% possible, but we can get close, and it's important > to keep track of how close we are... hence this section. > > A rough plan would be: Create a build system package > (``setuptools_pypackage`` or whatever) that knows how to speak > whatever hook language we come up with, and convert them into > setuptools calls. This will probably require some sort of hooking or > monkeypatching to setuptools to provide a way to extract the > ``setup_requires=`` argument when needed, and to provide a new version > of the sdist command that generates the new-style format. This all > seems doable and sufficient for a large proportion of packages (though > obviously we'll want to prototype such a system before we finalize > anything here). (Alternatively, these changes could be made to > setuptools itself rather than going into a separate package.) > > But there remain two obstacles that mean we probably won't be able to > automatically upgrade packages to the new format: > > 1) There currently exist packages which insist on particular packages > being available in their environment before setup.py is executed. This > means that if we decide to execute build scripts in an isolated > virtualenv-like environment, then projects will need to check whether > they do this, and if so then when upgrading to the new system they > will have to start explicitly declaring these dependencies (either via > ``setup_requires=`` or via static declaration in ``pypackage.cfg``). > > 2) There currently exist packages which do not declare consistent > metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different > ``install_requires=``). When upgrading to the new system, projects > will have to evaluate whether this applies to them, and if so they > will need to either stop doing that, or else add ``__SDIST_DYNAMIC__`` > annotations at appropriate places. > > We'll also presumably need some API for packages to describe which > parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for > the packages that need it (a new argument to ``setup()`` or some > setting in ``setup.cfg`` or something). I'm confused here. And it's just now become clear *why* I'm confused. The sdist format MUST be a generated format - i.e., we should insist (in principle at least) that it's only ever generated by tools. Otherwise it's way too easy for people to just zip up their source tree, hand craft something generic (that over-uses __SDIST_DYNAMIC__) and say "here's an sdist". Obviously, people always *can* manually create an sdist but we need to pin down the spec tightly, or we've not improved things. That's why I'm concerned about __SDIST_DYNAMIC__ and it's also what confuses me about the above transition plan. For people using setuptools currently, the transition should be simply that they upgrade setuptools, and the "setup.py sdist" command in the new setuptools generates the new sdist format. By default, the setuptools sdist process assumes everything is static and requires the user to modify the setup.py to explicitly mark which metadata they want to be left to build time. That way, we get a relatively transparent transition, while avoiding overuse of dynamic metadata. If setup.py has to explicitly mark dynamic metadata, that also allows us to reject attempts to make name and version dynamic. Which is good. Paul From dholth at gmail.com Tue Oct 27 08:29:57 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 27 Oct 2015 12:29:57 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: Mr. grumpy pants, Now you're trolling us by describing flit. Older versions of pip could be able to install it, just add a setup.py shim. Nearly trivial. A new simple build system could come with the agile build system support. It's just a matter of timing. On Sun, Oct 25, 2015 at 9:49 PM Donald Stufft wrote: > On October 25, 2015 at 7:17:12 PM, Nick Coghlan (ncoghlan at gmail.com) > wrote: > > > > And now that I've got my "Mr grumpy pants" reaction to the earlier > > parts of the thread out of my system (sorry about that), I'd like to > > revisit Chris's idea of having a *simple* replacement for setuptools > > as a build system. > > > > I?m not *against* it, but it has some problems: > > * Older versions of pip and setuptools will not be able to install it at > all. This includes anyplace setuptools is used implicitly such as > ``setup.py install`` from another package that depends on it or > setup_requires or test_requires etc. > > * A lot of downstream folks will probably be mad at us for introducing a > second way of doing things (particularly since it won?t come with the agile > build system support that we have on the road map that will make this a lot > better for non setuptools build systems). > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Oct 27 09:12:28 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 27 Oct 2015 13:12:28 +0000 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: The drawback of .zip is file size since it compresses each file individually rather than giving the compression algorithm a larger input, it's a great format otherwise. Ubiquitous including Apple iOS packages, Java, word processor file formats. And most Python packages are small. We must do the hard work to support Unicode file names, and spaces and accent marks in home directory names (historically a problem on Windows), in our packaging system. It is the right thing to do. It is not the publisher's fault that your system has broken Unicode. On Tue, Oct 27, 2015 at 6:43 AM Paul Moore wrote: > On 26 October 2015 at 06:04, Nathaniel Smith wrote: > > Here's a second round of text towards making a build-system > > independent interface between pip and source trees/sdists. My idea > > this time is to take a divide-and-conquer approach: this text tries to > > summarize all the stuff that it seemed like we had mostly reached > > consensus on in the previous thread + call, with blank chunks marked > > "TBD" where there are specific points that still need To Be > > Determined. So my hope is that everyone will read what's here and > > agree that it's great as far as it goes, and then we can go through > > and fill in each missing piece one at a time. > > I'll comment on what's here, but ignore the TBD items - I'd rather (as > you suggest) leave discussion of those details till the basic idea is > agreed. > > > Abstract > > ======== > > > > Distutils delenda est. > > While this makes a nice tagline, I'd rather something less negative. > Distutils does not "need" to be destroyed. It's perfectly adequate > (although hardly user friendly) for a lot of cases - I'd be willing to > suggest *most* users can work just fine with distutils. > > I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric > limited - as Nick pointed out this whole area is as much a political > issue as a technical one. > > > Extended abstract > > ================= > > > > While ``distutils`` / ``setuptools`` have taken us a long way, they > > suffer from three serious problems: (a) they're missing important > > features like autoconfiguration and usable build-time dependency > > declaration, (b) extending them is quirky, complicated, and fragile, > > (c) it's very difficult to use anything else, because they provide the > > standard interface for installing python packages expected by both > > users and installation tools like ``pip``. > > Again, this is overstated. You very nearly lost me right here - people > won't read the details of the proposal if they disagree with the > abstract(s). Specifically: > > * The features in (a) are only important to *some* parts of the > community. The scientific community is the major one, and is a huge > influence over the direction we want to go in, but again, not crucial > to many people. And even where they might be useful (e.g., Windows > users building pyyaml, lxml, pillow, ...) the description implies > "working out what's there" rather than "allowing users to easily > manage non-Python dependencies", which gives the wrong impression. > > * The features in (b) are highly specialised. Very few people extend > setuptools/distutils. And those who do, have often invested a lot of > effort in doing so. Sure, they'd rather not have needed to, but now > that they have, a replacement system simply means that work is lost. > Arguably, fixing (b) is only useful for people (like the scientific > community) who have needed to extend setuptools and have been unable > to achieve their goals that way. That's an even smaller part of the > community. > > > Previous efforts (e.g. distutils2 or setuptools itself) have attempted > > to solve problems (a) and/or (b). We propose to solve (c). > > Agreed - this is a good approach. But it's at odds with your abstract, > which says distutils must die. Here you're saying you want to allow > people to keep using distutils but allow people with specialised needs > to choose an alternative. Or are you offering an alternative to people > who use distutils? > > The whole of the above is confusing on the face of it. The details > below clarify a lot, as does knowing how the previous discussions have > gone. But it would help a lot if the introduction to this PEP were > clearer. > > > The goal of this PEP is get distutils-sig out of the business of being > > a gatekeeper for Python build systems. If you want to use distutils, > > great; if you want to use something else, then that should be easy to > > do using standardized methods. The difficulty of interfacing with > > distutils means that there aren't many such systems right now, but to > > give a sense of what we're thinking about see `flit > > `_ or `bento > > `_. Fortunately, wheels have now > > solved many of the hard problems here -- e.g. it's no longer necessary > > that a build system also know about every possible installation > > configuration -- so pretty much all we really need from a build system > > is that it have some way to spit out standard-compliant wheels. > > OK. Although I see a risk here that if I want to build package FOO, I > now have to worry whether FOO's build system supports Windows, as well > as worrying whether FOO itself supports Windows. > > There's still a role for some "gatekeeper" (not a good word IMO, maybe > "coordinator") to provide a certain level of support or review of > build systems, and a point of contact for users with build issues (the > point of this proposal is to some extent that people don't need to > *know* what build system a project uses, so suggesting everyone has to > direct issues to the correct build system support forum isn't > necessarily practical). > > > We therefore propose a new, relatively minimal interface for > > installation tools like ``pip`` to interact with package source trees > > and source distributions. > > > > In addition, we propose a wheel-inspired static metadata format for > > sdists, suitable for tools like PyPI and pip's resolver. > > > > > > Terminology and goals > > ===================== > > > > A *source tree* is something like a VCS checkout. We need a standard > > interface for installing from this format, to support usages like > > ``pip install some-directory/``. > > > > A *source distribution* is a static snapshot representing a particular > > release of some source code, like ``lxml-3.4.4.zip``. Source > > distributions serve many purposes: they form an archival record of > > releases, they provide a stupid-simple de facto standard for tools > > that want to ingest and process large corpora of code, possibly > > written in many languages (e.g. code search), they act as the input to > > downstream packaging systems like Debian/Fedora/Conda/..., and so > > forth. In the Python ecosystem they additionally have a particularly > > important role to play, because packaging tools like ``pip`` are able > > to use source distributions to fulfill binary dependencies, e.g. if > > there is a distribution ``foo.whl`` which declares a dependency on > > ``bar``, then we need to support the case where ``pip install bar`` or > > ``pip install foo`` automatically locates the sdist for ``bar``, > > downloads it, builds it, and installs the resulting package. > > This is somewhat misleading, given that you go on to specify the > format below, but maybe that's only an issue for someone like me who > saw the previous debate over "source distribution" (as a bundled up > source tree) vs "sdist" as a specified format. If I understand, you've > now discarded the former sense of source distribution, and are > sticking with the latter (specified format) definition. > > > Source distributions are also known as "sdists" for short. > > > > > > Source trees > > ============ > > > > We retroactively declare the legacy source tree format involving > > ``setup.py`` to be "version 0". We don't try to specify it further; > > its de facto specification is encoded in the source code and > > documentation of ``distutils``, ``setuptools``, ``pip``, and other > > tools. > > > > A "version 1" (or greater) source tree is any directory which contains > > a file named ``pypackage.cfg``, which will -- in some manner whose > > details are TBD -- describe the package's build dependencies and how > > to invoke the build system. This mechanism: > > > > - Will allow for both static and dynamic specification of build > dependencies > > > > - Will have some degree of isolation of different builds from each > > other, so that it will be possible for a single run of pip to install > > one package that build-depends on ``foo = 1.1`` and another package > > that build-depends on ``foo = 1.2``. > > All good so far. > > > - Will leave the actual installation of the package in the hands of > > the build/installation tool (i.e. individual package build systems > > will not need to know about things like --user versus --global or make > > decisions about when and how to modify .pth files) > > This seems completely backwards to me. It's pip's job to do the actual > install. The build tool should *only* focus on generating standard > conforming binary wheels - otherwise what's the point of the > separation of concerns that wheels provide? > > Or maybe I'm confused by the term "build/installation tool" - by that > did you actually mean pip, rather than the build system? > > (TBDs omitted) > > > Source distributions > > ==================== > > > > [possibly this should get split off into a separate PEP, but I'll keep > > it together for now for ease of discussion] > > > > A "version 1" (or greater) source distribution is a file meeting the > > following criteria: > > > > - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where > > {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant > > version number, and {EXT} is a compliant archive format. > > > > The set of compliant archive formats is: zip, [TBD] > > > > [QUESTION: should we continue to allow .tar.gz and friends? In > > practice by "allow" I mean something like "accept new-style sdists on > > PyPI in this format". I'm inclined not to -- zip is the most > > universally supported format around, it allows file-based random > > access (unlike tar-based things) which is useful for pulling out > > metadata without decompressing the whole thing, and standardizing on > > one format dodges distracting and pointless discussions about which > > format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to > > continue to support other archive formats when passed explicitly on > > the command line. Any objections?] > > +1 on having a single archive format, and zip seems like the best choice. > > > Similar to wheels, the archive is Unicode, and the filenames inside > > the archive are encoded in UTF-8. > > This isn't the job of the sdist format to specify. It should be > implicit in the choice of archive format. > > Having said that, I'd go with > > 1. The sdist filename MUST support the full range of package names as > specified in PEP 426 (https://www.python.org/dev/peps/pep-0426/#name) > and versions as in PEP 440 > (https://www.python.org/dev/peps/pep-0440/). That's actually far less > than full Unicode. > 2. The archive format MUST support arbitrary Unicode filenames. That > means zip is OK, but tar.gz isn't unless you specify UTF-8 is used > (the tar format doesn't allow for an encoding declaration - see > https://docs.python.org/3.5/library/tarfile.html#tar-unicode for > details on Unicode issues in the tar format). > > Having said that I'd also go with "filenames in the archive SHOULD be > limited to ASCII" - because we have had issues with pip where test > files have Unicode filenames, and builds break because they get > mangled on systems with weird encoding setups... IIRC, these are > typically related to .tar.gz sdists, which (due to the lack of > encoding support) result in files being unpacked with the wrong names. > So maybe if we enforce zip format we don't need to add this > limitation. > > > - When unpacked, it MUST contain a single directory directory tree > > named ``{PACKAGE}-{VERSION}``. > > > > - This directory tree MUST be a valid version 1 (or greater) source > > tree as defined above. > > > > - It MUST additionally contain a directory named > > ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the > > following contents: > > > > - ``SDIST``: Mandatory. Same record-oriented format as a wheel's > > ``WHEEL`` file, but with different fields:: > > > > SDist-Version: 1.0 > > Generator: setuptools sdist 20.1 > > > > ``SDist-Version`` is the version number of this specification. > > Software that processes sdists should warn if ``SDist-Version`` is > > greater than the version it supports, and must fail if > > ``SDist-Version`` has a greater major version than the version it > > supports. > > > > ``Generator`` is the name and optionally the version of the > > software that produced the archive. > > > > - ``RECORD``: Mandatory. A list of all files contained in the sdist > > (except for the RECORD file itself and any signature files) together > > with their hashes, as specified in PEP 427. > > > > - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as > > specified in PEP 427. > > > > - ``METADATA``: Mandatory. Metadata version 1.1 or greater format > > metadata, with an additional rule that fields may contain the special > > sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value > > of this field cannot be determined until build time. If a "multiple > > use field" is present with the value ``__SDIST_DYNAMIC__``, then this > > field MUST occur exactly once, e.g.:: > > > > # Okay: > > Requires-Dist: lxml (> 3.3) > > Requires-Dist: requests > > > > # no Requires-Dist lines at all is okay > > # (meaning: this package's requirements are the empty set) > > > > # Okay, requirements will be determined at build time: > > Requires-Dist: __SDIST_DYNAMIC__ > > > > # NOT okay: > > Requires-Dist: lxml (> 3.3) > > Requires-Dist: __SDIST_DYNAMIC__ > > > > (The use of a special token allows us to distinguish between > > multiple use fields whose value is statically the empty list versus > > one whose value is dynamic; it also allows us to distinguish between > > optional fields which are statically not present versus ones whose > > value is dynamic.) > > > > When this sdist is built, the resulting wheel MUST have metadata > > which is identical to the metadata present in this file, except that > > any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have > > arbitrary values in the wheel. > > > > A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for > > the package name or version (i.e., these must be given statically), > > and these MUST match the {PACKAGE} and {VERSION} of the sdist as > > described above. > > This seems pretty good at first reading. > > > [TBD: do we want to forbid the use of dynamic metadata for any > > other fields? I assume PyPI will enforce some stricter rules at least, > > but I don't know if we want to make that part of the spec, or just > > part of PyPI's administrative rules.] > > This covers the main point of contention. It would be bad if build > systems started using __SDIST_DYNAMIC__ just because "it's easier". > > Maybe add > > * A valid sdist SHOULD NOT use the __SDIST_DYNAMIC__ mechanism any > more than necessary (i.e., if the metadata is the same in all > generated wheels, it does not need to use the __SDIST_DYNAMIC__ > mechanism, and so should not do so). > > > This is intentionally a close analogue of a wheel's ``.dist-info`` > > directory; intention is that as future metadata standards are defined, > > the specifications for the ``.sdist-info`` and ``.dist-info`` > > directories will evolve in synchrony. > > > > > > Evolutionary notes > > ================== > > > > A goal here is to make it as simple as possible to convert old-style > > sdists to new-style sdists. (E.g., this is one motivation for > > supporting dynamic build requirements.) The ideal would be that there > > would be a single static pypackage.cfg that could be dropped into any > > "version 0" VCS checkout to convert it to the new shiny. This is > > probably not 100% possible, but we can get close, and it's important > > to keep track of how close we are... hence this section. > > > > A rough plan would be: Create a build system package > > (``setuptools_pypackage`` or whatever) that knows how to speak > > whatever hook language we come up with, and convert them into > > setuptools calls. This will probably require some sort of hooking or > > monkeypatching to setuptools to provide a way to extract the > > ``setup_requires=`` argument when needed, and to provide a new version > > of the sdist command that generates the new-style format. This all > > seems doable and sufficient for a large proportion of packages (though > > obviously we'll want to prototype such a system before we finalize > > anything here). (Alternatively, these changes could be made to > > setuptools itself rather than going into a separate package.) > > > > But there remain two obstacles that mean we probably won't be able to > > automatically upgrade packages to the new format: > > > > 1) There currently exist packages which insist on particular packages > > being available in their environment before setup.py is executed. This > > means that if we decide to execute build scripts in an isolated > > virtualenv-like environment, then projects will need to check whether > > they do this, and if so then when upgrading to the new system they > > will have to start explicitly declaring these dependencies (either via > > ``setup_requires=`` or via static declaration in ``pypackage.cfg``). > > > > 2) There currently exist packages which do not declare consistent > > metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different > > ``install_requires=``). When upgrading to the new system, projects > > will have to evaluate whether this applies to them, and if so they > > will need to either stop doing that, or else add ``__SDIST_DYNAMIC__`` > > annotations at appropriate places. > > > > We'll also presumably need some API for packages to describe which > > parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for > > the packages that need it (a new argument to ``setup()`` or some > > setting in ``setup.cfg`` or something). > > I'm confused here. And it's just now become clear *why* I'm confused. > > The sdist format MUST be a generated format - i.e., we should insist > (in principle at least) that it's only ever generated by tools. > Otherwise it's way too easy for people to just zip up their source > tree, hand craft something generic (that over-uses __SDIST_DYNAMIC__) > and say "here's an sdist". Obviously, people always *can* manually > create an sdist but we need to pin down the spec tightly, or we've not > improved things. > > That's why I'm concerned about __SDIST_DYNAMIC__ and it's also what > confuses me about the above transition plan. > > For people using setuptools currently, the transition should be simply > that they upgrade setuptools, and the "setup.py sdist" command in the > new setuptools generates the new sdist format. By default, the > setuptools sdist process assumes everything is static and requires the > user to modify the setup.py to explicitly mark which metadata they > want to be left to build time. That way, we get a relatively > transparent transition, while avoiding overuse of dynamic metadata. > > If setup.py has to explicitly mark dynamic metadata, that also allows > us to reject attempts to make name and version dynamic. Which is good. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From soltysh at gmail.com Mon Oct 26 16:21:29 2015 From: soltysh at gmail.com (Maciej Szulik) Date: Mon, 26 Oct 2015 21:21:29 +0100 Subject: [Distutils] (no subject) Message-ID: Thanks to Nick Coghlan and Barry Warsaw we've setup a new SIG dedicated to discussing python features from different distributions point of view. Here is Nick's reasoning: > With the Python 3 migration, and the growth in interest in user level > package management for development purposes, what do you think of the idea > of setting up a new Linux SIG to have those discussions? I know it's a case > of "yet another mailing list", but I think it will be worthwhile to have a > clear point of collaboration within the Python ecosystem, rather than > expecting Pythonistas to know how to reach out to (other) distros directly. The list is available @ https://mail.python.org/mailman/listinfo/linux-sig Maciej -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Oct 27 09:28:09 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 27 Oct 2015 13:28:09 +0000 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: It looks like you would normally be able to say just 'flit' in the bootstrap-requires or provided-by, unless the requirements were for build code bundled inside the sdist. On Tue, Oct 27, 2015 at 1:07 AM Robert Collins wrote: > On 27 October 2015 at 14:06, Robert Collins > wrote: > > > We can rev this in schema version 2. There's no prose in the PEP about > > how that should work, so I'll add that now. > > Done: https://gist.github.com/rbtcollins/666c12aec869237f7cf7#upgrades > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Oct 27 10:00:25 2015 From: cournape at gmail.com (David Cournapeau) Date: Tue, 27 Oct 2015 14:00:25 +0000 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 1:12 PM, Daniel Holth wrote: > The drawback of .zip is file size since it compresses each file > individually rather than giving the compression algorithm a larger input, > it's a great format otherwise. Ubiquitous including Apple iOS packages, > Java, word processor file formats. And most Python packages are small. > I don't really buy the indexing advantages, especially w/ the current implementation of zipfile in python (e.g. loading the whole set of archives at creation time) A common way to solve the fast metadata access from archive is to archive separately the metadata data and data (e.g. a zipfile containing 2 zipfiles, one being the original sdist, the other one containing the metadata). David > We must do the hard work to support Unicode file names, and spaces and > accent marks in home directory names (historically a problem on Windows), > in our packaging system. It is the right thing to do. It is not the > publisher's fault that your system has broken Unicode. > > On Tue, Oct 27, 2015 at 6:43 AM Paul Moore wrote: > >> On 26 October 2015 at 06:04, Nathaniel Smith wrote: >> > Here's a second round of text towards making a build-system >> > independent interface between pip and source trees/sdists. My idea >> > this time is to take a divide-and-conquer approach: this text tries to >> > summarize all the stuff that it seemed like we had mostly reached >> > consensus on in the previous thread + call, with blank chunks marked >> > "TBD" where there are specific points that still need To Be >> > Determined. So my hope is that everyone will read what's here and >> > agree that it's great as far as it goes, and then we can go through >> > and fill in each missing piece one at a time. >> >> I'll comment on what's here, but ignore the TBD items - I'd rather (as >> you suggest) leave discussion of those details till the basic idea is >> agreed. >> >> > Abstract >> > ======== >> > >> > Distutils delenda est. >> >> While this makes a nice tagline, I'd rather something less negative. >> Distutils does not "need" to be destroyed. It's perfectly adequate >> (although hardly user friendly) for a lot of cases - I'd be willing to >> suggest *most* users can work just fine with distutils. >> >> I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric >> limited - as Nick pointed out this whole area is as much a political >> issue as a technical one. >> >> > Extended abstract >> > ================= >> > >> > While ``distutils`` / ``setuptools`` have taken us a long way, they >> > suffer from three serious problems: (a) they're missing important >> > features like autoconfiguration and usable build-time dependency >> > declaration, (b) extending them is quirky, complicated, and fragile, >> > (c) it's very difficult to use anything else, because they provide the >> > standard interface for installing python packages expected by both >> > users and installation tools like ``pip``. >> >> Again, this is overstated. You very nearly lost me right here - people >> won't read the details of the proposal if they disagree with the >> abstract(s). Specifically: >> >> * The features in (a) are only important to *some* parts of the >> community. The scientific community is the major one, and is a huge >> influence over the direction we want to go in, but again, not crucial >> to many people. And even where they might be useful (e.g., Windows >> users building pyyaml, lxml, pillow, ...) the description implies >> "working out what's there" rather than "allowing users to easily >> manage non-Python dependencies", which gives the wrong impression. >> >> * The features in (b) are highly specialised. Very few people extend >> setuptools/distutils. And those who do, have often invested a lot of >> effort in doing so. Sure, they'd rather not have needed to, but now >> that they have, a replacement system simply means that work is lost. >> Arguably, fixing (b) is only useful for people (like the scientific >> community) who have needed to extend setuptools and have been unable >> to achieve their goals that way. That's an even smaller part of the >> community. >> >> > Previous efforts (e.g. distutils2 or setuptools itself) have attempted >> > to solve problems (a) and/or (b). We propose to solve (c). >> >> Agreed - this is a good approach. But it's at odds with your abstract, >> which says distutils must die. Here you're saying you want to allow >> people to keep using distutils but allow people with specialised needs >> to choose an alternative. Or are you offering an alternative to people >> who use distutils? >> >> The whole of the above is confusing on the face of it. The details >> below clarify a lot, as does knowing how the previous discussions have >> gone. But it would help a lot if the introduction to this PEP were >> clearer. >> >> > The goal of this PEP is get distutils-sig out of the business of being >> > a gatekeeper for Python build systems. If you want to use distutils, >> > great; if you want to use something else, then that should be easy to >> > do using standardized methods. The difficulty of interfacing with >> > distutils means that there aren't many such systems right now, but to >> > give a sense of what we're thinking about see `flit >> > `_ or `bento >> > `_. Fortunately, wheels have now >> > solved many of the hard problems here -- e.g. it's no longer necessary >> > that a build system also know about every possible installation >> > configuration -- so pretty much all we really need from a build system >> > is that it have some way to spit out standard-compliant wheels. >> >> OK. Although I see a risk here that if I want to build package FOO, I >> now have to worry whether FOO's build system supports Windows, as well >> as worrying whether FOO itself supports Windows. >> >> There's still a role for some "gatekeeper" (not a good word IMO, maybe >> "coordinator") to provide a certain level of support or review of >> build systems, and a point of contact for users with build issues (the >> point of this proposal is to some extent that people don't need to >> *know* what build system a project uses, so suggesting everyone has to >> direct issues to the correct build system support forum isn't >> necessarily practical). >> >> > We therefore propose a new, relatively minimal interface for >> > installation tools like ``pip`` to interact with package source trees >> > and source distributions. >> > >> > In addition, we propose a wheel-inspired static metadata format for >> > sdists, suitable for tools like PyPI and pip's resolver. >> > >> > >> > Terminology and goals >> > ===================== >> > >> > A *source tree* is something like a VCS checkout. We need a standard >> > interface for installing from this format, to support usages like >> > ``pip install some-directory/``. >> > >> > A *source distribution* is a static snapshot representing a particular >> > release of some source code, like ``lxml-3.4.4.zip``. Source >> > distributions serve many purposes: they form an archival record of >> > releases, they provide a stupid-simple de facto standard for tools >> > that want to ingest and process large corpora of code, possibly >> > written in many languages (e.g. code search), they act as the input to >> > downstream packaging systems like Debian/Fedora/Conda/..., and so >> > forth. In the Python ecosystem they additionally have a particularly >> > important role to play, because packaging tools like ``pip`` are able >> > to use source distributions to fulfill binary dependencies, e.g. if >> > there is a distribution ``foo.whl`` which declares a dependency on >> > ``bar``, then we need to support the case where ``pip install bar`` or >> > ``pip install foo`` automatically locates the sdist for ``bar``, >> > downloads it, builds it, and installs the resulting package. >> >> This is somewhat misleading, given that you go on to specify the >> format below, but maybe that's only an issue for someone like me who >> saw the previous debate over "source distribution" (as a bundled up >> source tree) vs "sdist" as a specified format. If I understand, you've >> now discarded the former sense of source distribution, and are >> sticking with the latter (specified format) definition. >> >> > Source distributions are also known as "sdists" for short. >> > >> > >> > Source trees >> > ============ >> > >> > We retroactively declare the legacy source tree format involving >> > ``setup.py`` to be "version 0". We don't try to specify it further; >> > its de facto specification is encoded in the source code and >> > documentation of ``distutils``, ``setuptools``, ``pip``, and other >> > tools. >> > >> > A "version 1" (or greater) source tree is any directory which contains >> > a file named ``pypackage.cfg``, which will -- in some manner whose >> > details are TBD -- describe the package's build dependencies and how >> > to invoke the build system. This mechanism: >> > >> > - Will allow for both static and dynamic specification of build >> dependencies >> > >> > - Will have some degree of isolation of different builds from each >> > other, so that it will be possible for a single run of pip to install >> > one package that build-depends on ``foo = 1.1`` and another package >> > that build-depends on ``foo = 1.2``. >> >> All good so far. >> >> > - Will leave the actual installation of the package in the hands of >> > the build/installation tool (i.e. individual package build systems >> > will not need to know about things like --user versus --global or make >> > decisions about when and how to modify .pth files) >> >> This seems completely backwards to me. It's pip's job to do the actual >> install. The build tool should *only* focus on generating standard >> conforming binary wheels - otherwise what's the point of the >> separation of concerns that wheels provide? >> >> Or maybe I'm confused by the term "build/installation tool" - by that >> did you actually mean pip, rather than the build system? >> >> (TBDs omitted) >> >> > Source distributions >> > ==================== >> > >> > [possibly this should get split off into a separate PEP, but I'll keep >> > it together for now for ease of discussion] >> > >> > A "version 1" (or greater) source distribution is a file meeting the >> > following criteria: >> > >> > - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where >> > {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant >> > version number, and {EXT} is a compliant archive format. >> > >> > The set of compliant archive formats is: zip, [TBD] >> > >> > [QUESTION: should we continue to allow .tar.gz and friends? In >> > practice by "allow" I mean something like "accept new-style sdists on >> > PyPI in this format". I'm inclined not to -- zip is the most >> > universally supported format around, it allows file-based random >> > access (unlike tar-based things) which is useful for pulling out >> > metadata without decompressing the whole thing, and standardizing on >> > one format dodges distracting and pointless discussions about which >> > format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to >> > continue to support other archive formats when passed explicitly on >> > the command line. Any objections?] >> >> +1 on having a single archive format, and zip seems like the best choice. >> >> > Similar to wheels, the archive is Unicode, and the filenames inside >> > the archive are encoded in UTF-8. >> >> This isn't the job of the sdist format to specify. It should be >> implicit in the choice of archive format. >> >> Having said that, I'd go with >> >> 1. The sdist filename MUST support the full range of package names as >> specified in PEP 426 (https://www.python.org/dev/peps/pep-0426/#name) >> and versions as in PEP 440 >> (https://www.python.org/dev/peps/pep-0440/). That's actually far less >> than full Unicode. >> 2. The archive format MUST support arbitrary Unicode filenames. That >> means zip is OK, but tar.gz isn't unless you specify UTF-8 is used >> (the tar format doesn't allow for an encoding declaration - see >> https://docs.python.org/3.5/library/tarfile.html#tar-unicode for >> details on Unicode issues in the tar format). >> >> Having said that I'd also go with "filenames in the archive SHOULD be >> limited to ASCII" - because we have had issues with pip where test >> files have Unicode filenames, and builds break because they get >> mangled on systems with weird encoding setups... IIRC, these are >> typically related to .tar.gz sdists, which (due to the lack of >> encoding support) result in files being unpacked with the wrong names. >> So maybe if we enforce zip format we don't need to add this >> limitation. >> >> > - When unpacked, it MUST contain a single directory directory tree >> > named ``{PACKAGE}-{VERSION}``. >> > >> > - This directory tree MUST be a valid version 1 (or greater) source >> > tree as defined above. >> > >> > - It MUST additionally contain a directory named >> > ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the >> > following contents: >> > >> > - ``SDIST``: Mandatory. Same record-oriented format as a wheel's >> > ``WHEEL`` file, but with different fields:: >> > >> > SDist-Version: 1.0 >> > Generator: setuptools sdist 20.1 >> > >> > ``SDist-Version`` is the version number of this specification. >> > Software that processes sdists should warn if ``SDist-Version`` is >> > greater than the version it supports, and must fail if >> > ``SDist-Version`` has a greater major version than the version it >> > supports. >> > >> > ``Generator`` is the name and optionally the version of the >> > software that produced the archive. >> > >> > - ``RECORD``: Mandatory. A list of all files contained in the sdist >> > (except for the RECORD file itself and any signature files) together >> > with their hashes, as specified in PEP 427. >> > >> > - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as >> > specified in PEP 427. >> > >> > - ``METADATA``: Mandatory. Metadata version 1.1 or greater format >> > metadata, with an additional rule that fields may contain the special >> > sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value >> > of this field cannot be determined until build time. If a "multiple >> > use field" is present with the value ``__SDIST_DYNAMIC__``, then this >> > field MUST occur exactly once, e.g.:: >> > >> > # Okay: >> > Requires-Dist: lxml (> 3.3) >> > Requires-Dist: requests >> > >> > # no Requires-Dist lines at all is okay >> > # (meaning: this package's requirements are the empty set) >> > >> > # Okay, requirements will be determined at build time: >> > Requires-Dist: __SDIST_DYNAMIC__ >> > >> > # NOT okay: >> > Requires-Dist: lxml (> 3.3) >> > Requires-Dist: __SDIST_DYNAMIC__ >> > >> > (The use of a special token allows us to distinguish between >> > multiple use fields whose value is statically the empty list versus >> > one whose value is dynamic; it also allows us to distinguish between >> > optional fields which are statically not present versus ones whose >> > value is dynamic.) >> > >> > When this sdist is built, the resulting wheel MUST have metadata >> > which is identical to the metadata present in this file, except that >> > any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have >> > arbitrary values in the wheel. >> > >> > A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for >> > the package name or version (i.e., these must be given statically), >> > and these MUST match the {PACKAGE} and {VERSION} of the sdist as >> > described above. >> >> This seems pretty good at first reading. >> >> > [TBD: do we want to forbid the use of dynamic metadata for any >> > other fields? I assume PyPI will enforce some stricter rules at least, >> > but I don't know if we want to make that part of the spec, or just >> > part of PyPI's administrative rules.] >> >> This covers the main point of contention. It would be bad if build >> systems started using __SDIST_DYNAMIC__ just because "it's easier". >> >> Maybe add >> >> * A valid sdist SHOULD NOT use the __SDIST_DYNAMIC__ mechanism any >> more than necessary (i.e., if the metadata is the same in all >> generated wheels, it does not need to use the __SDIST_DYNAMIC__ >> mechanism, and so should not do so). >> >> > This is intentionally a close analogue of a wheel's ``.dist-info`` >> > directory; intention is that as future metadata standards are defined, >> > the specifications for the ``.sdist-info`` and ``.dist-info`` >> > directories will evolve in synchrony. >> > >> > >> > Evolutionary notes >> > ================== >> > >> > A goal here is to make it as simple as possible to convert old-style >> > sdists to new-style sdists. (E.g., this is one motivation for >> > supporting dynamic build requirements.) The ideal would be that there >> > would be a single static pypackage.cfg that could be dropped into any >> > "version 0" VCS checkout to convert it to the new shiny. This is >> > probably not 100% possible, but we can get close, and it's important >> > to keep track of how close we are... hence this section. >> > >> > A rough plan would be: Create a build system package >> > (``setuptools_pypackage`` or whatever) that knows how to speak >> > whatever hook language we come up with, and convert them into >> > setuptools calls. This will probably require some sort of hooking or >> > monkeypatching to setuptools to provide a way to extract the >> > ``setup_requires=`` argument when needed, and to provide a new version >> > of the sdist command that generates the new-style format. This all >> > seems doable and sufficient for a large proportion of packages (though >> > obviously we'll want to prototype such a system before we finalize >> > anything here). (Alternatively, these changes could be made to >> > setuptools itself rather than going into a separate package.) >> > >> > But there remain two obstacles that mean we probably won't be able to >> > automatically upgrade packages to the new format: >> > >> > 1) There currently exist packages which insist on particular packages >> > being available in their environment before setup.py is executed. This >> > means that if we decide to execute build scripts in an isolated >> > virtualenv-like environment, then projects will need to check whether >> > they do this, and if so then when upgrading to the new system they >> > will have to start explicitly declaring these dependencies (either via >> > ``setup_requires=`` or via static declaration in ``pypackage.cfg``). >> > >> > 2) There currently exist packages which do not declare consistent >> > metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different >> > ``install_requires=``). When upgrading to the new system, projects >> > will have to evaluate whether this applies to them, and if so they >> > will need to either stop doing that, or else add ``__SDIST_DYNAMIC__`` >> > annotations at appropriate places. >> > >> > We'll also presumably need some API for packages to describe which >> > parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for >> > the packages that need it (a new argument to ``setup()`` or some >> > setting in ``setup.cfg`` or something). >> >> I'm confused here. And it's just now become clear *why* I'm confused. >> >> The sdist format MUST be a generated format - i.e., we should insist >> (in principle at least) that it's only ever generated by tools. >> Otherwise it's way too easy for people to just zip up their source >> tree, hand craft something generic (that over-uses __SDIST_DYNAMIC__) >> and say "here's an sdist". Obviously, people always *can* manually >> create an sdist but we need to pin down the spec tightly, or we've not >> improved things. >> >> That's why I'm concerned about __SDIST_DYNAMIC__ and it's also what >> confuses me about the above transition plan. >> >> For people using setuptools currently, the transition should be simply >> that they upgrade setuptools, and the "setup.py sdist" command in the >> new setuptools generates the new sdist format. By default, the >> setuptools sdist process assumes everything is static and requires the >> user to modify the setup.py to explicitly mark which metadata they >> want to be left to build time. That way, we get a relatively >> transparent transition, while avoiding overuse of dynamic metadata. >> >> If setup.py has to explicitly mark dynamic metadata, that also allows >> us to reject attempts to make name and version dynamic. Which is good. >> >> Paul >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Oct 27 10:45:20 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Oct 2015 14:45:20 +0000 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On 27 October 2015 at 13:12, Daniel Holth wrote: > We must do the hard work to support Unicode file names, and spaces and > accent marks in home directory names (historically a problem on Windows), in > our packaging system. It is the right thing to do. It is not the publisher's > fault that your system has broken Unicode. In the examples I'm thinking of, the publisher used a format (.tar.gz) that didn't properly support Unicode, in the sense that it didn't include an encoding for the bytes it used to represent filenames. IMO, that is something we shouldn't allow, by rejecting file formats that don't support Unicode properly. Whose fault it is, is not important - it's just as easy to say that it's not the end user's fault that the publisher made an unwarranted assumption about encodings. What's important is that things work for everyone, and the interoperability standards don't leave room for people to make such assumptions. Paul PS Consider this a retraction of my suggestion that filenames in sdists should be pure ASCII. But still, sdists shouldn't contain files that can't be used on target systems - e.g., 2 files whose names differ only in case, files containing characters like :, ? or * that are invalid on Windows... Whether this needs to be noted in the standard, or whether it's just a case of directing users' bug reports back to the publisher, is an open question, though. From brett at python.org Tue Oct 27 12:45:15 2015 From: brett at python.org (Brett Cannon) Date: Tue, 27 Oct 2015 16:45:15 +0000 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Tue, 27 Oct 2015 at 02:17 Robert Collins wrote: > On 27 October 2015 at 21:47, David Cournapeau wrote: > > > Another simple solution for this particular case is to add conflict rules > > between packages that provide the same requirement (that's what php's > > composer do IIRC). > > > > The case of safety against malicious forks is handled quite explicitly in > > composer, we may want to look at how they do it when considering > solutions > > (e.g. https://github.com/composer/composer/issues/2690, though it has > > changed a bit since then) > > > > Adding the provides/conflict concepts to pip resolver will complexify it > > quite significantly, both in terms of running time complexity (since at > that > > point you are solving a NP-complete problem) and in terms of > implementation. > > But we also know for real cases this is doable, even in pure python > > (composer handles all the cases you are mentioning, and is in pure php). > > We already require a full NP-complete solver because of <, <= and ~ > version operators. > > I haven't adsorbed this proposal enough to comment on the reification > aspect yet. > > I'm worried about provides and conflicts in general, but not from a > resolver code perspective - thats a one-ish-time-cost, but instead > from a user experience perspective. > So from my perspective as someone who (I think) grasps what the problems that everyone is trying to solve is but not knowing enough to know how stuff is done now (all my projects on PyPI are pure Python), Nathaniel's proposal makes total sense to me. I would think it would be easy to explain to a scientist that "to get scipy, run `python3.5 -m pip install numpy`, but if you want fast over open source and use Intel's MKL library, do `python3.5 -m pip install numpy[mkl]`. I think the syntax clearly shows it's a modification/tweak/special version of numpy and it makes sense that I want to install something that provides numpy while relying on MKL. Nathaniel's comment about how this might actually give pip a leg up on conda also sounds nice to me as I have enough worry about having a fissure in 1D along the Python 2/3 line, and I'm constantly worried that the scientific community is going to riot and make it a 2D fissure along Python 2/3, pip/conda axes and split effort, documentation, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Oct 28 00:50:14 2015 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 27 Oct 2015 21:50:14 -0700 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: > > > Current draft text in rendered form at: > https://gist.github.com/rbtcollins/666c12aec869237f7cf7 > > Thanks for working on this. Overall I like the idea, but have some comments/questions 1) *Please*, *please*, *please* let's start doing PEP conversations as PRs to pypa/interoperability-peps : ) There's a place in there for unnumbered PEPs. Review will be better and faster IMO as PRs. If the PR gets too cluttered with old conversations, just start clean with a new PR. We can announce the PRs here, but don't discuss them on the mailing list. The same goes for trimming down PEP426 as you mention. Let's do that as a PR to pypa/interoperability-peps to the existing document. 2) Ditto on Ralf's concerns about readability. I only really understood it after seeing the examples you gave to Ralf (they need to be in the PEP, not just in response to Ralf). There's a few places I'd want to alter the wording, but I likely won't bother, unless it's done as a PR. 3) You refer to "source distribution". What does that mean exactly in this PEP? just the current setuptools notion? 4) Although using a process interface is not necessarily a problem, I don't agree with your point on why a python interface would be unworkable. You're assuming that pip would try to import all the build tools (for every dependency it's processing) in the master process. An alternative could be that pip would have it's own tool (that runs as a subprocess in an isolated env) that knows how to load and work with python build interfaces. You could argue that a python api is an advantage, because build tools aren't forced to grow a certain cli for pip, they just have to add a shim module that conforms to the interface. But my point is not to argue for the Python API, but for you to remove an argument that from what I can tell, isn't really an argument for one or the other. 5) at one pt you said "--dump-build-requires would just output a constant string: {"schema_version": "2.0", "build_requires": []}" you meant "metadata_version", right? 6) Can you explain better why we need to manage a pypa.yaml schema version *and* a build description schema version. Why not just a version for pypa.yaml, and if anything changes (in the yaml schema or the build "description" api), then just bump the version. 7) it's unclear when pip get's to run "dist-info" and when the result might be different. For example, we've discussed that run time dependencies may get clarifed *after* the build process.... so this command might produce different results at different times? --Marcus > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Wed Oct 28 01:02:51 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Oct 2015 16:02:51 +1100 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) References: Message-ID: <85h9lbmvpg.fsf_-_@benfinney.id.au> Marcus Smith writes: > 1) *Please*, *please*, *please* let's start doing PEP conversations as > PRs to pypa/interoperability-peps : ) Please keep the conversation on a mailing list where one can participate without needing to sign up with any particular service provider. Your proposal would have the effect of excluding people from the conversation if they don't agree to have a GitHub account. I think it's valuable to avoid that barrier to entry, needing only an email account. -- \ ?'Tis strange, ? but true; for truth is always strange; / | `\ Stranger than fiction.? ??Lord? George Gordon Noel Byron, _Don | _o__) Juan_ | Ben Finney From njs at pobox.com Wed Oct 28 01:03:59 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Oct 2015 22:03:59 -0700 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins wrote: > Since Nathaniel seems busy, I've taken the liberty of drafting a > narrow PEP based on the conversations that arose from the prior > discussion. > > It (naturally) has my unique flavor, but builds on the work Nathaniel > had put together, so I've put his name as a co-author even though he > hasn't seen a word of it until now :) - all errors and mistakes are > therefore mine... > > Current draft text in rendered form at: > https://gist.github.com/rbtcollins/666c12aec869237f7cf7 > > I've run it past Donald and he has a number of concerns - I think > we'll need to discuss them here, and possibly in another hangout, to > get a path forward. Now that I've had a chance to read it properly... First impression: there's surprisingly little overlap between this and my simultaneously-posted draft [1] -- my draft focuses on trying to only document the stuff that everyone seemed to agree on, includes a proposal for static metadata in sdists (since Donald seemed to be saying that he considered this a mandatory component of any proposal to update how sdists work), and tries to set out a blueprint for how to organize the remaining issues, whereas yours spends most of its time on the controversial details that I decided to skip over for this draft. Being an engineer, there are a number of these details that I'm tempted to quibble over :-), but I will restrain myself for now and wait first to see how the other thread develops and whether we do have general agreement on the points I wrote, since this seems like a complex enough topic that it could easily spiral become a conversational morass if we aren't careful to keep the different issues organized? -n [1] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html -- Nathaniel J. Smith -- http://vorpus.org From ralf.gommers at gmail.com Wed Oct 28 01:58:17 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 28 Oct 2015 06:58:17 +0100 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On Wed, Oct 28, 2015 at 6:03 AM, Nathaniel Smith wrote: > On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins > wrote: > > Since Nathaniel seems busy, I've taken the liberty of drafting a > > narrow PEP based on the conversations that arose from the prior > > discussion. > > > > It (naturally) has my unique flavor, but builds on the work Nathaniel > > had put together, so I've put his name as a co-author even though he > > hasn't seen a word of it until now :) - all errors and mistakes are > > therefore mine... > > > > Current draft text in rendered form at: > > https://gist.github.com/rbtcollins/666c12aec869237f7cf7 > > > > I've run it past Donald and he has a number of concerns - I think > > we'll need to discuss them here, and possibly in another hangout, to > > get a path forward. > > Now that I've had a chance to read it properly... > > First impression: there's surprisingly little overlap between this and > my simultaneously-posted draft [1] -- Which is good, double work has been kept to a minimum - it's like you two actually coordinated this:) > my draft focuses on trying to > only document the stuff that everyone seemed to agree on, includes a > proposal for static metadata in sdists (since Donald seemed to be > saying that he considered this a mandatory component of any proposal > to update how sdists work), and tries to set out a blueprint for how > to organize the remaining issues, whereas yours spends most of its > time on the controversial details that I decided to skip over for this > draft. > Imho they're not details. The controversial parts of your draft are still mostly in the metadata part. If you'd split your draft in two, then you'd see that the first one is pretty short and the second half of it is only TBDs. And those TBDs are exactly what Robert's draft fills in. @Robert: thanks for the example, very helpful. I'll look at it in more detail later. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 28 02:17:54 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 27 Oct 2015 23:17:54 -0700 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On Oct 27, 2015 10:58 PM, "Ralf Gommers" wrote: > > > > On Wed, Oct 28, 2015 at 6:03 AM, Nathaniel Smith wrote: >> >> On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins >> wrote: >> > Since Nathaniel seems busy, I've taken the liberty of drafting a >> > narrow PEP based on the conversations that arose from the prior >> > discussion. >> > >> > It (naturally) has my unique flavor, but builds on the work Nathaniel >> > had put together, so I've put his name as a co-author even though he >> > hasn't seen a word of it until now :) - all errors and mistakes are >> > therefore mine... >> > >> > Current draft text in rendered form at: >> > https://gist.github.com/rbtcollins/666c12aec869237f7cf7 >> > >> > I've run it past Donald and he has a number of concerns - I think >> > we'll need to discuss them here, and possibly in another hangout, to >> > get a path forward. >> >> Now that I've had a chance to read it properly... >> >> First impression: there's surprisingly little overlap between this and >> my simultaneously-posted draft [1] -- > > > Which is good, double work has been kept to a minimum - it's like you two actually coordinated this:) > >> >> my draft focuses on trying to >> only document the stuff that everyone seemed to agree on, includes a >> proposal for static metadata in sdists (since Donald seemed to be >> saying that he considered this a mandatory component of any proposal >> to update how sdists work), and tries to set out a blueprint for how >> to organize the remaining issues, whereas yours spends most of its >> time on the controversial details that I decided to skip over for this >> draft. > > > Imho they're not details. The controversial parts of your draft are still mostly in the metadata part. If you'd split your draft in two, then you'd see that the first one is pretty short and the second half of it is only TBDs. And those TBDs are exactly what Robert's draft fills in. Sure. When I say they're details, I don't mean they're unimportant; I just mean that after that last thread got so big and tangled, I don't want to spend time debating options for JSON formats if we don't yet have agreement on the big picture of what we're trying to do, or try to debate semantics and ipc conventions etc simultaneously. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 28 02:19:36 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 28 Oct 2015 19:19:36 +1300 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) In-Reply-To: <85h9lbmvpg.fsf_-_@benfinney.id.au> References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: On 28 October 2015 at 18:02, Ben Finney wrote: > Marcus Smith writes: > >> 1) *Please*, *please*, *please* let's start doing PEP conversations as >> PRs to pypa/interoperability-peps : ) > > Please keep the conversation on a mailing list where one can participate > without needing to sign up with any particular service provider. One has to sign up with the mailing list, so there's no functional difference there. > Your proposal would have the effect of excluding people from the > conversation if they don't agree to have a GitHub account. I think it's > valuable to avoid that barrier to entry, needing only an email account. They can always reply to emails. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 28 02:22:55 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 28 Oct 2015 19:22:55 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 28 October 2015 at 02:28, Daniel Holth wrote: > It looks like you would normally be able to say just 'flit' in the > bootstrap-requires or provided-by, unless the requirements were for build > code bundled inside the sdist. Right - you'd say flit in bootstrap-requires, and pip would cache based on the version of flit it chose to run. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 28 02:34:11 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 28 Oct 2015 19:34:11 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 28 October 2015 at 17:50, Marcus Smith wrote: >> >> Current draft text in rendered form at: >> https://gist.github.com/rbtcollins/666c12aec869237f7cf7 >> > > Thanks for working on this. > Overall I like the idea, but have some comments/questions > > > 1) *Please*, *please*, *please* let's start doing PEP conversations as PRs > to pypa/interoperability-peps : ) There's a place in there for unnumbered > PEPs. Review will be better and faster IMO as PRs. If the PR gets too > cluttered with old conversations, just start clean with a new PR. We can > announce the PRs here, but don't discuss them on the mailing list. The > same goes for trimming down PEP426 as you mention. Let's do that as a PR > to pypa/interoperability-peps to the existing document. https://github.com/pypa/interoperability-peps/pull/54 > 2) Ditto on Ralf's concerns about readability. I only really understood it > after seeing the examples you gave to Ralf (they need to be in the PEP, not > just in response to Ralf). There's a few places I'd want to alter the > wording, but I likely won't bother, unless it's done as a PR. I'll work on an iteration for that, the pr is just what I had in the gist. > 3) You refer to "source distribution". What does that mean exactly in this > PEP? just the current setuptools notion? yes. I don't change the definition of source distribution at all in this PEP. > 4) Although using a process interface is not necessarily a problem, I don't > agree with your point on why a python interface would be unworkable. You're > assuming that pip would try to import all the build tools (for every > dependency it's processing) in the master process. An alternative could be > that pip would have it's own tool (that runs as a subprocess in an isolated > env) that knows how to load and work with python build interfaces. You > could argue that a python api is an advantage, because build tools aren't That would mean that pip has to have the same exact version of it embedded in the environment that build tools will be run in, which it doesn't have today. I worry that that will be hard to manage, and hard for folk to debug. > forced to grow a certain cli for pip, they just have to add a shim module > that conforms to the interface. But my point is not to argue for the > Python API, but for you to remove an argument that from what I can tell, > isn't really an argument for one or the other. I'll review the text I have there, but the argument does seem valid to me. > 5) at one pt you said "--dump-build-requires would just output a constant > string: {"schema_version": "2.0", "build_requires": []}" you meant > "metadata_version", right? Yes, sorry. > 6) Can you explain better why we need to manage a pypa.yaml schema version > *and* a build description schema version. Why not just a version for > pypa.yaml, and if anything changes (in the yaml schema or the build > "description" api), then just bump the version. I see no problem with evolving them in lockstep, but because the documents are separate, we either have to infer the version of one, or we have to be explicit on each. I'd rather avoid the chance for a bug where something tries to parse a v2 schema build description with a v1 schema parser. Making them self identifying avoids that. > 7) it's unclear when pip get's to run "dist-info" and when the result might > be different. For example, we've discussed that run time dependencies may > get clarifed *after* the build process.... so this command might produce > different results at different times? Pip would run dist-info when determining the install-requires and extras for the package. That would be required to generate the actual final dependencies. It would only be run once I expect. I'll clarify that expected behaviour in the PEP. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 28 02:35:51 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 28 Oct 2015 19:35:51 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 28 October 2015 at 18:03, Nathaniel Smith wrote: > On Sun, Oct 25, 2015 at 11:01 PM, Robert Collins > wrote: ... >> I've run it past Donald and he has a number of concerns - I think >> we'll need to discuss them here, and possibly in another hangout, to >> get a path forward. > > Now that I've had a chance to read it properly... > > First impression: there's surprisingly little overlap between this and > my simultaneously-posted draft [1] -- my draft focuses on trying to > only document the stuff that everyone seemed to agree on, includes a > proposal for static metadata in sdists (since Donald seemed to be > saying that he considered this a mandatory component of any proposal > to update how sdists work), and tries to set out a blueprint for how > to organize the remaining issues, whereas yours spends most of its > time on the controversial details that I decided to skip over for this > draft. Right - I'm avoiding changing how sdists works at all: that is not necessary for introducing the ability to run arbitrary build tools. > Being an engineer, there are a number of these details that I'm > tempted to quibble over :-), but I will restrain myself for now and > wait first to see how the other thread develops and whether we do have > general agreement on the points I wrote, since this seems like a > complex enough topic that it could easily spiral become a > conversational morass if we aren't careful to keep the different > issues organized? Agreed, separate issues, separate threads :). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ben+python at benfinney.id.au Wed Oct 28 02:36:40 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 28 Oct 2015 17:36:40 +1100 Subject: [Distutils] Please don't impose additional barriers to participation References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: <85d1vzmrd3.fsf@benfinney.id.au> Robert Collins writes: > On 28 October 2015 at 18:02, Ben Finney wrote: > > Please keep the conversation on a mailing list where one can > > participate without needing to sign up with any particular service > > provider. > > One has to sign up with the mailing list, so there's no functional > difference there. You've stated that point in the past, and it's still untrue. The functional difference is that the mailing list is provided by effectively the same organisation that one is wanting to participating in discussion with. An objector to the terms of service of the mailing list is much less likely to want to also participate as part of this community. The same is certainly not true of GitHub. Please don't make a GitHub account a requirement for participating in this community's discussions on an equal footing with other participants. > > Your proposal would have the effect of excluding people from the > > conversation if they don't agree to have a GitHub account. I think > > it's valuable to avoid that barrier to entry, needing only an email > > account. > > They can always reply to emails. As first-class citizens, like today? Or as outsiders looking in? The proposal is to move the primary discussion forum *away* from this forum, and make GitHub participants first-class citizens of that forum. Either there is no significant advantage to having the discussion forum take place primarily on the walled garden of GitHub, in which case the proposal seems pointless on its face. Or there is some significant advantage being proposed, in which case my objections stand: don't restrict that advantage to only those who are willing to use a GitHub account to do so. -- \ ?If you go flying back through time and you see somebody else | `\ flying forward into the future, it's probably best to avoid eye | _o__) contact.? ?Jack Handey | Ben Finney From mal at egenix.com Wed Oct 28 04:31:28 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 28 Oct 2015 09:31:28 +0100 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: <85h9lbmvpg.fsf_-_@benfinney.id.au> References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: <563087E0.9050104@egenix.com> On 28.10.2015 06:02, Ben Finney wrote: > Marcus Smith writes: > >> 1) *Please*, *please*, *please* let's start doing PEP conversations as >> PRs to pypa/interoperability-peps : ) > > Please keep the conversation on a mailing list where one can participate > without needing to sign up with any particular service provider. > > Your proposal would have the effect of excluding people from the > conversation if they don't agree to have a GitHub account. I think it's > valuable to avoid that barrier to entry, needing only an email account. I agree with Ben. Discussions on PEPs need to happen on mailing lists, not hidden away on some issue tracker or PR ticket. PEP generally affect a large number of Python users, so it's vital to get as much feedback as possible. This is not possible using a PR approach. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From qwcode at gmail.com Wed Oct 28 03:30:37 2015 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 28 Oct 2015 00:30:37 -0700 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) In-Reply-To: <85h9lbmvpg.fsf_-_@benfinney.id.au> References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: my intention certainly wasn't to try to exclude anybody. for me, it's the practical matter of the PR UI being more effective than a mailing list thread (in this case referring to a gist), and that we can track proposals easier and link to them from issues (in that same repo) and other PyPA docs. also, to be clear, this isn't about thinking PyPA would sidestep the PEP approval process all together. It's about managing drafts, reviews and updates into that process. certainly, this isn't my decision alone to make this change... just stating my preference, and hoping to get others to agree. Part of my motivation for bringing it up, was do due to trying write a "PyPA Roadmap" recently for pypa.io, and wanting it to be easier to track and link to ideas that people are coming up with (ideas that don't immediately start as draft PEPs) --Marcus On Tue, Oct 27, 2015 at 10:02 PM, Ben Finney wrote: > Marcus Smith writes: > > > 1) *Please*, *please*, *please* let's start doing PEP conversations as > > PRs to pypa/interoperability-peps : ) > > Please keep the conversation on a mailing list where one can participate > without needing to sign up with any particular service provider. > > Your proposal would have the effect of excluding people from the > conversation if they don't agree to have a GitHub account. I think it's > valuable to avoid that barrier to entry, needing only an email account. > > -- > \ ?'Tis strange, ? but true; for truth is always strange; / | > `\ Stranger than fiction.? ??Lord? George Gordon Noel Byron, _Don | > _o__) Juan_ | > Ben Finney > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 28 04:39:52 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2015 09:39:52 +0100 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: On 28 Oct 2015 09:32, "Marcus Smith" wrote: > > my intention certainly wasn't to try to exclude anybody. for me, it's the practical matter of the PR UI being more effective than a mailing list thread (in this case referring to a gist), and that we can track proposals easier and link to them from issues (in that same repo) and other PyPA docs. > > also, to be clear, this isn't about thinking PyPA would sidestep the PEP approval process all together. It's about managing drafts, reviews and updates into that process. > > certainly, this isn't my decision alone to make this change... just stating my preference, and hoping to get others to agree. Part of my motivation for bringing it up, was do due to trying write a "PyPA Roadmap" recently for pypa.io, and wanting it to be easier to track and link to ideas that people are coming up with (ideas that don't immediately start as draft PEPs) I'd certainly like us to move line by line comments (etc) to the interoperability PEPs repo. Overall design discussions would stay here on the mailing list, and if folks submitting PEPs don't want to create a GitHub account, they can send them here and we'll create the PR on their behalf. If python-dev ends up adopting GitLab for the main PEPs repo, then we should be able to move the whole process there, rather than needing to maintain a separate copy. Cheers, Nick. > > --Marcus > > > On Tue, Oct 27, 2015 at 10:02 PM, Ben Finney wrote: >> >> Marcus Smith writes: >> >> > 1) *Please*, *please*, *please* let's start doing PEP conversations as >> > PRs to pypa/interoperability-peps : ) >> >> Please keep the conversation on a mailing list where one can participate >> without needing to sign up with any particular service provider. >> >> Your proposal would have the effect of excluding people from the >> conversation if they don't agree to have a GitHub account. I think it's >> valuable to avoid that barrier to entry, needing only an email account. >> >> -- >> \ ?'Tis strange, ? but true; for truth is always strange; / | >> `\ Stranger than fiction.? ??Lord? George Gordon Noel Byron, _Don | >> _o__) Juan_ | >> Ben Finney >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 28 04:43:49 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2015 09:43:49 +0100 Subject: [Distutils] New PyPUG Tutorials In-Reply-To: References: Message-ID: On 26 Oct 2015 01:10, "Paul Moore" wrote: > > On 24 October 2015 at 20:25, Marcus Smith wrote: > > As someone who has slowly chipped away on the PyPUG for the last 2 years, > > I'm excited about the prospect of being able to make a big jump in quality > > very quickly. > > Agreed - this is great news :-) Huzzah! Cheers, Nick. P.S. I probably should have mentioned previously that I'm travelling at the moment, so may be slow to respond to email threads until Nov 8 or so. Trying to catch up on distutils-sig specifically this morning :) > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 28 04:54:33 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Oct 2015 08:54:33 +0000 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 28 October 2015 at 06:34, Robert Collins wrote: >> 1) *Please*, *please*, *please* let's start doing PEP conversations as PRs >> to pypa/interoperability-peps : ) There's a place in there for unnumbered >> PEPs. Review will be better and faster IMO as PRs. If the PR gets too >> cluttered with old conversations, just start clean with a new PR. We can >> announce the PRs here, but don't discuss them on the mailing list. The >> same goes for trimming down PEP426 as you mention. Let's do that as a PR >> to pypa/interoperability-peps to the existing document. > > https://github.com/pypa/interoperability-peps/pull/54 I'm concerned that if the discussion all ends up on the PR, then people (like me) who follow the mailing list but aren't watching the PR discussion[1] will miss important chunks of the debate. I guess I could subscribe to the PR discussion, but the typical lack of quoting in PR conversations makes it much harder to follow threads there. I'm happy to see the reference versions and change history of PEPs in the repository, but can we not keep discussions of the principles on the public list? Otherwise, we'll have a first run of the discussion on github, then a rerun when it finally gets posted to distutils-sig... Paul [1] I'm not sure why, as part of the PyPA group on github, I don't receive notifications when PRs like this one are added. Do I need to subscribe to something that I haven't? From ncoghlan at gmail.com Wed Oct 28 05:04:48 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2015 10:04:48 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On 27 Oct 2015 13:30, "Daniel Holth" wrote: > > Mr. grumpy pants, > > Now you're trolling us by describing flit. I'd managed to miss Flit's creation, so I simply wasn't aware of it. Now that I've had a chance to remedy that oversight, yes, flit sounds exactly like what I meant, so it could be worth recommending it ahead of a full distutils/setuptools based setup.py for simple projects. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 28 05:08:10 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 28 Oct 2015 02:08:10 -0700 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 3:43 AM, Paul Moore wrote: > > On 26 October 2015 at 06:04, Nathaniel Smith wrote: > > Here's a second round of text towards making a build-system > > independent interface between pip and source trees/sdists. My idea > > this time is to take a divide-and-conquer approach: this text tries to > > summarize all the stuff that it seemed like we had mostly reached > > consensus on in the previous thread + call, with blank chunks marked > > "TBD" where there are specific points that still need To Be > > Determined. So my hope is that everyone will read what's here and > > agree that it's great as far as it goes, and then we can go through > > and fill in each missing piece one at a time. > > I'll comment on what's here, but ignore the TBD items - I'd rather (as > you suggest) leave discussion of those details till the basic idea is > agreed. > > > Abstract > > ======== > > > > Distutils delenda est. > > While this makes a nice tagline, I'd rather something less negative. > Distutils does not "need" to be destroyed. It's perfectly adequate > (although hardly user friendly) for a lot of cases - I'd be willing to > suggest *most* users can work just fine with distutils. > > I'm not a fan of distutils, but I'd prefer it if we kept the rhetoric > limited - as Nick pointed out this whole area is as much a political > issue as a technical one. > > > Extended abstract > > ================= > > > > While ``distutils`` / ``setuptools`` have taken us a long way, they > > suffer from three serious problems: (a) they're missing important > > features like autoconfiguration and usable build-time dependency > > declaration, (b) extending them is quirky, complicated, and fragile, > > (c) it's very difficult to use anything else, because they provide the > > standard interface for installing python packages expected by both > > users and installation tools like ``pip``. > > Again, this is overstated. You very nearly lost me right here - people > won't read the details of the proposal if they disagree with the > abstract(s). Specifically: > > * The features in (a) are only important to *some* parts of the > community. The scientific community is the major one, and is a huge > influence over the direction we want to go in, but again, not crucial > to many people. And even where they might be useful (e.g., Windows > users building pyyaml, lxml, pillow, ...) the description implies > "working out what's there" rather than "allowing users to easily > manage non-Python dependencies", which gives the wrong impression. > > * The features in (b) are highly specialised. Very few people extend > setuptools/distutils. And those who do, have often invested a lot of > effort in doing so. Sure, they'd rather not have needed to, but now > that they have, a replacement system simply means that work is lost. > Arguably, fixing (b) is only useful for people (like the scientific > community) who have needed to extend setuptools and have been unable > to achieve their goals that way. That's an even smaller part of the > community. > > > Previous efforts (e.g. distutils2 or setuptools itself) have attempted > > to solve problems (a) and/or (b). We propose to solve (c). > > Agreed - this is a good approach. But it's at odds with your abstract, > which says distutils must die. Here you're saying you want to allow > people to keep using distutils but allow people with specialised needs > to choose an alternative. Or are you offering an alternative to people > who use distutils? > > The whole of the above is confusing on the face of it. The details > below clarify a lot, as does knowing how the previous discussions have > gone. But it would help a lot if the introduction to this PEP were > clearer. Fair enough, I'll dial it back. :-) My personal prediction is that within a year of this support becoming widespread, we'll see build systems that are just better than distutils on all axes for all projects, not just the ones with weird specialised needs -- AFAICT the distutils architectures has remained basically unchanged since Python 2.0, and we've gained a bit more experience with Python packaging in the last 15 years :-). But yeah, sure, if you think it'll bother people then there's no point in that. > > The goal of this PEP is get distutils-sig out of the business of being > > a gatekeeper for Python build systems. If you want to use distutils, > > great; if you want to use something else, then that should be easy to > > do using standardized methods. The difficulty of interfacing with > > distutils means that there aren't many such systems right now, but to > > give a sense of what we're thinking about see `flit > > `_ or `bento > > `_. Fortunately, wheels have now > > solved many of the hard problems here -- e.g. it's no longer necessary > > that a build system also know about every possible installation > > configuration -- so pretty much all we really need from a build system > > is that it have some way to spit out standard-compliant wheels. > > OK. Although I see a risk here that if I want to build package FOO, I > now have to worry whether FOO's build system supports Windows, as well > as worrying whether FOO itself supports Windows. > > There's still a role for some "gatekeeper" (not a good word IMO, maybe > "coordinator") to provide a certain level of support or review of > build systems, and a point of contact for users with build issues (the > point of this proposal is to some extent that people don't need to > *know* what build system a project uses, so suggesting everyone has to > direct issues to the correct build system support forum isn't > necessarily practical). I see what you mean, but I don't think there's much that can or should be done about it in the form of a PEP? I assume that what will happen is that if you can't build a package, you'll file a bug with the maintainers of that package, and then it's their job to figure out whether to patch around the issue locally, file their own bug upstream with whatever build system package they're using, switch to a new build system, or whatever. I think we can generally trust individual projects and the community at large to figure out what the trade-offs between different systems are, once the different systems start existing. Though it may well make sense for the PyPA packaging guide to add a set of best-practice guidelines for build system implementors. > > We therefore propose a new, relatively minimal interface for > > installation tools like ``pip`` to interact with package source trees > > and source distributions. > > > > In addition, we propose a wheel-inspired static metadata format for > > sdists, suitable for tools like PyPI and pip's resolver. > > > > > > Terminology and goals > > ===================== > > > > A *source tree* is something like a VCS checkout. We need a standard > > interface for installing from this format, to support usages like > > ``pip install some-directory/``. > > > > A *source distribution* is a static snapshot representing a particular > > release of some source code, like ``lxml-3.4.4.zip``. Source > > distributions serve many purposes: they form an archival record of > > releases, they provide a stupid-simple de facto standard for tools > > that want to ingest and process large corpora of code, possibly > > written in many languages (e.g. code search), they act as the input to > > downstream packaging systems like Debian/Fedora/Conda/..., and so > > forth. In the Python ecosystem they additionally have a particularly > > important role to play, because packaging tools like ``pip`` are able > > to use source distributions to fulfill binary dependencies, e.g. if > > there is a distribution ``foo.whl`` which declares a dependency on > > ``bar``, then we need to support the case where ``pip install bar`` or > > ``pip install foo`` automatically locates the sdist for ``bar``, > > downloads it, builds it, and installs the resulting package. > > This is somewhat misleading, given that you go on to specify the > format below, but maybe that's only an issue for someone like me who > saw the previous debate over "source distribution" (as a bundled up > source tree) vs "sdist" as a specified format. If I understand, you've > now discarded the former sense of source distribution, and are > sticking with the latter (specified format) definition. The "sdists" in this draft try to compromise between the various concepts that were proposed in the previous thread: you can generally treat them like bundled up source trees (they have a single directory that unpacks into something that's laid out similarly to a VCS checkout), but they also contain additional static metadata to make PyPI and pip happy (or at least, as much static metadata as they can). > > Source distributions are also known as "sdists" for short. > > > > > > Source trees > > ============ > > > > We retroactively declare the legacy source tree format involving > > ``setup.py`` to be "version 0". We don't try to specify it further; > > its de facto specification is encoded in the source code and > > documentation of ``distutils``, ``setuptools``, ``pip``, and other > > tools. > > > > A "version 1" (or greater) source tree is any directory which contains > > a file named ``pypackage.cfg``, which will -- in some manner whose > > details are TBD -- describe the package's build dependencies and how > > to invoke the build system. This mechanism: > > > > - Will allow for both static and dynamic specification of build dependencies > > > > - Will have some degree of isolation of different builds from each > > other, so that it will be possible for a single run of pip to install > > one package that build-depends on ``foo = 1.1`` and another package > > that build-depends on ``foo = 1.2``. > > All good so far. > > > - Will leave the actual installation of the package in the hands of > > the build/installation tool (i.e. individual package build systems > > will not need to know about things like --user versus --global or make > > decisions about when and how to modify .pth files) > > This seems completely backwards to me. It's pip's job to do the actual > install. The build tool should *only* focus on generating standard > conforming binary wheels - otherwise what's the point of the > separation of concerns that wheels provide? > > Or maybe I'm confused by the term "build/installation tool" - by that > did you actually mean pip, rather than the build system? Yeah, I was just unclear here -- the "build/installation tool" was supposed to be pip (because pip installs packages! ...and also builds them), as contrasted with the "individual package build systems" which don't know anything about installing. I'll reword. This bullet point is rather substantive, actually, since if adopted then it rules out the proposed semantics for the "develop" operation in Robert's PEP. (In current pip and in his proposal, "pip install -e" is unlike regular "pip install", in that "pip install -e" doesn't actually install anything, it just calls "setup.py develop", which does the actual installation. One consequence of this AFAICT is that if you try passing any of the standard installation target options to "pip install -e", like "--target" or whatever, then it blows up...) > (TBDs omitted) > > > Source distributions > > ==================== > > > > [possibly this should get split off into a separate PEP, but I'll keep > > it together for now for ease of discussion] > > > > A "version 1" (or greater) source distribution is a file meeting the > > following criteria: > > > > - It MUST have a name of the form: {PACKAGE}-{VERSION}.{EXT}, where > > {PACKAGE} is the package name, {VERSION} is a PEP 440-compliant > > version number, and {EXT} is a compliant archive format. > > > > The set of compliant archive formats is: zip, [TBD] > > > > [QUESTION: should we continue to allow .tar.gz and friends? In > > practice by "allow" I mean something like "accept new-style sdists on > > PyPI in this format". I'm inclined not to -- zip is the most > > universally supported format around, it allows file-based random > > access (unlike tar-based things) which is useful for pulling out > > metadata without decompressing the whole thing, and standardizing on > > one format dodges distracting and pointless discussions about which > > format to use, i.e. it's TOOWTDI-compliant. Of course pip is free to > > continue to support other archive formats when passed explicitly on > > the command line. Any objections?] > > +1 on having a single archive format, and zip seems like the best choice. > > > Similar to wheels, the archive is Unicode, and the filenames inside > > the archive are encoded in UTF-8. > > This isn't the job of the sdist format to specify. It should be > implicit in the choice of archive format. There's a silly typo in the quoted line -- it was supposed to read: Similar to wheels, the archive *filename* is Unicode, and the filenames inside the archive are encoded in UTF-8. These two points were just lifted from PEP 427 without thinking about it too much -- see https://www.python.org/dev/peps/pep-0427/#id12 Now that I reread that section of PEP 427, the underscore replacement probably makes sense for sdists as well. > Having said that, I'd go with > > 1. The sdist filename MUST support the full range of package names as > specified in PEP 426 (https://www.python.org/dev/peps/pep-0426/#name) > and versions as in PEP 440 > (https://www.python.org/dev/peps/pep-0440/). That's actually far less > than full Unicode. > 2. The archive format MUST support arbitrary Unicode filenames. That > means zip is OK, but tar.gz isn't unless you specify UTF-8 is used > (the tar format doesn't allow for an encoding declaration - see > https://docs.python.org/3.5/library/tarfile.html#tar-unicode for > details on Unicode issues in the tar format). > > Having said that I'd also go with "filenames in the archive SHOULD be > limited to ASCII" - because we have had issues with pip where test > files have Unicode filenames, and builds break because they get > mangled on systems with weird encoding setups... IIRC, these are > typically related to .tar.gz sdists, which (due to the lack of > encoding support) result in files being unpacked with the wrong names. > So maybe if we enforce zip format we don't need to add this > limitation. Especially if we go with zip as the one true archive format, then I think we should just use the same rules for all this stuff as wheels do. No need to re-invent the... well, you know. > > - When unpacked, it MUST contain a single directory directory tree > > named ``{PACKAGE}-{VERSION}``. > > > > - This directory tree MUST be a valid version 1 (or greater) source > > tree as defined above. > > > > - It MUST additionally contain a directory named > > ``{PACKAGE}-{VERSION}.sdist-info`` (notice the ``s``), with the > > following contents: > > > > - ``SDIST``: Mandatory. Same record-oriented format as a wheel's > > ``WHEEL`` file, but with different fields:: > > > > SDist-Version: 1.0 > > Generator: setuptools sdist 20.1 > > > > ``SDist-Version`` is the version number of this specification. > > Software that processes sdists should warn if ``SDist-Version`` is > > greater than the version it supports, and must fail if > > ``SDist-Version`` has a greater major version than the version it > > supports. > > > > ``Generator`` is the name and optionally the version of the > > software that produced the archive. > > > > - ``RECORD``: Mandatory. A list of all files contained in the sdist > > (except for the RECORD file itself and any signature files) together > > with their hashes, as specified in PEP 427. > > > > - ``RECORD.jws``, ``RECORD.p7s``: Optional. Signature files as > > specified in PEP 427. > > > > - ``METADATA``: Mandatory. Metadata version 1.1 or greater format > > metadata, with an additional rule that fields may contain the special > > sentinel value ``__SDIST_DYNAMIC__``, which indicates that the value > > of this field cannot be determined until build time. If a "multiple > > use field" is present with the value ``__SDIST_DYNAMIC__``, then this > > field MUST occur exactly once, e.g.:: > > > > # Okay: > > Requires-Dist: lxml (> 3.3) > > Requires-Dist: requests > > > > # no Requires-Dist lines at all is okay > > # (meaning: this package's requirements are the empty set) > > > > # Okay, requirements will be determined at build time: > > Requires-Dist: __SDIST_DYNAMIC__ > > > > # NOT okay: > > Requires-Dist: lxml (> 3.3) > > Requires-Dist: __SDIST_DYNAMIC__ > > > > (The use of a special token allows us to distinguish between > > multiple use fields whose value is statically the empty list versus > > one whose value is dynamic; it also allows us to distinguish between > > optional fields which are statically not present versus ones whose > > value is dynamic.) > > > > When this sdist is built, the resulting wheel MUST have metadata > > which is identical to the metadata present in this file, except that > > any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have > > arbitrary values in the wheel. > > > > A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for > > the package name or version (i.e., these must be given statically), > > and these MUST match the {PACKAGE} and {VERSION} of the sdist as > > described above. > > This seems pretty good at first reading. > > > [TBD: do we want to forbid the use of dynamic metadata for any > > other fields? I assume PyPI will enforce some stricter rules at least, > > but I don't know if we want to make that part of the spec, or just > > part of PyPI's administrative rules.] > > This covers the main point of contention. It would be bad if build > systems started using __SDIST_DYNAMIC__ just because "it's easier". > > Maybe add > > * A valid sdist SHOULD NOT use the __SDIST_DYNAMIC__ mechanism any > more than necessary (i.e., if the metadata is the same in all > generated wheels, it does not need to use the __SDIST_DYNAMIC__ > mechanism, and so should not do so). > > > This is intentionally a close analogue of a wheel's ``.dist-info`` > > directory; intention is that as future metadata standards are defined, > > the specifications for the ``.sdist-info`` and ``.dist-info`` > > directories will evolve in synchrony. > > > > > > Evolutionary notes > > ================== > > > > A goal here is to make it as simple as possible to convert old-style > > sdists to new-style sdists. (E.g., this is one motivation for > > supporting dynamic build requirements.) The ideal would be that there > > would be a single static pypackage.cfg that could be dropped into any > > "version 0" VCS checkout to convert it to the new shiny. This is > > probably not 100% possible, but we can get close, and it's important > > to keep track of how close we are... hence this section. > > > > A rough plan would be: Create a build system package > > (``setuptools_pypackage`` or whatever) that knows how to speak > > whatever hook language we come up with, and convert them into > > setuptools calls. This will probably require some sort of hooking or > > monkeypatching to setuptools to provide a way to extract the > > ``setup_requires=`` argument when needed, and to provide a new version > > of the sdist command that generates the new-style format. This all > > seems doable and sufficient for a large proportion of packages (though > > obviously we'll want to prototype such a system before we finalize > > anything here). (Alternatively, these changes could be made to > > setuptools itself rather than going into a separate package.) > > > > But there remain two obstacles that mean we probably won't be able to > > automatically upgrade packages to the new format: > > > > 1) There currently exist packages which insist on particular packages > > being available in their environment before setup.py is executed. This > > means that if we decide to execute build scripts in an isolated > > virtualenv-like environment, then projects will need to check whether > > they do this, and if so then when upgrading to the new system they > > will have to start explicitly declaring these dependencies (either via > > ``setup_requires=`` or via static declaration in ``pypackage.cfg``). > > > > 2) There currently exist packages which do not declare consistent > > metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different > > ``install_requires=``). When upgrading to the new system, projects > > will have to evaluate whether this applies to them, and if so they > > will need to either stop doing that, or else add ``__SDIST_DYNAMIC__`` > > annotations at appropriate places. > > > > We'll also presumably need some API for packages to describe which > > parts of the METADATA file should be marked ``__SDIST_DYNAMIC__``, for > > the packages that need it (a new argument to ``setup()`` or some > > setting in ``setup.cfg`` or something). > > I'm confused here. And it's just now become clear *why* I'm confused. > > The sdist format MUST be a generated format - i.e., we should insist > (in principle at least) that it's only ever generated by tools. > Otherwise it's way too easy for people to just zip up their source > tree, hand craft something generic (that over-uses __SDIST_DYNAMIC__) > and say "here's an sdist". Obviously, people always *can* manually > create an sdist but we need to pin down the spec tightly, or we've not > improved things. The mandatory RECORD file makes it pretty much impossible to generate an sdist manually. > That's why I'm concerned about __SDIST_DYNAMIC__ and it's also what > confuses me about the above transition plan. > > For people using setuptools currently, the transition should be simply > that they upgrade setuptools, and the "setup.py sdist" command in the > new setuptools generates the new sdist format. By default, the > setuptools sdist process assumes everything is static and requires the > user to modify the setup.py to explicitly mark which metadata they > want to be left to build time. That way, we get a relatively > transparent transition, while avoiding overuse of dynamic metadata. My assumption was that when a project flips the switch to move to the new format (not sure what that switch looks like, but presumably we will have one), then one of the things that happens is that "setup.py sdist" starts running the equivalent of "egg_info" and stuffing all the resulting metadata into {PACKAGE}-{VERSION}.sdist-info/ (along with generating a RECORD file etc.). So the default would be to assume all metadata is static. But right now that is not actually true for all projects (for both good and bad reasons), so this means that before they flip that switch they need to either adjust their setup.py to make it true, or else they need to use some new API that setuptools will add to let them specify which fields should be marked as dynamic. This API will be purely a setuptools-internal thing, though, nothing that the PEP itself needs to concern itself with. > If setup.py has to explicitly mark dynamic metadata, that also allows > us to reject attempts to make name and version dynamic. Which is good. Presumably PyPI will also reject packages with dynamic names or versions, so any build system that tries to get away with this will quickly realize the error of their ways. -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Wed Oct 28 05:18:23 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 28 Oct 2015 02:18:23 -0700 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 7:00 AM, David Cournapeau wrote: > > > On Tue, Oct 27, 2015 at 1:12 PM, Daniel Holth wrote: >> >> The drawback of .zip is file size since it compresses each file >> individually rather than giving the compression algorithm a larger input, >> it's a great format otherwise. Ubiquitous including Apple iOS packages, >> Java, word processor file formats. And most Python packages are small. > > > I don't really buy the indexing advantages, especially w/ the current > implementation of zipfile in python (e.g. loading the whole set of archives > at creation time) Can you elaborate about what you mean? AFAICT from a quick skim of the source code, zipfile does eagerly read in the table of contents for the zip file (i.e., it reads out the list of files and their metadata), but no actual files are decompressed until you ask for them individually, and when you do request a specific file then it can be accessed in O(1) time. This is really different from .tar.gz, where you have to decompress the entire archive just to get a list of files, and then you need to decompress the whole thing again each time you want to access a single file inside. (Regarding the size thing, yeah, .tar.gz is smaller, and .tar.bz2 smaller than that, and .tar.xz smaller again, ... but this doesn't strike me as an argument for throwing up our hands and leaving the choice to individual projects, because it's not like they know what the optimal trade-off is either. IMO we should pick one, and zip is Good Enough.) -n -- Nathaniel J. Smith -- http://vorpus.org From ncoghlan at gmail.com Wed Oct 28 06:24:58 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Oct 2015 11:24:58 +0100 Subject: [Distutils] Removing the aspirational aspects of PEP 426 Message-ID: Caught up on distutils-sig this morning - I don't have any additional comments on Robert's and Nathaniel's draft PEPs except to say "I really like the look of where they're going, and look forward to reading the next iterations on them that take into account the feedback already received" :) The idea of treating extras as subdistributions with their own wheel metadata also sounds promising (and analogous to building multiple binary RPMs from a single source RPM). >From an sdist metadata perspective, though, I think the right thing to do is to descope PEP 426 to just the stuff we *need* for the build system improvements, and defer everything else (e.g. JSON-LD, SPDX, richer dependency semantics, etc) to a future metadata 3.0 proposal (or potentially metadata extensions, or 2.x format updates). The one major enhancement I think would be worth keeping is the metadata extension system (including mandatory extension support), since that provides a way to experiment with ideas that we may later standardise in 3.0. I'm not sure when I'd have time to work on that myself though, so I'm definitely open to expressions of interest in taking a hatchet to the PEP. Nathaniel, Robert, perhaps you'd be interested in that as part of the build interface standardisation? Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Oct 28 06:39:00 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 28 Oct 2015 03:39:00 -0700 Subject: [Distutils] Removing the aspirational aspects of PEP 426 In-Reply-To: References: Message-ID: On Oct 28, 2015 3:25 AM, "Nick Coghlan" wrote: > [...] > From an sdist metadata perspective, though, I think the right thing to do is to descope PEP 426 to just the stuff we *need* for the build system improvements, and defer everything else (e.g. JSON-LD, SPDX, richer dependency semantics, etc) to a future metadata 3.0 proposal (or potentially metadata extensions, or 2.x format updates). I think PEP 426 is actually orthogonal to these proposals. AFAICT, the only reason Robert's PEP as written requires PEP 426 is that he needs a standard serializable format to list dependencies... but he actually defines such a format about 10 lines above for the static bootstrap-requirements key, i.e. a list of specifier strings. So it actually makes more sense to use that for dynamic requirements too for internal consistency, and leave PEP 426 out of it. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Oct 28 08:32:15 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 28 Oct 2015 12:32:15 +0000 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: Nathaniel, I'm not sure what the software is supposed to do with fine grained dynamic metadata that would make very much sense to the end user. I think you could probably get away with a single flag Dynamic: true / false. Iff true, pip runs the dist-info command after installing bootstrap dependencies. You could still complain if the name & version changed. Of course in a VCS checkout or during development you probably always want the regenerate-metadata behavior. On Wed, Oct 28, 2015 at 5:18 AM Nathaniel Smith wrote: > On Tue, Oct 27, 2015 at 7:00 AM, David Cournapeau > wrote: > > > > > > On Tue, Oct 27, 2015 at 1:12 PM, Daniel Holth wrote: > >> > >> The drawback of .zip is file size since it compresses each file > >> individually rather than giving the compression algorithm a larger > input, > >> it's a great format otherwise. Ubiquitous including Apple iOS packages, > >> Java, word processor file formats. And most Python packages are small. > > > > > > I don't really buy the indexing advantages, especially w/ the current > > implementation of zipfile in python (e.g. loading the whole set of > archives > > at creation time) > > Can you elaborate about what you mean? AFAICT from a quick skim of the > source code, zipfile does eagerly read in the table of contents for > the zip file (i.e., it reads out the list of files and their > metadata), but no actual files are decompressed until you ask for them > individually, and when you do request a specific file then it can be > accessed in O(1) time. This is really different from .tar.gz, where > you have to decompress the entire archive just to get a list of files, > and then you need to decompress the whole thing again each time you > want to access a single file inside. > > (Regarding the size thing, yeah, .tar.gz is smaller, and .tar.bz2 > smaller than that, and .tar.xz smaller again, ... but this doesn't > strike me as an argument for throwing up our hands and leaving the > choice to individual projects, because it's not like they know what > the optimal trade-off is either. IMO we should pick one, and zip is > Good Enough.) > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Oct 28 10:55:51 2015 From: cournape at gmail.com (David Cournapeau) Date: Wed, 28 Oct 2015 14:55:51 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote: > > On 27 Oct 2015 13:30, "Daniel Holth" wrote: > > > > Mr. grumpy pants, > > > > Now you're trolling us by describing flit. > > I'd managed to miss Flit's creation, so I simply wasn't aware of it. > > Now that I've had a chance to remedy that oversight, yes, flit sounds > exactly like what I meant, so it could be worth recommending it ahead of a > full distutils/setuptools based setup.py for simple projects. > IMO, flit should be officially recommended until it at least supports building sdist. Not having this feature is a significant pain point for downstream distributors, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 28 10:57:01 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 28 Oct 2015 10:57:01 -0400 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On October 28, 2015 at 10:55:54 AM, David Cournapeau (cournape at gmail.com) wrote: > On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote: > > > > > On 27 Oct 2015 13:30, "Daniel Holth" wrote: > > > > > > Mr. grumpy pants, > > > > > > Now you're trolling us by describing flit. > > > > I'd managed to miss Flit's creation, so I simply wasn't aware of it. > > > > Now that I've had a chance to remedy that oversight, yes, flit sounds > > exactly like what I meant, so it could be worth recommending it ahead of a > > full distutils/setuptools based setup.py for simple projects. > > > > IMO, flit should be officially recommended until it at least supports > building sdist. Not having this feature is a significant pain point for > downstream distributors, > Assuming you meant ?shouldn?t? I agree. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Wed Oct 28 11:41:03 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 28 Oct 2015 15:41:03 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: Well it sounds like we're ready to actually fix that, by defining a reasonable universal Python sdist format. On Wed, Oct 28, 2015 at 10:57 AM Donald Stufft wrote: > On October 28, 2015 at 10:55:54 AM, David Cournapeau (cournape at gmail.com) > wrote: > > On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote: > > > > > > > > On 27 Oct 2015 13:30, "Daniel Holth" wrote: > > > > > > > > Mr. grumpy pants, > > > > > > > > Now you're trolling us by describing flit. > > > > > > I'd managed to miss Flit's creation, so I simply wasn't aware of it. > > > > > > Now that I've had a chance to remedy that oversight, yes, flit sounds > > > exactly like what I meant, so it could be worth recommending it ahead > of a > > > full distutils/setuptools based setup.py for simple projects. > > > > > > > IMO, flit should be officially recommended until it at least supports > > building sdist. Not having this feature is a significant pain point for > > downstream distributors, > > > > > Assuming you meant ?shouldn?t? I agree. > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at gedmin.as Wed Oct 28 13:08:12 2015 From: marius at gedmin.as (Marius Gedminas) Date: Wed, 28 Oct 2015 19:08:12 +0200 Subject: [Distutils] Environment markers: ready for prime time? Message-ID: <20151028170812.GC28934@platonas> Hi! pip 7 enables wheel caching by default, which is good. Wheel caching means you can't compute dynamic dependencies any more -- i.e. things like setup( ... install_requires=[...] + (['enum34'] if sys.version_info[0] < 3 else []), ...) will cause problems. As far as I understand, you're supposed to use environment markers instead. Problem 1: where's the documentation? E.g. https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/ has no mention of the word "marker". Try to google "setuptools environment marker" (and how is a user going to discover the magical keyword they need to google is "environment marker")? Try to find anything resembling documentation on the 1st page. The best I could find was http://docs.openstack.org/developer/pbr/#environment-markers which only works if you use pbr. Even the spec (https://www.python.org/dev/peps/pep-0426/#environment-markers) only shows how the markers are supposed to appear in the JSON metadata. No clue is provided for poor writers of setup.py files. I somehow discovered the syntax once (I don't remember how -- most likely kind people in #pypa spoon-fed me), but now I'm cargo-culting my existing setup.py files that already use environment markers. Problem 2: what are the minimum versions of the tools that your users must have before you can rely on environment markers? - setuptools >= 0.7 ("Added experimental environment marker support") - wheel >= 0.24 (if you have wheel 0.23 or older, environment markers are silently broken and have fun figuring out why: https://github.com/pypa/pip/issues/2870). - does the pip version matter at all? I think not; please correct me if I'm wrong. Some official answers from the hard-working PyPA visionaries would be welcome. Marius Gedminas -- I once held a little hand That made my sad heart sing. Twas the loveliest hand I'd ever held, Four Aces and a King -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 173 bytes Desc: Digital signature URL: From dholth at gmail.com Wed Oct 28 13:36:38 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 28 Oct 2015 17:36:38 +0000 Subject: [Distutils] Environment markers: ready for prime time? In-Reply-To: <20151028170812.GC28934@platonas> References: <20151028170812.GC28934@platonas> Message-ID: bdist_wheel predates marker support in setuptools and didn't support pulling markers from setup.py until 0.24.0, as noted in the wheel release notes. Older versions of bdist_wheel could read conditional requirements from a section in setup.cfg but could not translate markers from setuptools' requires.txt. Wheel's own source code demonstrates the syntax. pip uses pkg_resources to parse dependencies for wheels and an old enough pip doesn't use wheels at all. On Wed, Oct 28, 2015 at 1:15 PM Marius Gedminas wrote: > Hi! > > pip 7 enables wheel caching by default, which is good. > > Wheel caching means you can't compute dynamic dependencies any more -- > i.e. things like > > setup( > ... > install_requires=[...] + (['enum34'] if sys.version_info[0] < 3 else > []), > ...) > > will cause problems. > > As far as I understand, you're supposed to use environment markers > instead. > > > Problem 1: where's the documentation? E.g. > https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/ > has no mention of the word "marker". > > Try to google "setuptools environment marker" (and how is a user going > to discover the magical keyword they need to google is "environment > marker")? Try to find anything resembling documentation on the 1st > page. The best I could find was > http://docs.openstack.org/developer/pbr/#environment-markers which only > works if you use pbr. > > Even the spec ( > https://www.python.org/dev/peps/pep-0426/#environment-markers) > only shows how the markers are supposed to appear in the JSON metadata. > No clue is provided for poor writers of setup.py files. > > I somehow discovered the syntax once (I don't remember how -- most > likely kind people in #pypa spoon-fed me), but now I'm cargo-culting my > existing setup.py files that already use environment markers. > > > Problem 2: what are the minimum versions of the tools that your users > must have before you can rely on environment markers? > > - setuptools >= 0.7 ("Added experimental environment marker support") > > - wheel >= 0.24 (if you have wheel 0.23 or older, environment markers are > silently broken and have fun figuring out why: > https://github.com/pypa/pip/issues/2870). > > - does the pip version matter at all? I think not; please correct me if > I'm wrong. > > > Some official answers from the hard-working PyPA visionaries would be > welcome. > > Marius Gedminas > -- > I once held a little hand > That made my sad heart sing. > Twas the loveliest hand I'd ever held, > Four Aces and a King > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 28 14:08:10 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 28 Oct 2015 20:08:10 +0200 Subject: [Distutils] Environment markers: ready for prime time? In-Reply-To: <20151028170812.GC28934@platonas> References: <20151028170812.GC28934@platonas> Message-ID: On Wed, Oct 28, 2015 at 7:08 PM, Marius Gedminas wrote: > - does the pip version matter at all? I think not; please correct me if > I'm wrong. > ?From what I see, pip only supports env markers in sdists since 6.0 ( https://github.com/pypa/pip/pull/2153) so I guess one can only count on env marker suport since 6.0, right?? How much usage do the pre-6.0 pip versions get? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 28 14:27:09 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Wed, 28 Oct 2015 20:27:09 +0200 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: <563087E0.9050104@egenix.com> References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On Wed, Oct 28, 2015 at 10:31 AM, M.-A. Lemburg wrote: > I agree with Ben. Discussions on PEPs need to happen on mailing lists, > not hidden away on some issue tracker or PR ticket. > ?I think some balance is needed here. Every so often discussions on the mailing list get so disorganized and slide into so many irrelevant things I can't afford to participate?. I think the PR thing should at least be tried and MAL/Ben haven't provided any pertinent arguments against making a github account anyway. ?Plus the PyPA shown willingness to accommodate feedback from the mailinglist?. I'd expect draft updates will be posted once in a while on distutils-sig for people who don't have a web browser or github account. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Wed Oct 28 14:33:06 2015 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Wed, 28 Oct 2015 13:33:06 -0500 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: <563087E0.9050104@egenix.com> References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On Wed, Oct 28, 2015 at 3:31 AM, M.-A. Lemburg wrote: > On 28.10.2015 06:02, Ben Finney wrote: >> Marcus Smith writes: >> >>> 1) *Please*, *please*, *please* let's start doing PEP conversations as >>> PRs to pypa/interoperability-peps : ) >> >> Please keep the conversation on a mailing list where one can participate >> without needing to sign up with any particular service provider. >> >> Your proposal would have the effect of excluding people from the >> conversation if they don't agree to have a GitHub account. I think it's >> valuable to avoid that barrier to entry, needing only an email account. > > I agree with Ben. Discussions on PEPs need to happen on mailing lists, > not hidden away on some issue tracker or PR ticket. Others may be willing to tolerate your FUD, but without concrete reasons against GitHub (other than zomg it's a proprietary service) I don't see a reason to not use the pull request flow on an open repository that is free for people to clone, fork, contribute to, etc. GitHub isn't my preferred hosting platform for git but it is the defacto standard and it's workflow ubiquitous, documented, and far more user-friendly than mailing list threads (especially when they devolve into ideology wars). Also nothing precludes mailing list discussions, so without details about your objections, I don't see why this should be held up. From p.f.moore at gmail.com Wed Oct 28 14:42:01 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Oct 2015 18:42:01 +0000 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On 28 October 2015 at 18:33, Ian Cordasco wrote: > Also nothing precludes mailing list discussions, so without details > about your objections, I don't see why this should be held up. One immediate question - not at all FUD, nor intended as provocative, it's a genuine question. I thought right now I'd check if there had actually been any discussion on the PR that I'd missed [1]. But I don't have a link to the specific tracker issue, and offhand I can't recall the repo name. So how do I find the issue quickly? For discussions here, I can scan the list of message subjects in the distutils-sig folder in my email. Quickly checking on a tracker PR is harder, and so I'm *much* less likely to contribute. Result - when the PEP is posted back to the list once it's been thrashed out on the tracker, there's a *second* round of time-consuming debate as people who missed the first round join in with issues that have possibly already been discussed. I assume there's no intention of PEPs being discussed and approved, *all* without debate on distutils-sig? Paul [1] If I'm supposed to be getting notifications for comments on the PR (as a member of the PyPA group, shouldn't I be?) then it's not happening... I know I can subscribe to the PR, but I'm not clear why I should need to - I don't for pip issues, for instance... From p.f.moore at gmail.com Wed Oct 28 14:34:59 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Oct 2015 18:34:59 +0000 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On 28 October 2015 at 18:27, Ionel Cristian M?rie? wrote: > On Wed, Oct 28, 2015 at 10:31 AM, M.-A. Lemburg wrote: >> >> I agree with Ben. Discussions on PEPs need to happen on mailing lists, >> not hidden away on some issue tracker or PR ticket. > > > I think some balance is needed here. Every so often discussions on the > mailing list get so disorganized and slide into so many irrelevant things I > can't afford to participate. I think the PR thing should at least be tried > and MAL/Ben haven't provided any pertinent arguments against making a github > account anyway. > > Plus the PyPA shown willingness to accommodate feedback from the > mailinglist. I'd expect draft updates will be posted once in a while on > distutils-sig for people who don't have a web browser or github account. Note that I also prefer substantive discussions on the mailing list. I'm happy to have simple edits (typos, wording clarifications, etc) covered on the PR tracker, but for anything significant around the intent of the PEP, motivating use cases, design decisions, etc, I find the github tracker interface to be pretty awful. The lack of quoting in typical discussions makes it harder to follow threads, the need to switch from my mail client (where notifications will appear) to the github interface if I want to contribute in the "correct" form (reply by email works, but doesn't integrate cleanly in the "style" of the tracker discussion for anything but short interjections) makes it harder to contribute, and the fact that it's less conducive to "lurking" than a mailing list are concrete reasons, not related to wanting to use a github account, for preferring the mailing list. Call me an old fogey for preferring last year's technology if you must :-) Paul From mal at egenix.com Wed Oct 28 14:42:38 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 28 Oct 2015 19:42:38 +0100 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: <5631171E.6020808@egenix.com> On 28.10.2015 19:33, Ian Cordasco wrote: > On Wed, Oct 28, 2015 at 3:31 AM, M.-A. Lemburg wrote: >> On 28.10.2015 06:02, Ben Finney wrote: >>> Marcus Smith writes: >>> >>>> 1) *Please*, *please*, *please* let's start doing PEP conversations as >>>> PRs to pypa/interoperability-peps : ) >>> >>> Please keep the conversation on a mailing list where one can participate >>> without needing to sign up with any particular service provider. >>> >>> Your proposal would have the effect of excluding people from the >>> conversation if they don't agree to have a GitHub account. I think it's >>> valuable to avoid that barrier to entry, needing only an email account. >> >> I agree with Ben. Discussions on PEPs need to happen on mailing lists, >> not hidden away on some issue tracker or PR ticket. > > Others may be willing to tolerate your FUD, but without concrete > reasons against GitHub (other than zomg it's a proprietary service) I > don't see a reason to not use the pull request flow on an open > repository that is free for people to clone, fork, contribute to, etc. > > GitHub isn't my preferred hosting platform for git but it is the > defacto standard and it's workflow ubiquitous, documented, and far > more user-friendly than mailing list threads (especially when they > devolve into ideology wars). > > Also nothing precludes mailing list discussions, so without details > about your objections, I don't see why this should be held up. Ok, as first step, please change your tone and reread my reply. I'm not willing to tolerate your tone. Thanks. The argument here is not about Github or not, and it's not FUD. It's the same argument as is used when starting into discussions on the Python issue tracker and the reason for often moving those to the python-dev or -ideas mailing lists. Hiding away discussions is not the way to go about when discussing PEPs. Using PRs is fine when it comes to improving the language of the PEP or adding new aspects, but the reasoning for applying these changes should be based on the results of discussions on the relevant lists, distutils-sig in this case. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 28 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2015-10-23: Released mxODBC Connect 2.1.5 ... http://egenix.com/go85 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From donald at stufft.io Wed Oct 28 14:44:05 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 28 Oct 2015 14:44:05 -0400 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On October 28, 2015 at 2:42:19 PM, Paul Moore (p.f.moore at gmail.com) wrote: > > [1] If I'm supposed to be getting notifications for comments on the PR > (as a member of the PyPA group, shouldn't I be?) then it's not > happening... I know I can subscribe to the PR, but I'm not clear why I > should need to - I don't for pip issues, for instance... You?re probably not directly subscribed to that repo, there should be a follow button on?https://github.com/pypa/interoperability-peps. You got it automatically for pip because you were explicitly added as a committer on the pip repo. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Wed Oct 28 15:05:45 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 28 Oct 2015 19:05:45 +0000 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On Wed, Oct 28, 2015 at 11:41 AM Daniel Holth wrote: > Well it sounds like we're ready to actually fix that, by defining a > reasonable universal Python sdist format. > > On Wed, Oct 28, 2015 at 10:57 AM Donald Stufft wrote: > >> On October 28, 2015 at 10:55:54 AM, David Cournapeau (cournape at gmail.com) >> wrote: >> > On Wed, Oct 28, 2015 at 9:04 AM, Nick Coghlan wrote: >> > >> > > >> > > On 27 Oct 2015 13:30, "Daniel Holth" wrote: >> > > > >> > > > Mr. grumpy pants, >> > > > >> > > > Now you're trolling us by describing flit. >> > > >> > > I'd managed to miss Flit's creation, so I simply wasn't aware of it. >> > > >> > > Now that I've had a chance to remedy that oversight, yes, flit sounds >> > > exactly like what I meant, so it could be worth recommending it ahead >> of a >> > > full distutils/setuptools based setup.py for simple projects. >> > > >> > >> > IMO, flit should be officially recommended until it at least supports >> > building sdist. Not having this feature is a significant pain point for >> > downstream distributors, >> > >> >> >> Assuming you meant ?shouldn?t? I agree. >> > I feel like you are suggesting that the person who spent their time developing some software and then gave it to away, and then chose a build system that was convenient for them, is causing problems. We can prevent them from causing those problems by asking them to use the sdist file format. How much pain would we be looking at with accepting a github archive of a tag as the official source? Something like an extra debhelper script per build system? -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 28 15:10:29 2015 From: donald at stufft.io (Donald Stufft) Date: Wed, 28 Oct 2015 15:10:29 -0400 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On October 28, 2015 at 3:05:57 PM, Daniel Holth (dholth at gmail.com) wrote: > I feel like you are suggesting that the person who spent their time > developing some software and then gave it to away, and then chose a build > system that was convenient for them, is causing problems. We can prevent > them from causing those problems by asking them to use the sdist file > format. How much pain would we be looking at with accepting a github > archive of a tag as the official source? Something like an extra debhelper > script per build system? > No, I?m suggesting that the PyPA needs to be conservative with it?s recommendations and that we need to consider the broader ecosystem impact. Individual projects or people may have different criteria they use to determine what is an acceptable solution, but we have to walk the fine line between trying to do what?s best for many different groups of people (and we sometimes have to make things worse for X to make it better for Y). I don?t have any problems with flit, or if someone wants to use it. I have a problem with making it an official recommendation prior to the foundations being laid to make it what I consider to be a reasonable, ecosystem wide, solution. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Wed Oct 28 16:12:11 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Oct 2015 20:12:11 +0000 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On 28 October 2015 at 18:44, Donald Stufft wrote: > On October 28, 2015 at 2:42:19 PM, Paul Moore (p.f.moore at gmail.com) wrote: >> >> [1] If I'm supposed to be getting notifications for comments on the PR >> (as a member of the PyPA group, shouldn't I be?) then it's not >> happening... I know I can subscribe to the PR, but I'm not clear why I >> should need to - I don't for pip issues, for instance... > > You?re probably not directly subscribed to that repo, there should be a follow button on https://github.com/pypa/interoperability-peps. You got it automatically for pip because you were explicitly added as a committer on the pip repo. No subscribe button that I can see on the repo. I could subscribe on a per-issue basis, but that's a bit different. Note - I'm actually not sure I *want* to be subscribing to issue updates like this in any case - it'd be more email filters to file the messages, more management of email groups, etc. If people *really* want to discuss on the PR, why not subscribe distutils-sig to the PR, so that list members can contribute to the discussion as well? (That's actually probably a bad idea, because the differences between mailing list etiquette and PR discussion etiquette are pretty big - but that's sort of my point, I *prefer* the mailing list format for PEP discussions, and I don't think I'm the only one...) Paul From ralf.gommers at gmail.com Wed Oct 28 16:15:52 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 28 Oct 2015 21:15:52 +0100 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On Wed, Oct 28, 2015 at 9:12 PM, Paul Moore wrote: > On 28 October 2015 at 18:44, Donald Stufft wrote: > > On October 28, 2015 at 2:42:19 PM, Paul Moore (p.f.moore at gmail.com) > wrote: > >> > >> [1] If I'm supposed to be getting notifications for comments on the PR > >> (as a member of the PyPA group, shouldn't I be?) then it's not > >> happening... I know I can subscribe to the PR, but I'm not clear why I > >> should need to - I don't for pip issues, for instance... > > > > You?re probably not directly subscribed to that repo, there should be a > follow button on https://github.com/pypa/interoperability-peps. You got > it automatically for pip because you were explicitly added as a committer > on the pip repo. > > No subscribe button that I can see on the repo. I could subscribe on a > per-issue basis, but that's a bit different. > It's the "Watch" button on the right top of https://github.com/pypa/interoperability-peps Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Oct 28 18:22:18 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Oct 2015 22:22:18 +0000 Subject: [Distutils] Please don't impose additional barriers to participation In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> <563087E0.9050104@egenix.com> Message-ID: On 28 October 2015 at 20:15, Ralf Gommers wrote: > It's the "Watch" button on the right top of > https://github.com/pypa/interoperability-peps Doh. Thanks. Paul From ralf.gommers at gmail.com Wed Oct 28 19:30:50 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 29 Oct 2015 00:30:50 +0100 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 5:45 PM, Brett Cannon wrote: > > Nathaniel's comment about how this might actually give pip a leg up on > conda also sounds nice to me as I have enough worry about having a fissure > in 1D along the Python 2/3 line, and I'm constantly worried that the > scientific community is going to riot and make it a 2D fissure along Python > 2/3, pip/conda axes and split effort, documentation, etc. > If it helps you sleep: I'm confident that no one is planning this particular riot. It takes little work to support pip and conda - the hard issues are mostly with building, not installing. Smaller riots like breaking ``python setup.py install`` recommending ``pip install .`` instead[1] are in the cards though:) Ralf [1] http://article.gmane.org/gmane.comp.python.numeric.general/61757 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 28 19:52:05 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Oct 2015 00:52:05 +0100 Subject: [Distutils] Removing the aspirational aspects of PEP 426 In-Reply-To: References: Message-ID: On 28 Oct 2015 11:39, "Nathaniel Smith" wrote: > > On Oct 28, 2015 3:25 AM, "Nick Coghlan" wrote: > > > [...] > > > From an sdist metadata perspective, though, I think the right thing to do is to descope PEP 426 to just the stuff we *need* for the build system improvements, and defer everything else (e.g. JSON-LD, SPDX, richer dependency semantics, etc) to a future metadata 3.0 proposal (or potentially metadata extensions, or 2.x format updates). > > I think PEP 426 is actually orthogonal to these proposals. AFAICT, the only reason Robert's PEP as written requires PEP 426 is that he needs a standard serializable format to list dependencies... but he actually defines such a format about 10 lines above for the static bootstrap-requirements key, i.e. a list of specifier strings. So it actually makes more sense to use that for dynamic requirements too for internal consistency, and leave PEP 426 out of it. If you can avoid blocking on metadata 2.0 entirely, that's even better :) Cheers, Nick. > > -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 28 20:00:56 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Oct 2015 01:00:56 +0100 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On 28 Oct 2015 10:08, "Nathaniel Smith" wrote: >Though it may well make sense for > the PyPA packaging guide to add a set of best-practice guidelines for > build system implementors. What would be really nice is if the new specification came with a behavioural test suite (e.g. defined with a project like behave) that a build system developer or project owner could run to check all relevant pip commands are handled correctly for a given package. A spec can be ambiguous, but a behavioural test suite sets a minimum bar that build systems can readily check themselves against. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 28 20:25:24 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 29 Oct 2015 02:25:24 +0200 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On Mon, Oct 26, 2015 at 8:04 AM, Nathaniel Smith wrote: > [TBD: should builds be performed in a fully isolated environment, or > should they get access to packages that are already installed in the > target install environment? The former simplifies a number of things, > but Robert was skeptical it would be possible.] > ?Would be wasteful not to allow it. But then, some people might want isolation. Whom should be able to switch isolation on? Users? Publishers?? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 28 20:28:52 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Oct 2015 01:28:52 +0100 Subject: [Distutils] Time for a setuptools_lite?? In-Reply-To: References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: On 28 Oct 2015 20:10, "Donald Stufft" wrote: > > No, I?m suggesting that the PyPA needs to be conservative with it?s recommendations and that we need to consider the broader ecosystem impact. Individual projects or people may have different criteria they use to determine what is an acceptable solution, but we have to walk the fine line between trying to do what?s best for many different groups of people (and we sometimes have to make things worse for X to make it better for Y). Right. While consuming prebuilt binaries is an acceptable approach in many cases, it's not acceptable for Linux distros since it can easily lead to problems with things like reproducible builds and licensing compliance (especially for copyleft licenses with on demand source provision requirements). That said, for Fedora RPMs, we don't need sdist per se - we only need ready access to the "original sources". It's just that wheels don't count, since they're a nominally binary format, and "C ABI = None, Platform = None" in the filename isn't currently an entirely reliable indicator of a pure Python wheel file. (Even if those markers *were* entirely reliable, the "certain kinds of wheel files can be treated as if they were source archives" gets really messy conceptual - it's akin to the "8-bit text is an ill-defined subset of arbitrary 8-bit data" represented by the Python 2 str type). A HTTPS source control URL together with a commit hash *can* count, though, even in the absence of a source archive uploaded to PyPI, as the VCS information is enough for us to retrieve the original sources and put them in the SRPM. (I'm not as familiar with Debian's policies as I am with those for Fedora et al, but as far as I am aware, they want a reference to the original sources for similar reasons of build reproducibility, license compliance, and code auditability). > I don?t have any problems with flit, or if someone wants to use it. I have a problem with making it an official recommendation prior to the foundations being laid to make it what I consider to be a reasonable, ecosystem wide, solution. Right, any recommended solution needs to provide access to the original sources *in addition to* any already built wheel files. While Linux distros could technically cope with arbitrary source trees, I think we're close enough to having a cleaner sdist format defined that we can wait until it's defined and flit produces it in addition to a wheel file before switching the official recommendation for new users. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Wed Oct 28 20:30:32 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 29 Oct 2015 02:30:32 +0200 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: On Mon, Oct 26, 2015 at 8:04 AM, Nathaniel Smith wrote: > When this sdist is built, the resulting wheel MUST have metadata > which is identical to the metadata present in this file, except that > any fields with value ``__SDIST_DYNAMIC__`` in the sdist may have > arbitrary values in the wheel. > > A valid sdist MUST NOT use the ``__SDIST_DYNAMIC__`` mechanism for > the package name or version (i.e., these must be given statically), > and these MUST match the {PACKAGE} and {VERSION} of the sdist as > described above. > > [TBD: do we want to forbid the use of dynamic metadata for any > other fields? I assume PyPI will enforce some stricter rules at least, > but I don't know if we want to make that part of the spec, or just > part of PyPI's administrative rules.] > ?Unless I misunderstood the core goal of this new sdist (to be able to know the dependencies statically), it doesn't make sense to allow mixing things. Is there an usecase for dynamic requirements?? In that situation users can just as well use the current sdist format. There are no advantages to using the new sdist format if your requirements are dynamic right? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 28 20:40:15 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Oct 2015 01:40:15 +0100 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On 29 Oct 2015 00:31, "Ralf Gommers" wrote: > > > > On Tue, Oct 27, 2015 at 5:45 PM, Brett Cannon wrote: >> >> >> Nathaniel's comment about how this might actually give pip a leg up on conda also sounds nice to me as I have enough worry about having a fissure in 1D along the Python 2/3 line, and I'm constantly worried that the scientific community is going to riot and make it a 2D fissure along Python 2/3, pip/conda axes and split effort, documentation, etc. > > If it helps you sleep: I'm confident that no one is planning this particular riot. It takes little work to support pip and conda - the hard issues are mostly with building, not installing. Last time I checked "pip in a conda env" was also pretty well behaved, so conda seems to be settling in fairly well to being a per-user cross platform alternative to apt, yum/dnf, homebrew, nix, etc, rather than tackling the same Python specific niche as pip & virtualenv. > Smaller riots like breaking ``python setup.py install`` recommending ``pip install .`` instead[1] are in the cards though:) Given the PyPA panel at PyCon US a few years ago ended up being subtitled "'./setup.py install' must die", I'd be surprised if that provoked a riot. I guess even if it does, you'll have plenty of folks prepared to help with crowd control :) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 28 20:57:37 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 29 Oct 2015 13:57:37 +1300 Subject: [Distutils] Environment markers: ready for prime time? In-Reply-To: <20151028170812.GC28934@platonas> References: <20151028170812.GC28934@platonas> Message-ID: On 29 October 2015 at 06:08, Marius Gedminas wrote: > Hi! > > pip 7 enables wheel caching by default, which is good. > > Wheel caching means you can't compute dynamic dependencies any more -- > i.e. things like > > setup( > ... > install_requires=[...] + (['enum34'] if sys.version_info[0] < 3 else []), > ...) > > will cause problems. > > As far as I understand, you're supposed to use environment markers > instead. > > > Problem 1: where's the documentation? E.g. > https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/ > has no mention of the word "marker". > > Try to google "setuptools environment marker" (and how is a user going > to discover the magical keyword they need to google is "environment > marker")? Try to find anything resembling documentation on the 1st > page. The best I could find was > http://docs.openstack.org/developer/pbr/#environment-markers which only > works if you use pbr. > > Even the spec (https://www.python.org/dev/peps/pep-0426/#environment-markers) > only shows how the markers are supposed to appear in the JSON metadata. > No clue is provided for poor writers of setup.py files. > > I somehow discovered the syntax once (I don't remember how -- most > likely kind people in #pypa spoon-fed me), but now I'm cargo-culting my > existing setup.py files that already use environment markers. > > > Problem 2: what are the minimum versions of the tools that your users > must have before you can rely on environment markers? > > - setuptools >= 0.7 ("Added experimental environment marker support") > > - wheel >= 0.24 (if you have wheel 0.23 or older, environment markers are > silently broken and have fun figuring out why: > https://github.com/pypa/pip/issues/2870). > > - does the pip version matter at all? I think not; please correct me if > I'm wrong. > > > Some official answers from the hard-working PyPA visionaries would be welcome. Not an official, but ... https://github.com/pypa/pip/issues/3111 https://bugs.launchpad.net/pbr/+bug/1502692 may give some useful info for you. Don't forget that markers are broken vis-a-vis Python 2.7.10, which we need to fix asap. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Wed Oct 28 21:09:15 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 29 Oct 2015 14:09:15 +1300 Subject: [Distutils] Removing the aspirational aspects of PEP 426 In-Reply-To: References: Message-ID: On 28 October 2015 at 23:39, Nathaniel Smith wrote: > On Oct 28, 2015 3:25 AM, "Nick Coghlan" wrote: >> > [...] >> From an sdist metadata perspective, though, I think the right thing to do >> is to descope PEP 426 to just the stuff we *need* for the build system >> improvements, and defer everything else (e.g. JSON-LD, SPDX, richer >> dependency semantics, etc) to a future metadata 3.0 proposal (or potentially >> metadata extensions, or 2.x format updates). > > I think PEP 426 is actually orthogonal to these proposals. AFAICT, the only > reason Robert's PEP as written requires PEP 426 is that he needs a standard > serializable format to list dependencies... but he actually defines such a > format about 10 lines above for the static bootstrap-requirements key, i.e. > a list of specifier strings. So it actually makes more sense to use that for > dynamic requirements too for internal consistency, and leave PEP 426 out of > it. pip requires: - distribution name - install_requires [+extras] today. It will want external dependencies in future (and in the spec I put forward build dependencies would be obtained earlier so could be skipped). I'd rather not invent a *new* format for handling both of these, but i'm ok if Donald and Nick specifically are. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From njs at pobox.com Wed Oct 28 21:16:16 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 28 Oct 2015 18:16:16 -0700 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Wed, Oct 28, 2015 at 4:30 PM, Ralf Gommers wrote: > > On Tue, Oct 27, 2015 at 5:45 PM, Brett Cannon wrote: >> >> >> Nathaniel's comment about how this might actually give pip a leg up on >> conda also sounds nice to me as I have enough worry about having a fissure >> in 1D along the Python 2/3 line, and I'm constantly worried that the >> scientific community is going to riot and make it a 2D fissure along Python >> 2/3, pip/conda axes and split effort, documentation, etc. > > > If it helps you sleep: I'm confident that no one is planning this particular > riot. It takes little work to support pip and conda - the hard issues are > mostly with building, not installing. Well.... I wouldn't say "no one". You weren't there at the NumPy BoF at SciPy this year, where a substantial portion of the room started calling for exactly this, and I felt pretty alone up front trying to squash it almost singlehandedly. It was a bit awkward actually! The argument for numpy dropping pip support is actually somewhat compelling. It goes like this: conda users don't care if numpy breaks ABI, because conda already enforces that numpy-C-API-using-packages have to be recompiled every time a new numpy release comes out. Therefore, if we only supported conda, then we would be free to break ABI and clean up some of the 20 year old broken junk that we have lying around and add new features more quickly. Conclusion: continuing to support pip is hobbling innovation in the whole numerical ecosystem. IMO this is not compelling *enough* to cut off our many many users who are not using conda, plus a schism like this would have all kinds of knock-on costs (the value of a community grows like O(n**2), so splitting a community is expensive!). And given that you and I are both on the list of gatekeepers to such a change, yeah, it's not going to happen in the immediate future. But... if conda continues to gain mindshare at pip's expense, and they fix some of the more controversial sticking points (e.g. the current reliance on secret proprietary build recipes), and the pip/distutils side of things continues to stagnate WRT things like this... I dunno, I could imagine that argument becoming more and more compelling over the next few years. At that point I'm honestly not sure what happens, but I suspect that all the options are unpleasant. You and I have a fair amount of political capital, but it is finite. ...Or maybe I'm worrying over nothing and everything would be fine, but still, it'd be nice if we never have to find out because pip etc. get better enough that the issue goes away. What I'm saying is, it's not a coincidence that it was after SciPy this year that I finally subscribed to distutils-sig :-). -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Wed Oct 28 21:28:34 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 28 Oct 2015 18:28:34 -0700 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Wed, Oct 28, 2015 at 5:40 PM, Nick Coghlan wrote: > > On 29 Oct 2015 00:31, "Ralf Gommers" wrote: >> >> >> >> On Tue, Oct 27, 2015 at 5:45 PM, Brett Cannon wrote: >>> >>> >>> Nathaniel's comment about how this might actually give pip a leg up on >>> conda also sounds nice to me as I have enough worry about having a fissure >>> in 1D along the Python 2/3 line, and I'm constantly worried that the >>> scientific community is going to riot and make it a 2D fissure along Python >>> 2/3, pip/conda axes and split effort, documentation, etc. >> >> If it helps you sleep: I'm confident that no one is planning this >> particular riot. It takes little work to support pip and conda - the hard >> issues are mostly with building, not installing. > > Last time I checked "pip in a conda env" was also pretty well behaved, so > conda seems to be settling in fairly well to being a per-user cross platform > alternative to apt, yum/dnf, homebrew, nix, etc, rather than tackling the > same Python specific niche as pip & virtualenv. I'm not confident I understand all the details of how conda works these days, but AFAIK using pip in a conda env is pretty much the equivalent of using 'sudo pip' in RH/Fedora, i.e. it will happily stomp all over the same files that the package manager thinks it is managing, and probably you will get away with it (until you don't). I could be wrong, though. >> Smaller riots like breaking ``python setup.py install`` recommending >> ``pip install .`` instead[1] are in the cards though:) > > Given the PyPA panel at PyCon US a few years ago ended up being subtitled > "'./setup.py install' must die", I'd be surprised if that provoked a riot. I > guess even if it does, you'll have plenty of folks prepared to help with > crowd control :) I think the biggest pushback so far is from Debian, b/c Debian has standard distro-wide build scripts for python packages that have 'setup.py install' baked in, and there is perhaps some political delicacy to convincing them they should have 'pip install' baked in instead. In the short run I'm guessing we'll end up placating them by giving them an override envvar that lets them keep using setup.py install, but in the longer run this might be a good place to consider directing some crowd control / persuasion. -n -- Nathaniel J. Smith -- http://vorpus.org From dholth at gmail.com Wed Oct 28 22:51:00 2015 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Oct 2015 02:51:00 +0000 Subject: [Distutils] flit adapter for the current sdist format Message-ID: OK, so I wanted to see what it would take to create current-format sdists, defined as anything that has a setup.py that pip can install by running said setup.py. The following script takes a flit package (in the current directory and builds a pip-installable sdist. Tested against https://github.com/pohmelie/aioftp https://gist.github.com/dholth/e17f243085dcc539813f The first script is the archiver and the second script is the setup.py embedded into the sdist by the archiver. The embedded setup.py does an isolated install of flit into the setup-requires directory, if required, and uses flit to implement bdist_wheel. The trick is that the only thing pip really needs from egg_info is the current requirements, easily generated from wheel-style .dist-info. It might need slightly more metadata to pass on pypi and the argument handling is inflexible, but it demonstrates most of what would be required for a backwards compatible shim. If our new sdist format is an even easier way to adapt new build systems than writing this trivial setup.py shim, then we might have something. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Thu Oct 29 00:12:49 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 29 Oct 2015 15:12:49 +1100 Subject: [Distutils] Time for a setuptools_lite?? References: <5627C546.40004@ronnypfannschmidt.de> Message-ID: <858u6mmhxa.fsf@benfinney.id.au> Nick Coghlan writes: > That said, for Fedora RPMs, we don't need sdist per se - we only need > ready access to the "original sources". There's a temporal element to that, too. While ?ready access? to the source might be clear enough at the moment of the wheel's release, it is less clear in ten years time when the original source for a package still in Fedora is needed again. In practice, a URL to (what one hopes is) the source is not enough to provude assurance the source will be available long in the future. Only a known tarball (or equivalent fixed single-file archive form) of the actual source is going to provide that. > It's just that wheels don't count, since they're a nominally binary > format, and "C ABI = None, Platform = None" in the filename isn't > currently an entirely reliable indicator of a pure Python wheel file. One needn't say that wheel is ?nominally? anything; it is sufficient to ask ?is this what a recipient would need to have the source in a form suitable for further modification and redistribution?? A wheel distribution is fairly clearly *not* the preferred form of the work for a recipient to have to exercise freedom to modify and redistribute. The wheel distgribution was generated from source files edited by the developer, and so *those* file are the source form of the work, the wheel distribution is not. > A HTTPS source control URL together with a commit hash *can* count, > though, even in the absence of a source archive uploaded to PyPI, as > the VCS information is enough for us to retrieve the original sources > and put them in the SRPM. As I point out above, that's only reliably true if it is immediately turned into a more reliably-archived form than a URL to some hosting provider somewhere. So, in practice, the URL is not enough to provide good assurance one has the source form of the work for redistribution. > (I'm not as familiar with Debian's policies as I am with those for > Fedora et al, but as far as I am aware, they want a reference to the > original sources for similar reasons of build reproducibility, license > compliance, and code auditability). Not merely a reference, but (as explained above) The actual source form of the work, persistent over long periods of time as the known corresponding source form of that version of the work. > Right, any recommended solution needs to provide access to the > original sources *in addition to* any already built wheel files. Yes, thanks. -- \ ?Try adding ?as long as you don't breach the terms of service ? | `\ according to our sole judgement? to the end of any cloud | _o__) computing pitch.? ?Simon Phipps, 2010-12-11 | Ben Finney From p.f.moore at gmail.com Thu Oct 29 05:18:02 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Oct 2015 09:18:02 +0000 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On 29 October 2015 at 01:16, Nathaniel Smith wrote: > Well.... I wouldn't say "no one". You weren't there at the NumPy BoF > at SciPy this year, where a substantial portion of the room started > calling for exactly this, and I felt pretty alone up front trying to > squash it almost singlehandedly. It was a bit awkward actually! [...] > What I'm saying is, it's not a coincidence that it was after SciPy > this year that I finally subscribed to distutils-sig :-). Ouch. This is a scenario that I (as a casual numpy user with no particular interest in conda) am definitely concerned about - so thanks for fighting that fight on my behalf (and that of many others, I'm sure). Paul From qwcode at gmail.com Thu Oct 29 07:33:57 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 04:33:57 -0700 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: > > > > 4) Although using a process interface is not necessarily a problem, I > don't > > agree with your point on why a python interface would be unworkable. > You're > > assuming that pip would try to import all the build tools (for every > > dependency it's processing) in the master process. An alternative could > be > > that pip would have it's own tool (that runs as a subprocess in an > isolated > > env) that knows how to load and work with python build interfaces. You > > could argue that a python api is an advantage, because build tools > aren't > > That would mean that pip has to have the same exact version of it embedded > in the environment that build tools will be run in, which it doesn't have > today. > sorry, I can't make clear sense of your sentence here. : ) I'll just explain my point again. pip doesn't necessarily have to "interact with many different versions of the same build tool during a single invocation" if for example it's subprocessing the interactions to some "pip-build" tool that handles the imports and use of the python API. I.e. pips calls some "pip-build" too (per build), which does the import, not pip itself. and again, it's not about arguing for this idea, but just that your "in-process APIs are harder" argument doesn't decide the matter. > > I see no problem with evolving them in lockstep, it's unnecessarily complex IMO. if they're really in lockstep, then they're one thing it seems to me. I'd rather avoid the chance for a bug where something tries to parse a > v2 schema build > description with a v1 schema parser. but it won't happen? the pypa.yaml schema version would determine the parser version that's used. > > 7) it's unclear when pip get's to run "dist-info" and when the result > might > > be different. For example, we've discussed that run time dependencies > may > > get clarifed *after* the build process.... so this command might produce > > different results at different times? > > Pip would run dist-info when determining the install-requires and > extras for the package. you're not addressing the point about how the act of building can create new run time dependencies, per the whole long discussion with Nathaniel recently (his draft deals with this matter explicitly) --Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From takowl at gmail.com Thu Oct 29 07:42:45 2015 From: takowl at gmail.com (Thomas Kluyver) Date: Thu, 29 Oct 2015 11:42:45 +0000 Subject: [Distutils] flit adapter for the current sdist format In-Reply-To: References: Message-ID: Thanks Daniel, that's interesting to see. For flit, I don't see a major benefit in generating sdists - for pure Python packages, universal wheels are simpler and install quicker. I'd really like to improve installing directly from a source tree, such as a VCS checkout. To do that, any adapter would need to be checked into every project using flit, so it's important to me that it be as small and simple as possible. This was a key feature of Nathaniel's proposal for me: it would let 'pip install .' work with just a few lines of straightforward config. Best wishes, Thomas On 29 October 2015 at 02:51, Daniel Holth wrote: > OK, so I wanted to see what it would take to create current-format sdists, > defined as anything that has a setup.py that pip can install by running > said setup.py. The following script takes a flit package (in the current > directory and builds a pip-installable sdist. Tested against > https://github.com/pohmelie/aioftp > > https://gist.github.com/dholth/e17f243085dcc539813f > > The first script is the archiver and the second script is the setup.py > embedded into the sdist by the archiver. The embedded setup.py does an > isolated install of flit into the setup-requires directory, if required, > and uses flit to implement bdist_wheel. The trick is that the only thing > pip really needs from egg_info is the current requirements, easily > generated from wheel-style .dist-info. > > It might need slightly more metadata to pass on pypi and the argument > handling is inflexible, but it demonstrates most of what would be required > for a backwards compatible shim. > > If our new sdist format is an even easier way to adapt new build systems > than writing this trivial setup.py shim, then we might have something. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Oct 29 08:07:49 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 05:07:49 -0700 Subject: [Distutils] Second draft of a plan for a new source tree / sdist format In-Reply-To: References: Message-ID: > > > - Will allow for both static and dynamic specification of build > dependencies I think you need to fill in the story on dynamic dependencies, or otherwise this PEP will be a mystery to most people. I *think* I understand your motivation for this, based on hearing your plan (in another thread) of putting c libs into wheels, and actually declaring them as dependencies... and because these libs can vary based on build settings, your dependencies are "dynamic"... but all that needs to be explained in the PEP --Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 13:31:57 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 13:31:57 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils Message-ID: Hello! So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things. I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this: Standardize a key in setup.cfg that can be used for setup_requires. This will be a static list of items which need to be installed in order to execute the setup.py. Unlike the current key (which is inside of setup.py) you don't have to deal with problems of trying to defer the import of the items or dealing with setuptools at all since your setup.py will not be invoked until after those items are installed. For people who want to continue to use setuptools, this file is trivial to parse with the standard library, so they can actually parse it and stick it into a setup_requires as well which will have the benefit that the tool calling the setup.py can handle those dependencies but still have it working on older versions of those tools without requiring duplication. This can also help work to make setuptools less special. Right now you basically have to just manually install setuptools prior to installing anything that uses it (or at all for something like pip when installing from sdist). With this we could pretty easily make it so that the rule is something like "If there is a setup.cfg:setup_requires, then assume that everything that is needed to execute the setup.py is listed there, else assume an implicit 'setuptools' requirement". This means that end users won't need to have setuptools installed at all, because we'll treat it (mostly) like just another build tool. This would also (for the first time) make it possible for things to depend on a specific version of setuptools instead of having to support every version of setuptools ever. I think that could solve the problem of "bootstrapping" the requirements to execute a setup.cfg and extract that from being implicitly handled by setuptools. That still doesn't handle the problem of making it possible to actually invoke the now installed build system. I think that what we should do is figure out the minimal interface we need from a ``setup.py`` based on what already exists. This would be a much smaller API surface than what exists today and wouldn't (ideally) include anything that is an implementation detail of setuptools. We would also need to standard on what flags an arguments the various commands of setup.py MUST accept. This would of course not require a setup.py to implement _only_ that particular interface so additional commands that setuptools already provide can stay, they just won't be required. Off the top of my head, I think we'd probably want to have the ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need something like ``egg_info``, but I think we could probably diverge a bit from what setuptools does now and make it ``dist_info`` so it's backed by the same standard that Wheel is already. I think that these four commands can support everything we actually _need_ to support. This isn't everything we actually use today, but since this would be opt-in, we can actually be more restrictive about what build/installation tools would call and what level of fallback we need to support for them. The way it would work when what we have available is a sdist is something like this: We download a sdist and we notice that there is a setup.cfg:setup_requires. This toggles a stricter mode of operation where we no longer attempt to do as many fallbacks or hacks to try and work around various broken shims with setuptools. We read this key and install those items needed to execute the setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and build a wheel from that [1]. Once we have a wheel built, we'll feed that data back into the resolver [2] and use the runtime dependency information from within that wheel to continue resolving the dependencies. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not being installed in editable. In this case we'll call ``setup.py sdist`` first, then feed that into the above. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is being installed as an editable. In this case, we'll call ``setup.py build --inplace`` first, then do something to ensure that the inplace directory is on sys.path. This is currently undefined because I don't know exactly what we'd need to do to make this work, but I think it should be possible and will be more consistent. We'll probably also need something like ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat with removing setuptools specifics is something we'd need to figure out. So why do I think something like this would be better? * It uses interfaces that are already there (for the most part) which means ? that it's easier for people to adapt their current tooling to it, and to do ? it in a way that will work with existing legacy packages. * It defines an interface that build systems must adhere too, instead of ? defining an interface that lets you query for how you actually interact with ? the build system (sort of a plugin system). This removes indirection and in ? my experience, interfaces are generally less problematic then "plugins". * It doesn't drastically change the mechanics of how the underlying thing works ? so it's a much smaller iteration on existing concepts rather than throwing ? everything out and starting from scratch. * It moves us towards an overall installation "path" that I think will be ? benefical for us and will help to reduce the combinatorial explosion of ways ? that a set of dependencies can be installed. * It will be easier to integrate setuptools into this new system (since it ? largely already implements it) which means that it's easier for us to move ? projects to upgrade piece meal to this new system since it would only require ? dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could ? adjust setuptools (like adding a dist_info command) and have the implicit ? setup.cfg:setup_requires be 'setuptools>=somever' instead of just ? 'setuptools'. What this doesn't solve: * This essentially doesn't solve any of the dynamic vs static metadata issues ? with the legacy sdist format, but that's OK because it's just a small ? layering (a new setup.cfg feature) of a new feature combined with just ? standardizing what already exists. Drastically altering the sdist format to ? try and shoe-horn static metadata in there is probably not something that is ? going to work well in practice (and needs a more thought out, newer format). * Dynamically specified _build_ requirements (or really, build requirements at ? all other than via setup_requires). You can sort of kludge dynamically ? specified build requirements by making a sort of meta-package that you put in ? your static setup_requires that generats dynamic runtime requirements. I ? think keeping this step simple though and work on enabling breaking the ? dependency on setuptools/distutils in this step and then waiting to see if ? that is "enough" or if we need to layer on additional features (like dynamic ? or otherwise seperately declared build requirements). [1] This isn't related to the Wheel cache. In this stricter mode we would only ? ? ever build a wheel and install from that. If caching is on then we'll save ? ? that wheel and reuse it next time, if caching is not on then we'll just ? ? throw that wheel away at the end of the run. [2] Pretending for a minute that we have a real resolver. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Thu Oct 29 14:54:06 2015 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Oct 2015 18:54:06 +0000 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: I think that would be very handy, especially making setuptools not a special case. You could get it down to 3 lines in setup.cfg, a file that already exists and already gathers random settings. Case 1, pip both requires a single package, and looks for an entry point inside that package to build. [bootstrap] build-with:buildamajig buildamajig package: class Builder: def metadata(target="directory"): ... def wheel(): ... In buildamajig's setup.py: entry_points = { "pip.builder" : "builder = buildamajig:Builder" } Case 2 [bootstrap] requires=pbr pip makes sure pbr is importable inside setup.py, but uses setuptools to build the package. (Implicit build-with=pip_setuptools_adapter). Case 3 [bootstrap] build-with=buildtool:Foobar pip does not require a dependency, but noticing the : in the argument executes "from buildtool import Foobar ..." with the setup.cfg directory on sys.path; alternative: different key "builder=module:Object". Useful for embedded build scripts or anywhere entry points would be impractical. Case 4 [bootstrap] requires=random-dependency build-with=thing:Klass Legal to add requirements whether or not setup.py is used. On Thu, Oct 29, 2015 at 1:32 PM Donald Stufft wrote: > Hello! > > So I've been reading the various threads (well trying to, there's a lot > going > on in all of them) and I'm trying to reconile them all in my head and > sorting > out how things are going to look if we add them. I'm worried about adding > more > complication to an already underspecified format and making it a lot > harder for > tooling to work with these things. > > I'm wondering if there isn't a smaller step we can take towards allowing > better > interopability with diffrent build systems while also making the situation > clearer when someone is still using distutils or setuptools. For the sake > of > argument, let's say we do something like this: > > Standardize a key in setup.cfg that can be used for setup_requires. This > will > be a static list of items which need to be installed in order to execute > the > setup.py. Unlike the current key (which is inside of setup.py) you don't > have > to deal with problems of trying to defer the import of the items or dealing > with setuptools at all since your setup.py will not be invoked until after > those items are installed. For people who want to continue to use > setuptools, > this file is trivial to parse with the standard library, so they can > actually > parse it and stick it into a setup_requires as well which will have the > benefit > that the tool calling the setup.py can handle those dependencies but still > have > it working on older versions of those tools without requiring duplication. > > This can also help work to make setuptools less special. Right now you > basically have to just manually install setuptools prior to installing > anything > that uses it (or at all for something like pip when installing from sdist). > With this we could pretty easily make it so that the rule is something like > "If there is a setup.cfg:setup_requires, then assume that everything that > is > needed to execute the setup.py is listed there, else assume an implicit > 'setuptools' requirement". This means that end users won't need to have > setuptools installed at all, because we'll treat it (mostly) like just > another > build tool. This would also (for the first time) make it possible for > things to > depend on a specific version of setuptools instead of having to support > every > version of setuptools ever. > > I think that could solve the problem of "bootstrapping" the requirements to > execute a setup.cfg and extract that from being implicitly handled by > setuptools. That still doesn't handle the problem of making it possible to > actually invoke the now installed build system. > > I think that what we should do is figure out the minimal interface we need > from > a ``setup.py`` based on what already exists. This would be a much smaller > API > surface than what exists today and wouldn't (ideally) include anything > that is > an implementation detail of setuptools. We would also need to standard on > what > flags an arguments the various commands of setup.py MUST accept. This > would of > course not require a setup.py to implement _only_ that particular > interface so > additional commands that setuptools already provide can stay, they just > won't > be required. Off the top of my head, I think we'd probably want to have the > ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need > something like ``egg_info``, but I think we could probably diverge a bit > from > what setuptools does now and make it ``dist_info`` so it's backed by the > same > standard that Wheel is already. I think that these four commands can > support > everything we actually _need_ to support. This isn't everything we actually > use today, but since this would be opt-in, we can actually be more > restrictive > about what build/installation tools would call and what level of fallback > we > need to support for them. > > The way it would work when what we have available is a sdist is something > like > this: > > We download a sdist and we notice that there is a setup.cfg:setup_requires. > This toggles a stricter mode of operation where we no longer attempt to do > as > many fallbacks or hacks to try and work around various broken shims with > setuptools. We read this key and install those items needed to execute the > setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` > and > build a wheel from that [1]. Once we have a wheel built, we'll feed that > data > back into the resolver [2] and use the runtime dependency information from > within that wheel to continue resolving the dependencies. > > OR > > We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is > not > being installed in editable. In this case we'll call ``setup.py sdist`` > first, > then feed that into the above. > > OR > > We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is > being installed as an editable. In this case, we'll call > ``setup.py build --inplace`` first, then do something to ensure that the > inplace directory is on sys.path. This is currently undefined because I > don't > know exactly what we'd need to do to make this work, but I think it should > be > possible and will be more consistent. We'll probably also need something > like > ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards > compat > with removing setuptools specifics is something we'd need to figure out. > > So why do I think something like this would be better? > > * It uses interfaces that are already there (for the most part) which means > that it's easier for people to adapt their current tooling to it, and to > do > it in a way that will work with existing legacy packages. > > * It defines an interface that build systems must adhere too, instead of > defining an interface that lets you query for how you actually interact > with > the build system (sort of a plugin system). This removes indirection and > in > my experience, interfaces are generally less problematic then "plugins". > > * It doesn't drastically change the mechanics of how the underlying thing > works > so it's a much smaller iteration on existing concepts rather than > throwing > everything out and starting from scratch. > > * It moves us towards an overall installation "path" that I think will be > benefical for us and will help to reduce the combinatorial explosion of > ways > that a set of dependencies can be installed. > > * It will be easier to integrate setuptools into this new system (since it > largely already implements it) which means that it's easier for us to > move > projects to upgrade piece meal to this new system since it would only > require > dropping a ``setup.cfg`` with a ``setup_requires``. This also means we > could > adjust setuptools (like adding a dist_info command) and have the implicit > setup.cfg:setup_requires be 'setuptools>=somever' instead of just > 'setuptools'. > > What this doesn't solve: > > * This essentially doesn't solve any of the dynamic vs static metadata > issues > with the legacy sdist format, but that's OK because it's just a small > layering (a new setup.cfg feature) of a new feature combined with just > standardizing what already exists. Drastically altering the sdist format > to > try and shoe-horn static metadata in there is probably not something > that is > going to work well in practice (and needs a more thought out, newer > format). > > * Dynamically specified _build_ requirements (or really, build > requirements at > all other than via setup_requires). You can sort of kludge dynamically > specified build requirements by making a sort of meta-package that you > put in > your static setup_requires that generats dynamic runtime requirements. I > think keeping this step simple though and work on enabling breaking the > dependency on setuptools/distutils in this step and then waiting to see > if > that is "enough" or if we need to layer on additional features (like > dynamic > or otherwise seperately declared build requirements). > > > [1] This isn't related to the Wheel cache. In this stricter mode we would > only > ever build a wheel and install from that. If caching is on then we'll > save > that wheel and reuse it next time, if caching is not on then we'll just > throw that wheel away at the end of the run. > > [2] Pretending for a minute that we have a real resolver. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 15:02:47 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 15:02:47 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth at gmail.com) wrote: > I think that would be very handy, especially making setuptools not a > special case. You could get it down to 3 lines in setup.cfg, a file that > already exists and already gathers random settings. > > I don?t think we should try to use an entry point (at least not yet). One of the key points is to minimize the conceptual churn and to move forward slowly while still progressing. I don?t think that using a class really buys us much because anyone who is capable of writing that class is also capable of writing a small CLI interface to be used within a setup.py. It feels more like churn for churn?s sake than anything else and some sort of idealogical purity over pragmatism. I also don?t want to base any standard on setuptools entry points because they are not a standard (and if we did, the key should not mention pip at all, pip shouldn?t be special cased any more than setuptools should). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From brett at python.org Thu Oct 29 15:23:21 2015 From: brett at python.org (Brett Cannon) Date: Thu, 29 Oct 2015 19:23:21 +0000 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On Thu, 29 Oct 2015 at 12:03 Donald Stufft wrote: > On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth at gmail.com) wrote: > > I think that would be very handy, especially making setuptools not a > > special case. You could get it down to 3 lines in setup.cfg, a file that > > already exists and already gathers random settings. > > > > > > I don?t think we should try to use an entry point (at least not yet). One > of the key points is to minimize the conceptual churn and to move forward > slowly while still progressing. I don?t think that using a class really > buys us much because anyone who is capable of writing that class is also > capable of writing a small CLI interface to be used within a setup.py. It > feels more like churn for churn?s sake than anything else and some sort of > idealogical purity over pragmatism. > There is also the familiarity of standardizing the CLI that pip will use when calling setup.py. The one thing that Daniel's case 3 proposal has over Donald's, though, is that all data can be self-contained in setup.cfg. If a build tool chooses to have the user specify all data in setup.cfg then you just copy-and-paste the `[bootstrap]` part and then in some other section they specify the info the build tool wants. Donald's approach could do something similar but it also requires copying around a boilerplate setup.py that is never modified. It's a familiarity vs. simplicity issue. Personally I still like the simple setup.py approach for the familiarity of it as well as the backwards-compatibility/transition bit being so simple with older versions of setuptools. > > I also don?t want to base any standard on setuptools entry points because > they are not a standard (and if we did, the key should not mention pip at > all, pip shouldn?t be special cased any more than setuptools should). > Well, it kind of is a standard thanks to Paul and zipapp: https://docs.python.org/3/library/zipapp.html#module-zipapp > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 15:32:36 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 15:32:36 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 3:23:34 PM, Brett Cannon (brett at python.org) wrote: > On Thu, 29 Oct 2015 at 12:03 Donald Stufft wrote: > > > On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth at gmail.com) wrote: > > > I think that would be very handy, especially making setuptools not a > > > special case. You could get it down to 3 lines in setup.cfg, a file that > > > already exists and already gathers random settings. > > > > > > > > > > I don?t think we should try to use an entry point (at least not yet). One > > of the key points is to minimize the conceptual churn and to move forward > > slowly while still progressing. I don?t think that using a class really > > buys us much because anyone who is capable of writing that class is also > > capable of writing a small CLI interface to be used within a setup.py. It > > feels more like churn for churn?s sake than anything else and some sort of > > idealogical purity over pragmatism. > > > > There is also the familiarity of standardizing the CLI that pip will use > when calling setup.py. > > The one thing that Daniel's case 3 proposal has over Donald's, though, is > that all data can be self-contained in setup.cfg. If a build tool chooses > to have the user specify all data in setup.cfg then you just copy-and-paste > the `[bootstrap]` part and then in some other section they specify the info > the build tool wants. Donald's approach could do something similar but it > also requires copying around a boilerplate setup.py that is never modified. > It's a familiarity vs. simplicity issue. > > Personally I still like the simple setup.py approach for the familiarity of > it as well as the backwards-compatibility/transition bit being so simple > with older versions of setuptools. My thoughts are basically: * Defining the standard commands has benefits for the current ecosystem as well as the future (much like why PEP 503 defined the /simple/ interface even though I plan to make a new one in the future).? * We can always add the build-with key at a later point because adding things is much much easier than removing things so I think being conservative and taking smaller steps is better than taking medium steps. >? > > > > > I also don?t want to base any standard on setuptools entry points because > > they are not a standard (and if we did, the key should not mention pip at > > all, pip shouldn?t be special cased any more than setuptools should). > > > > Well, it kind of is a standard thanks to Paul and zipapp: > https://docs.python.org/3/library/zipapp.html#module-zipapp I?m not sure I follow? Where is the setuptools entry point there? I only see the foo.bar:Whatever syntax, which isn?t what I meant, I meant the actual feature of setuptools entry points which you need to use something like pkg_resources to fetch. > > > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > > DCFA > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Thu Oct 29 15:34:11 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Oct 2015 19:34:11 +0000 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On 29 October 2015 at 19:23, Brett Cannon wrote: >> I also don?t want to base any standard on setuptools entry points because >> they are not a standard (and if we did, the key should not mention pip at >> all, pip shouldn?t be special cased any more than setuptools should). > > > Well, it kind of is a standard thanks to Paul and zipapp: > https://docs.python.org/3/library/zipapp.html#module-zipapp Wait, what? That doesn't use setuptools entry points... Paul From dholth at gmail.com Thu Oct 29 15:39:52 2015 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Oct 2015 19:39:52 +0000 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: I find CLIs to be very difficult to implement, should I bootstrap require click? How does my bundled-in-sdist build system code generate executable wrappers for Windows? Plus on the command line all method signatures suck: metadata(arguments: list of strings in any order) wheel(arguments: list of strings in any order) It could be a module instead of a class but a method/function interface would have nice documented method signatures: def wheel(target: string): ... you'd copy the example into your new build system adapter, implement, and be done. Perhaps pip could also call more than one command per subprocess more easily if the need arose. If you want your def wheel() to shell out go ahead... I could live without the entry points, it adds one line to setup.cfg in my strawman. But we are doomed to standardize setuptools-compatible entry points eventually. On Thu, Oct 29, 2015 at 3:02 PM Donald Stufft wrote: > On October 29, 2015 at 2:54:19 PM, Daniel Holth (dholth at gmail.com) wrote: > > I think that would be very handy, especially making setuptools not a > > special case. You could get it down to 3 lines in setup.cfg, a file that > > already exists and already gathers random settings. > > > > > > I don?t think we should try to use an entry point (at least not yet). One > of the key points is to minimize the conceptual churn and to move forward > slowly while still progressing. I don?t think that using a class really > buys us much because anyone who is capable of writing that class is also > capable of writing a small CLI interface to be used within a setup.py. It > feels more like churn for churn?s sake than anything else and some sort of > idealogical purity over pragmatism. > > I also don?t want to base any standard on setuptools entry points because > they are not a standard (and if we did, the key should not mention pip at > all, pip shouldn?t be special cased any more than setuptools should). > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 16:03:08 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 16:03:08 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 3:40:04 PM, Daniel Holth (dholth at gmail.com) wrote: > I find CLIs to be very difficult to implement, should I bootstrap require > click? How does my bundled-in-sdist build system code generate executable > wrappers for Windows? Plus on the command line all method signatures suck: > > metadata(arguments: list of strings in any order) > wheel(arguments: list of strings in any order) > > It could be a module instead of a class but a method/function interface > would have nice documented method signatures: > > def wheel(target: string): ... > > you'd copy the example into your new build system adapter, implement, and > be done. Perhaps pip could also call more than one command per subprocess > more easily if the need arose. If you want your def wheel() to shell out go > ahead? Sure, your build tool can require click, or it can use argparse or anything it wants. It?s just Python and you no longer need to worry about trying to defer imports, so it becomes just another Python script at that point. Of course, since your build system can have it?s own dependencies now, if people really want a class based API like that it?d only take one person to make a wrapper that implements the CLI based build interface and layers on your build-with key. So if projects want to make a class based interface it?s pretty easy to make that possible. Ideally they?d use a custom section in the setup.cfg like: [cli2class] build-with = foobar:Whatever Like I said before, it?s easy to add stuff, hard to take it away. If someone did that and it worked out well, we could then add official support for that and remove the need to have the shim package without breaking things for anyone who used the shim package. Having the current interface defined is still then useful for everyone else. > > I could live without the entry points, it adds one line to setup.cfg in my > strawman. But we are doomed to standardize setuptools-compatible entry > points eventually. I?m sure we will end up standardizing setuptools-compatible entry points at some point, but we don?t need to do that right now. Small incremental changes that enable things to move forward without locking us into supporting something that has been largely untested. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Thu Oct 29 16:51:41 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 16:51:41 -0400 Subject: [Distutils] Brian Goetz - Stewardship: the Sobering Parts Message-ID: This was shown to me today, and I thought I?d share it here since it pretty much applies 100% to what we?re doing on this list too (other than the fact the examples are from Java) - https://www.youtube.com/watch?v=2y5Pv4yN0b0 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Thu Oct 29 17:01:07 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 14:01:07 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: > > > * This essentially doesn't solve any of the dynamic vs static metadata > issues > Nathaniel's grand plan requires dynamic run-time dependencies, so to be clear, this plan doesn't help the case that instigated most of the recent discussion, right? -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 17:03:34 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 17:03:34 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 5:01:32 PM, Marcus Smith (qwcode at gmail.com) wrote: > > > > > > * This essentially doesn't solve any of the dynamic vs static metadata > > issues > > > > Nathaniel's grand plan requires dynamic run-time dependencies, so to be > clear, this plan doesn't help the case that instigated most of the recent > discussion, right? It should still solve that problem because it allows a project to dynamically decide what dependencies they have (just like they can today) when they create the Wheel (but the Wheel itself must have static dependencies). I believe that still handles the case that Nathaniel needed solved. It just doesn?t solve our (pip/PyPI) desires to have static (or more static) metadata. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Thu Oct 29 17:04:01 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 14:04:01 -0700 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: > > > pip doesn't necessarily have to "interact with many different versions of > the same build tool during a single invocation" if for example it's > subprocessing the interactions to some "pip-build" tool that handles the > imports and use of the python API. I.e. pips calls some "pip-build" too > (per build), which does the import, not pip itself. > sorry, there's 2 confusing misspells in my sentence. here it is again: pip doesn't necessarily have to "interact with many different versions of the same build tool during a single invocation" if for example it's sub-processing the interactions to some "pip-build" tool that handles the imports and use of the python API. I.e. pip calls some "pip-build" tool (per build), which does the import, not pip itself. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Oct 29 17:09:25 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 14:09:25 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: > > > Nathaniel's grand plan requires dynamic run-time dependencies, so to be > > clear, this plan doesn't help the case that instigated most of the recent > > discussion, right? > > It should still solve that problem because it allows a project to > dynamically decide what dependencies they have (just like they can today) > help me out here... how can we dynamically construct dependencies as we're building wheels today? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Oct 29 17:10:58 2015 From: dholth at gmail.com (Daniel Holth) Date: Thu, 29 Oct 2015 21:10:58 +0000 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: When setup.py builds the wheel it can write any dependencies. On Thu, Oct 29, 2015, 17:09 Marcus Smith wrote: > > Nathaniel's grand plan requires dynamic run-time dependencies, so to be >> > clear, this plan doesn't help the case that instigated most of the >> recent >> > discussion, right? >> >> It should still solve that problem because it allows a project to >> dynamically decide what dependencies they have (just like they can today) >> > > help me out here... how can we dynamically construct dependencies as we're > building wheels today? > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 17:11:52 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 17:11:52 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 5:09:45 PM, Marcus Smith (qwcode at gmail.com) wrote: > > > > > Nathaniel's grand plan requires dynamic run-time dependencies, so to be > > > clear, this plan doesn't help the case that instigated most of the recent > > > discussion, right? > > > > It should still solve that problem because it allows a project to > > dynamically decide what dependencies they have (just like they can today) > > > > help me out here... how can we dynamically construct dependencies as we're > building wheels today? I?m not sure I understand the confusion? since a wheel is created by executing setup.py, you?d just have your build tool dynamically output different wheels based on the system you?re building on (or whatever axis is causing the dynamic dependencies). An example is like: https://github.com/pypa/twine/blob/a0c87357d9d5d588082c9a59f6efc6f6bc3d3498/setup.py#L28-L31 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From leorochael at gmail.com Thu Oct 29 17:43:48 2015 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Thu, 29 Oct 2015 19:43:48 -0200 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: This is a nice idea. It makes it easy to fix problematic `setup.py` files like this [1]: ``` from distutils.core import setup, Extension import numpy setup(ext_modules=[Extension("_cos_doubles", sources=["cos_doubles.c", "cos_doubles.i"], include_dirs=[numpy.get_include()])]) ``` [1] Documented here http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html Just add a setup.cfg with `numpy` in the `setup_requires` key and done. No more `ImportError: No module named numpy` on an empty virtualenv. A small layer on top of this idea could also mean a fix to people/systems who still do `setup.py install` (Debian current tooling was mentioned). Just change the `setup.py` above to: ``` from setuptools import setup, Extension, pip_bootstrap pip_bootstrap('setup.cfg') import numpy setup(ext_modules=[Extension("_cos_doubles", sources=["cos_doubles.c", "cos_doubles.i"], include_dirs=[numpy.get_include()])]) ``` The `pip_boostrap` function would: - Install pip as if it was a `setup_requires` (i.e., it gets dumped in the local directory, or somewhere more hygienic, but not in `site-packages`), maybe using `get-pip` so we can actually remove the `install` functionality from setuptools in the future - run `pip bootstrap setup.cfg`, so that pip installs `numpy` and any other setup_requires declared in setup.cfg (dumped in the local directory like today or somewhere more hygienic, but placing them on sys.path) - proceed as usual if `pip_bootstrap` was actually a (sufficiently magical) module instead of a function, even the second line above could be omitted. Cheers, Leo On 29 October 2015 at 15:31, Donald Stufft wrote: > Hello! > > So I've been reading the various threads (well trying to, there's a lot > going > on in all of them) and I'm trying to reconile them all in my head and > sorting > out how things are going to look if we add them. I'm worried about adding > more > complication to an already underspecified format and making it a lot > harder for > tooling to work with these things. > > I'm wondering if there isn't a smaller step we can take towards allowing > better > interopability with diffrent build systems while also making the situation > clearer when someone is still using distutils or setuptools. For the sake > of > argument, let's say we do something like this: > > Standardize a key in setup.cfg that can be used for setup_requires. This > will > be a static list of items which need to be installed in order to execute > the > setup.py. Unlike the current key (which is inside of setup.py) you don't > have > to deal with problems of trying to defer the import of the items or dealing > with setuptools at all since your setup.py will not be invoked until after > those items are installed. For people who want to continue to use > setuptools, > this file is trivial to parse with the standard library, so they can > actually > parse it and stick it into a setup_requires as well which will have the > benefit > that the tool calling the setup.py can handle those dependencies but still > have > it working on older versions of those tools without requiring duplication. > > This can also help work to make setuptools less special. Right now you > basically have to just manually install setuptools prior to installing > anything > that uses it (or at all for something like pip when installing from sdist). > With this we could pretty easily make it so that the rule is something like > "If there is a setup.cfg:setup_requires, then assume that everything that > is > needed to execute the setup.py is listed there, else assume an implicit > 'setuptools' requirement". This means that end users won't need to have > setuptools installed at all, because we'll treat it (mostly) like just > another > build tool. This would also (for the first time) make it possible for > things to > depend on a specific version of setuptools instead of having to support > every > version of setuptools ever. > > I think that could solve the problem of "bootstrapping" the requirements to > execute a setup.cfg and extract that from being implicitly handled by > setuptools. That still doesn't handle the problem of making it possible to > actually invoke the now installed build system. > > I think that what we should do is figure out the minimal interface we need > from > a ``setup.py`` based on what already exists. This would be a much smaller > API > surface than what exists today and wouldn't (ideally) include anything > that is > an implementation detail of setuptools. We would also need to standard on > what > flags an arguments the various commands of setup.py MUST accept. This > would of > course not require a setup.py to implement _only_ that particular > interface so > additional commands that setuptools already provide can stay, they just > won't > be required. Off the top of my head, I think we'd probably want to have the > ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need > something like ``egg_info``, but I think we could probably diverge a bit > from > what setuptools does now and make it ``dist_info`` so it's backed by the > same > standard that Wheel is already. I think that these four commands can > support > everything we actually _need_ to support. This isn't everything we actually > use today, but since this would be opt-in, we can actually be more > restrictive > about what build/installation tools would call and what level of fallback > we > need to support for them. > > The way it would work when what we have available is a sdist is something > like > this: > > We download a sdist and we notice that there is a setup.cfg:setup_requires. > This toggles a stricter mode of operation where we no longer attempt to do > as > many fallbacks or hacks to try and work around various broken shims with > setuptools. We read this key and install those items needed to execute the > setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` > and > build a wheel from that [1]. Once we have a wheel built, we'll feed that > data > back into the resolver [2] and use the runtime dependency information from > within that wheel to continue resolving the dependencies. > > OR > > We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is > not > being installed in editable. In this case we'll call ``setup.py sdist`` > first, > then feed that into the above. > > OR > > We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is > being installed as an editable. In this case, we'll call > ``setup.py build --inplace`` first, then do something to ensure that the > inplace directory is on sys.path. This is currently undefined because I > don't > know exactly what we'd need to do to make this work, but I think it should > be > possible and will be more consistent. We'll probably also need something > like > ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards > compat > with removing setuptools specifics is something we'd need to figure out. > > So why do I think something like this would be better? > > * It uses interfaces that are already there (for the most part) which means > that it's easier for people to adapt their current tooling to it, and to > do > it in a way that will work with existing legacy packages. > > * It defines an interface that build systems must adhere too, instead of > defining an interface that lets you query for how you actually interact > with > the build system (sort of a plugin system). This removes indirection and > in > my experience, interfaces are generally less problematic then "plugins". > > * It doesn't drastically change the mechanics of how the underlying thing > works > so it's a much smaller iteration on existing concepts rather than > throwing > everything out and starting from scratch. > > * It moves us towards an overall installation "path" that I think will be > benefical for us and will help to reduce the combinatorial explosion of > ways > that a set of dependencies can be installed. > > * It will be easier to integrate setuptools into this new system (since it > largely already implements it) which means that it's easier for us to > move > projects to upgrade piece meal to this new system since it would only > require > dropping a ``setup.cfg`` with a ``setup_requires``. This also means we > could > adjust setuptools (like adding a dist_info command) and have the implicit > setup.cfg:setup_requires be 'setuptools>=somever' instead of just > 'setuptools'. > > What this doesn't solve: > > * This essentially doesn't solve any of the dynamic vs static metadata > issues > with the legacy sdist format, but that's OK because it's just a small > layering (a new setup.cfg feature) of a new feature combined with just > standardizing what already exists. Drastically altering the sdist format > to > try and shoe-horn static metadata in there is probably not something > that is > going to work well in practice (and needs a more thought out, newer > format). > > * Dynamically specified _build_ requirements (or really, build > requirements at > all other than via setup_requires). You can sort of kludge dynamically > specified build requirements by making a sort of meta-package that you > put in > your static setup_requires that generats dynamic runtime requirements. I > think keeping this step simple though and work on enabling breaking the > dependency on setuptools/distutils in this step and then waiting to see > if > that is "enough" or if we need to layer on additional features (like > dynamic > or otherwise seperately declared build requirements). > > > [1] This isn't related to the Wheel cache. In this stricter mode we would > only > ever build a wheel and install from that. If caching is on then we'll > save > that wheel and reuse it next time, if caching is not on then we'll just > throw that wheel away at the end of the run. > > [2] Pretending for a minute that we have a real resolver. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Oct 29 18:32:01 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 15:32:01 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: > > > since a wheel is created by executing setup.py, you?d just have your build > tool dynamically output different wheels based on the system you?re > building on (or whatever axis is causing the dynamic dependencies). understood, but I guess I was under the [mistaken?] impression, that dynamic support is needed literally in the wheel build step. for example, going with your idea above, that we need to support 3rd party tools being dynamic in the "setup.py bdist_wheel" step, not simply in the "setup.py dist-info" step. and as it is, pip does't understand this possibility. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 18:36:36 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 18:36:36 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 6:32:21 PM, Marcus Smith (qwcode at gmail.com) wrote: > > > > > > since a wheel is created by executing setup.py, you?d just have your build > > tool dynamically output different wheels based on the system you?re > > building on (or whatever axis is causing the dynamic dependencies). > > > understood, but I guess I was under the [mistaken?] impression, that > dynamic support is needed literally in the wheel build step. > > for example, going with your idea above, that we need to support 3rd party > tools being dynamic in the "setup.py bdist_wheel" step, not simply in the > "setup.py dist-info" step. > > and as it is, pip does't understand this possibility. So in the new path, we wouldn?t call ``setup.py dist-info`` for ?normal? installations, we?d compile a wheel and inspect that for dependencies. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From robertc at robertcollins.net Thu Oct 29 18:50:01 2015 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 30 Oct 2015 11:50:01 +1300 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On 30 October 2015 at 06:31, Donald Stufft wrote: tl;dr: - requiring wheel before dependencies is a Big Deal IMO - I don't care about pypa.yaml vs setup.cfg - argue with the flit folk :) - I don't care about setup.py-as-interface vs described-by-metadata - argue with the flit folk - I *do* want someone to pick choices on the 2 previous points so we can move forward :) > Hello! > > So I've been reading the various threads (well trying to, there's a lot going > on in all of them) and I'm trying to reconile them all in my head and sorting > out how things are going to look if we add them. I'm worried about adding more > complication to an already underspecified format and making it a lot harder for > tooling to work with these things. Thats fair enough, and thanks for putting the effort in :). > I'm wondering if there isn't a smaller step we can take towards allowing better > interopability with diffrent build systems while also making the situation > clearer when someone is still using distutils or setuptools. For the sake of > argument, let's say we do something like this: So - my proposal ended up where it did because: - the flit folk really didn't want to have to have setup.py in their tree at all - they strongly objected to that. - setup.cfg had a similar (but weaker) objection - we obviously need a place for pip to key off of to drive things - building packages with non-trivial build times is something we *really* need to keep out of the inner loop of resolving dependencies. Building to wheel first is a *huge* potential cost, *especially* when we haven't solved the wheels-for-linux problem. I've previously (on this list, 6? 8? months backs) proposed using setup.cfg directly to do build time requirements and use that as the interface. I'm still happy doing that, so if thats the direction you and Nick (or whatever the BDFL-equivalents here) want to take, I can easily strip out the indirection involved in defining the interface to the build tool and just assume setup.py with those setuptools-like commands. The specific complexity in the interface was to avoid the emergent need to update config in many trees when flit etc do change things. Similarly, if you want to add the dependency on defining 'develop' as something pip does itself to this PEP, I can help write that PEP (which I think we should do anywhich way), and pivot from asking for develop to be done, and onto having a build inplace facility. However, the big step you're proposing that I think is fundamentally unsound is that of requiring a wheel be built before dependencies can be queried. Wheels are *expensive* to build in enough cases that I really believe we have a choice between being able to fix issue 988 or utilising wheels as the interface to get at install metadata. Remember too that the numpy ABI discussion is going to cause install dependencies to depend on the version of numpy present at build-time, so the resolver is going to have to re-evaluate the dependencies for those distributions when it selects a new numpy version: so its not even a one-time-per-package cost, its a one-time-per-arc-of-the-N!-graph-paths cost. What you're describing is in all other regards basically my PEP, so +1 sure. -Rob From qwcode at gmail.com Thu Oct 29 18:51:18 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 15:51:18 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: > > > > for example, going with your idea above, that we need to support 3rd > party > > tools being dynamic in the "setup.py bdist_wheel" step, not simply in the > > "setup.py dist-info" step. > > > > and as it is, pip does't understand this possibility. > > So in the new path, we wouldn?t call ``setup.py dist-info`` for ?normal? > installations, we?d compile a wheel and inspect that for dependencies. > ok, so a pretty significant change to pip's resolution internals. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Thu Oct 29 18:56:45 2015 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 30 Oct 2015 11:56:45 +1300 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On 30 October 2015 at 00:33, Marcus Smith wrote: >> >> > 4) Although using a process interface is not necessarily a problem, I >> > don't >> > agree with your point on why a python interface would be unworkable. >> > You're >> > assuming that pip would try to import all the build tools (for every >> > dependency it's processing) in the master process. An alternative could >> > be >> > that pip would have it's own tool (that runs as a subprocess in an >> > isolated >> > env) that knows how to load and work with python build interfaces. You >> > could argue that a python api is an advantage, because build tools >> > aren't >> >> That would mean that pip has to have the same exact version of it embedded >> in the environment that build tools will be run in, which it doesn't have >> today. > > > sorry, I can't make clear sense of your sentence here. : ) > > I'll just explain my point again. > > pip doesn't necessarily have to "interact with many different versions of > the same build tool during a single invocation" if for example it's > subprocessing the interactions to some "pip-build" tool that handles the > imports and use of the python API. I.e. pips calls some "pip-build" too > (per build), which does the import, not pip itself. > > and again, it's not about arguing for this idea, but just that your > "in-process APIs are harder" argument doesn't decide the matter. I agree that it doesn't decide the matter - I added a section to rationale about this contention. We can literally do whatever we want to do - my reading of the interactions is that its going to be less fragile overall to not have to have a Python->command process thunk that is a separate interface we have to carry around in Pip everywhere. But we can make literally anything work, so I'm probably going to just say 'Nick and Donald can decide' and not argue about this. >> >> >> I see no problem with evolving them in lockstep, > > > it's unnecessarily complex IMO. if they're really in lockstep, then they're > one thing it seems to me. I don't understand. >> I'd rather avoid the chance for a bug where something tries to parse a >> v2 schema build >> description with a v1 schema parser. > > > but it won't happen? the pypa.yaml schema version would determine the > parser version that's used. We have two separate documents. If we mark the schema version in one, and not in the other, then we introduce room for skew. Yes, anything that skewed would be buggy, but there'd be no clue that such a bug was happening to anyone debugging it. All my experience debugging network protocols and data storage engines tells me to be explicit about the version of anything that can *possibly* be evaluated separately. >> >> > 7) it's unclear when pip get's to run "dist-info" and when the result >> > might >> > be different. For example, we've discussed that run time dependencies >> > may >> > get clarifed *after* the build process.... so this command might produce >> > different results at different times? >> >> Pip would run dist-info when determining the install-requires and >> extras for the package. > > > you're not addressing the point about how the act of building can create new > run time dependencies, per the whole long discussion with Nathaniel > recently (his draft deals with this matter explicitly) I don't follow how there is an issue: the dist-info hook would be responsible for figuring that out. In the numpy ABI case it would do discovery of the ABI it would build against and then use that. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Thu Oct 29 18:58:57 2015 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 30 Oct 2015 11:58:57 +1300 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On 30 October 2015 at 11:50, Robert Collins wrote: ... > However, the big step you're proposing that I think is fundamentally > unsound is that of requiring a wheel be built before dependencies can > be queried. Wheels are *expensive* to build in enough cases that I > really believe we have a choice between being able to fix issue 988 or > utilising wheels as the interface to get at install metadata. Remember > too that the numpy ABI discussion is going to cause install > dependencies to depend on the version of numpy present at build-time, > so the resolver is going to have to re-evaluate the dependencies for > those distributions when it selects a new numpy version: so its not > even a one-time-per-package cost, its a > one-time-per-arc-of-the-N!-graph-paths cost. On further thought there is one mitigating factor here - we'd have to be building numpy fully anyway for the cases where numpy is a build-dependency for things further up the stack, so even in my most paranoid world we will *anyway* have to pay that cost - but we can avoid the cost of multiple full rebuilds of C extensions built on top of numpy. I think avoiding that cost is still worth it, its just a little more into the grey area rather than being absolutely obviously worth it. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ralf.gommers at gmail.com Thu Oct 29 18:54:19 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 29 Oct 2015 23:54:19 +0100 Subject: [Distutils] build system abstraction PEP In-Reply-To: References: Message-ID: On Tue, Oct 27, 2015 at 12:23 AM, Robert Collins wrote: > On 27 October 2015 at 10:32, Ralf Gommers wrote: > > > > > (2) Complex example: to build a Scipy wheel on Windows with MinGW the > > command is ``python setup.py config --compiler=mingw32 build > > --compiler=mingw32 bdist_wheel``. > > So in this case the build tool needs to know about the compiler stuff > itself- pip doesn't know. We have a way in pip to tunnel stuff down to > setuptools today; thats incompatible with dynamically building wheels > on the fly for 'pip install' - so I'm not sure it needs to be > reflected here. > It looks like you made a start at https://github.com/rbtcollins/interoperability-peps/blob/build-system-abstraction/build-system-abstraction.rst#handling-of-compiler-options "Instead we recommend that individual build tools should have a config file mechanism to provide such pervasive settings across all things built locally." makes sense, at least up to "across". The same settings for everything built locally isn't appropriate - should be able to have a config file for one project. Example: you may want to build with MSVC and with MinGW on Windows. Also, it seems to me like there should be a way to pass the full path of a config file to the build tool via pip. Can be easily done via an optional key "config-file" in the build tool description. Example: right now numpy distributes a site.cfg.example that users can rename to site.cfg and usually put right next to setup.py. When one uses pip, it may go off building in some tmpdir, change path envvars, etc. - so how does the build tool find that config file? Ralf > We'll need some more input on that I think. > ... > > mechanism) to the build tool. If it is out of scope, I'd be interested to > > see what you think are use-cases with complex requirements that are > enabled > > by this PEP. > > The PEP is aimed at enabling additional build-tools to be on parity > with setuptools in *pip install* invocations that go through the > wheel-autobuild-path in pip. > > The complex examples of passing arbitrary options to setup.py > currently bypasses wheel building in pip, and so can't be tackled at > all :(. > > But we can work on including that with some thought. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 29 19:05:14 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 29 Oct 2015 19:05:14 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On October 29, 2015 at 6:59:00 PM, Robert Collins (robertc at robertcollins.net) wrote: > On 30 October 2015 at 11:50, Robert Collins wrote: > ... > > However, the big step you're proposing that I think is fundamentally > > unsound is that of requiring a wheel be built before dependencies can > > be queried. Wheels are *expensive* to build in enough cases that I > > really believe we have a choice between being able to fix issue 988 or > > utilising wheels as the interface to get at install metadata. Remember > > too that the numpy ABI discussion is going to cause install > > dependencies to depend on the version of numpy present at build-time, > > so the resolver is going to have to re-evaluate the dependencies for > > those distributions when it selects a new numpy version: so its not > > even a one-time-per-package cost, its a > > one-time-per-arc-of-the-N!-graph-paths cost. > > On further thought there is one mitigating factor here - we'd have to > be building numpy fully anyway for the cases where numpy is a > build-dependency for things further up the stack, so even in my most > paranoid world we will *anyway* have to pay that cost - but we can > avoid the cost of multiple full rebuilds of C extensions built on top > of numpy. I think avoiding that cost is still worth it, its just a > little more into the grey area rather than being absolutely obviously > worth it. > We?d only need to pay that cost on the assumption that we don?t have a Wheel cached already right? Either in the machine local cache (~/.caches/pip/wheels/) or a process local cache (temp directory?). Since opting into the new build system mandates the ability to build wheels, for these items we can always assume we can build a wheel.? I mentioned it on IRC, but just for the folks who aren?t on IRC, I?m also not dead set on requiring a wheel build during resolution. I did that because we currently have a bit of a race condition since we use ``setup.py egg_info`` to query the dependencies and then we run ``setup.py bdist_wheel`` to build the thing and the dependencies are not guaranteed to be the same between the invocation of these two commands. If we moved to always building Wheels then we eliminate the race condition and we make the required interface smaller. I wouldn?t be opposed to including something like ``setup.py dist-info`` in the interface if we included an assertion stating that, given no state on the machine changes (no additional packages installed, not invoking from a different Python, etc), that the metadata produced by that command must be equal to the metadata produced in the wheel, and we can put an assertion in pip to ensure it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From qwcode at gmail.com Thu Oct 29 19:09:21 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 16:09:21 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: > > > However, the big step you're proposing that I think is fundamentally > unsound is that of requiring a wheel be built before dependencies can > be queried. so, Robert, to be clear, you think 3rd party build tools can get the dynamic-dependency support they need just by implementing something dynamic via "setup,py dist-info"? (and it doesn't need to happen literally in the wheel build step?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 29 19:23:00 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 29 Oct 2015 16:23:00 -0700 Subject: [Distutils] metadata in sdists (was Re: Second draft of a plan for a new source tree / sdist format) Message-ID: On Wed, Oct 28, 2015 at 5:32 AM, Daniel Holth wrote: > Nathaniel, > > I'm not sure what the software is supposed to do with fine grained dynamic > metadata that would make very much sense to the end user. I think you could > probably get away with a single flag Dynamic: true / false. Iff true, pip > runs the dist-info command after installing bootstrap dependencies. You > could still complain if the name & version changed. Of course in a VCS > checkout or during development you probably always want the > regenerate-metadata behavior. So, right, the draft I just posted proposes that sdists should contain static wheel-style metadata where possible, with a fine-grained mechanism for marking specific parts as being dynamically generated at build time. The different motivations that I'm trying to balance here are: - Donald wants the fields that PyPI cares about (name, version, summary, long_description, author, homepage, trove classifiers, etc.) to *always* be statically present in sdists, even if there is some other metadata that can't be made static (i.e., install-requirements) - Robert wants the (name, version, install-requirements) information to be available statically as often as possible, because that's the information that the resolver needs to know when it's considering trying to install some (name, version) pair, and he wants this information to be cheap to access because the resolver may have to backtrack and consider many different versions of the same package. Installing build-requirements and then running egg-info/dist-info is already pretty expensive, so it would be good if install-requirements could be static in the 99% of cases where they are known. - I'm still holding out some small hope of killing off the "egg-info"/"dist-info" step entirely, because any time you have two different operations and it's a bug if they can get out of sync (in this case: the dist-info step, and then the actual wheel building step), then maintaining and testing is a hassle and eventually that bug will happen. Basically it's a violation of DRY -- you have two sources of the same data and they're both supposed to be authoritative. And we know from experience with egg_info that there will be strong temptations for people to cheat, which creates all kinds of headaches. So how can we balance these different design goals? Obviously a new sdist format should provide static metadata for the fields PyPI cares about -- we basically require this already for sdists, we just encode it in a weird out-of-band way during the PyPI upload instead of recording it authoritatively in the sdist itself. Then that leaves a 2x2 space of plausible design options: we can either support dist-info as a separate operation from building, or not; and we can make a best-effort to provide static install-requirements when they're available, or we can not bother and just never provide static install-requirements. None of this matters for 'pip install ' None of this matters for installations using wheels. None of this matters for installations where the requirements are straightforward to satisfy (e.g. cases where the current pseudo-resolver works okay). None of this affects correctness -- it's purely an optimization. But maybe it's an important optimization in certain specific cases. The case where this matters is like: suppose you just did 'pip install scikit-learn', and scikit-learn requires scipy and numpy. And scipy also requires numpy, so our dependency graph forms a triangle. And let's say that scikit-learn is happy with any version of scipy, but scikit-learn and scipy both have versioned requirements on numpy. So for example, scikit-learn might require numpy (< 1.9). Meanwhile, the latest version of scipy requires numpy (>= 1.9)... so there's no version of numpy that satisfies both at once, which means we can't use this version of scipy. So you have to consider the next-to-latest version of scipy. But then it turns out that it also requires numpy (>= 1.9). So then you have to consider the next-to-next-to-latest version of scipy, which turns out to require numpy (>= 1.8), so phew, we can use numpy = 1.8 to satisfy both of these at once and we're done. And, to make this maximally annoying, it turns out that there is no scipy wheel for your platform, so all this checking of scipy's install-requires has to be done based on whatever information is inside the scipy sdist. And finally, the reason we use scipy in this example is that building scipy turns out to be really slow (~8 minutes on my laptop), definitely an outlier among python packages. There's three ways this can play out: Option 1, "fast": if the scipy sdists have static install-requires metadata, then this is pretty trivial. You might have to download the sdists for the first two non-working versions, but once you have the files you can immediately tell from the metadata that they aren't going to be usable and move on. (Of course, scipy itself probably *can't* have static install-requires metadata, but most packages can.) Option 2, "slow": if the scipy sdists don't have static install-requires metadata, but do support the dist-info operation, then you do something like: - download and unpack scipy-XX.zip - discover that it build-requires numpy. If there's no scipy wheel for your platform, there's presumably no numpy wheel either, so we'll have to build it. - download and build numpy, and install your newly built numpy into scipy's build environment (~2 minutes?) - run scipy's dist-info (a few seconds?) - go back to the top and repeat for the next candidate Option 3, "slowest": if you don't have static install-requires *and* you don't have the dist-info operation, then it's identical to the "slow" option, except instead of dist-info you have to actually run the full scipy build at each step, so it takes ~10 minutes per cycle instead of ~2 minutes per cycle (or maybe a bit less depending on whether the build-requirements work out so that you can re-use the same build of numpy for both attempts, etc.) Now, our four design options break down like this: no dist-info / never record install-requirements in sdist: 100% of sdists are in the "slowest" path no dist-info / record install-requirements in sdist when available: 95% of sdists are on "fast" path, 5% (the ones with dynamic install-requirements, like scipy) are on the "slowest" path dist-info / never record install-requirements in sdist (= current sdist format): 100% of sdists are in the "slow" path dist-info / record install-requirements in sdist when available: 95% of sdists are on "fast" path, 5% are on the "slow" path *** Summary *** There are two different ways to speed up resolution of dependencies involving sdists. static install-requirements are super fast when they work, and their complexity cost is low but non-zero. (We know we want to include lots of other metadata in sdists, and we already have standard ways to represent lots of metadata + install-requirements, so the only new thing is the __SDIST_DYNAMIC__ bit.) dist-info is slower and always works, but has a higher complexity cost. (Every build backend needs to implement this extra operation, and it's a bit fragile.) If you think that: - automagic building from sdists is an important case to support seamlessly and will remain so indefinitely, and - these kinds of complicated conflicts are common, and - it's important to optimize them to be resolved as fast as possible, then you'll want dist-info + static install-requirements. (IIUC this is Robert's position.) OTOH if you think that: - automagic building of sdists is a mixed blessing at best (at least in this particular example, what mostly actually happens is that people get angry and swear at their computer because they didn't want 'pip install scikit-learn' to go rebuild scipy and numpy, argh why is it my laptop suddenly swapping to death / argh I don't actually have a functional toolchain installed because I'm on windows / argh the resulting builds will seem to work at first but in fact be horribly slow because my BLAS is in funny place where it won't be autodetected and so they'll fall back on unoptimized routines / ... see also Paul's requests to just turn off sdist building entirely), and - by the time these new formats are implemented and in wide use in a few years then the increased maturity of the wheel ecosystem + Linux wheels on PyPI + Donald's planned automated build servers will mean that automagic building of sdists will only be rarely needed practice, and - these kinds of pathological requirements conflicts are going to be pretty rare and anyway when they do arise then you don't care whether it takes 60 minutes or merely 20 to sort out scikit-learn's dependencies because you're probably going to hit Control-C either way... then you might prefer the no-dist-info + no static install-requirements approach, because it simplifies the build system / packaging stuff :-). (That was the reasoning behind my original proposal.) And if you like that second argument, but have gotten push-back from people like Robert, then you might try writing up a proposal for static metadata in sdists, in the hopes that no-dist-info / yes-static-metadata might be an acceptable compromise between the above two positions :-). This discussion is important structurally is because it's where the design of a new build system interface becomes coupled to the design of a new sdist format: if you want to get a new build system interface done as soon as possible and don't want to touch the static metadata stuff, then the choice is between "no dist-info / no static metadata" versus "yes dist-info / no static-metadata", and if these are your options then it pushes you strongly towards supporting a dist-info operation. But then you risk saddling all build systems with the requirement to implement this dist-info thing, even if it later turns out that static sdist metadata makes it unnecessary. As mentioned in the other thread about promoting extras to first-class packages, there is still some hope that we might be able to get away with having full static metadata in all sdists, in which case the dist-info operation becomes completely vestigial -- but that discussion is going to be much more involved and certainly requires the addition of a real resolver for pip. So we probably don't want to block all progress on better build system while waiting for that. One possible compromise to streamline things might be: - give up on specifying a new sdist format for now (sorry Donald!) -- it's a good idea, we have some idea how to do it, but treat it as a separate project - include a dist-info operation in the new standard build system interface... - ...but make it optional. The cost of making it optional is that pip would need to be prepared with a fallback if the dist-info operation is not available -- so the get-install-requirements step would look like: def get_install_requirements(source_directory): if supports_dist_info(source_directory): dist_info_directory = run_dist_info_hook(source_directory) return dist_info_to_install_requirements(dist_info_directory) else: wheel = run_wheel_build_hook(source_directory) return wheel_to_install_requirements(wheel) Compared to a design where dist-info is mandatory, this is some extra complexity added to pip, but not a huge amount -- the above is pseudo-code, but aside from supports_dist_info all the functions it calls are ones that need to exist anyway, and the extra logic is neatly encapsulated within get_install_requirements. The minor advantage of doing this is that if some project/build system doesn't need this logic they don't need to implement it (e.g. for flit, which only supports pure python packages, there's absolutely no use -- it can build a wheel as fast as it can generate just the dist-info, and it doesn't even matter because projects using flit will ~always have wheels available and the resolver will ~never need to look at an sdist). The major advantage is that if later on it turns out that dist-info is useless (e.g. because sdists start shipping with full static metadata), then making it optional now means that we'll have the option of just dropping the support from build systems and from pip without breaking backcompat. (Compare to the situation where we make it mandatory now: since there will be versions of pip in the wild that blow up if the operation is not provided, every build system may be stuck providing some sort of dist-info support for a decade or whatever, even if it's not needed.) Alternatively we could just leave dist-info out for now, and implement it later as an optional optimization that a build system can provide if it turns out to matter :-). This would definitely help move things forward more quickly, because I wouldn't be surprised if the most complicated part of implementing Robert's current spec turned out to be the code needed to validate that dist-info and build-wheel actually return matching metadata :-). You absolutely need to do this by default, because otherwise we'll be back where we are now with people intentionally playing horrible fragile tricks, and weird accidental inconsistencies going undetected and creating subtle bugs. But to implement this checking you need some complex flow control to make sure that the eventual build-wheel step has access to the original metadata, plus the code for checking metadata-equality might be non-trivial... -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Thu Oct 29 19:30:08 2015 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 29 Oct 2015 16:30:08 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: On Thu, Oct 29, 2015 at 4:09 PM, Marcus Smith wrote: >> >> However, the big step you're proposing that I think is fundamentally >> unsound is that of requiring a wheel be built before dependencies can >> be queried. > > > so, Robert, to be clear, you think 3rd party build tools can get the > dynamic-dependency support they need just by implementing something dynamic > via "setup,py dist-info"? (and it doesn't need to happen literally in the > wheel build step?) Sure -- 'dist-info' would be run in the same environment as bdist_wheel, a few seconds before bdist_wheel was run (unless the output from dist-info convinces us that we need a different version of this package and so we skip running bdist_wheel entirely). So at least in principle, anything bdist_wheel can do/know/figure-out is something that dist-info could also do/know/figure-out. The question about whether to provide a 'dist-info' step is whether the optimization it provides is worth the extra complexity it imposes on build systems and the opportunities for what Donald called "race conditions" (= accidental or intentional skew between the output of dist-info and the actual metadata in the eventual wheel). See also the long email I just sent in its own thread going into more detail on what this trade-off entails :-) -n -- Nathaniel J. Smith -- http://vorpus.org From qwcode at gmail.com Thu Oct 29 19:33:13 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 16:33:13 -0700 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: > > > > so, Robert, to be clear, you think 3rd party build tools can get the > > dynamic-dependency support they need just by implementing something > dynamic > > via "setup,py dist-info"? (and it doesn't need to happen literally in the > > wheel build step?) > > Sure -- 'dist-info' would be run in the same environment as > bdist_wheel, a few seconds before bdist_wheel was run (unless the > output from dist-info convinces us that we need a different version of > this package and so we skip running bdist_wheel entirely). So at least > in principle, anything bdist_wheel can do/know/figure-out is something > that dist-info could also do/know/figure-out. The question about > whether to provide a 'dist-info' step is whether the optimization it > provides is worth the extra complexity it imposes on build systems and > the opportunities for what Donald called "race conditions" (= > accidental or intentional skew between the output of dist-info and the > actual metadata in the eventual wheel). > thankyou!! for the clarity : ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Oct 30 01:52:54 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 22:52:54 -0700 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: > If python-dev ends up adopting GitLab for the main PEPs repo, then we > should be able to move the whole process there, rather than needing to > maintain a separate copy. > will that be as open as pypa/interoperability-peps? if it's closed off such that only python devs can log PRs against PEPs once they're in the system, then that's no fun. If so, I'd still want to keep pypa/interoperability-peps as the front end tool for the actual change management. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Oct 30 02:27:31 2015 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 29 Oct 2015 23:27:31 -0700 Subject: [Distutils] BDFL Delegates for distutils-sig PEPs In-Reply-To: References: Message-ID: > > > ================================= > Whenever a new PEP is put forward on distutils-sig, any PyPA core > reviewer that believes they are suitably experienced to make the final > decision on that PEP may offer to serve as the BDFL's delegate (or > "PEP czar") for that PEP. If their self-nomination is accepted by the > other PyPA core reviewer, the lead PyPI maintainer and the lead > CPython representative on distutils-sig, then they will have the > authority to approve (or reject) that PEP. > ================================= > Nick: just be clear, if nobody nominates themselves, then you still remain (by default) the active delegate who's responsible for ruling on a Pypa-related PEP? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Oct 30 03:58:50 2015 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 30 Oct 2015 08:58:50 +0100 Subject: [Distutils] wacky idea about reifying extras In-Reply-To: References: Message-ID: On Thu, Oct 29, 2015 at 2:16 AM, Nathaniel Smith wrote: > On Wed, Oct 28, 2015 at 4:30 PM, Ralf Gommers > wrote: > > > > On Tue, Oct 27, 2015 at 5:45 PM, Brett Cannon wrote: > >> > >> > >> Nathaniel's comment about how this might actually give pip a leg up on > >> conda also sounds nice to me as I have enough worry about having a > fissure > >> in 1D along the Python 2/3 line, and I'm constantly worried that the > >> scientific community is going to riot and make it a 2D fissure along > Python > >> 2/3, pip/conda axes and split effort, documentation, etc. > > > > > > If it helps you sleep: I'm confident that no one is planning this > particular > > riot. It takes little work to support pip and conda - the hard issues are > > mostly with building, not installing. > > Well.... I wouldn't say "no one". You weren't there at the NumPy BoF > at SciPy this year, where a substantial portion of the room started > calling for exactly this, and I felt pretty alone up front trying to > squash it almost singlehandedly. It was a bit awkward actually! > Hmm, guess I missed something. Still confident that it won't happen, because (a) it doesn't make too much sense to me, and (b) there's probably little overlap between the people that want that and the people that do the actual build/packaging maintenance work (outside of conda people themselves). > The argument for numpy dropping pip support is actually somewhat > compelling. It goes like this: conda users don't care if numpy breaks > ABI, because conda already enforces that numpy-C-API-using-packages > have to be recompiled every time a new numpy release comes out. > Therefore, if we only supported conda, then we would be free to break > ABI and clean up some of the 20 year old broken junk that we have > lying around and add new features more quickly. Conclusion: continuing > to support pip is hobbling innovation in the whole numerical > ecosystem. > IMO this is not compelling *enough* to cut off our many many users who > are not using conda, Agreed. It's also not like those are the only options. If breaking ABI became so valuable that it needs to be done, I'd rather put the burden of that on packagers of projects that rely on numpy and would have to create lots of new installers rather than on users that expect "pip install" to work. Ralf > plus a schism like this would have all kinds of > knock-on costs (the value of a community grows like O(n**2), so > splitting a community is expensive!). And given that you and I are > both on the list of gatekeepers to such a change, yeah, it's not going > to happen in the immediate future. > > But... if conda continues to gain mindshare at pip's expense, and they > fix some of the more controversial sticking points (e.g. the current > reliance on secret proprietary build recipes), and the pip/distutils > side of things continues to stagnate WRT things like this... I dunno, > I could imagine that argument becoming more and more compelling over > the next few years. At that point I'm honestly not sure what happens, > but I suspect that all the options are unpleasant. You and I have a > fair amount of political capital, but it is finite. ...Or maybe I'm > worrying over nothing and everything would be fine, but still, it'd be > nice if we never have to find out because pip etc. get better enough > that the issue goes away. > > What I'm saying is, it's not a coincidence that it was after SciPy > this year that I finally subscribed to distutils-sig :-). > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at gedmin.as Fri Oct 30 04:06:54 2015 From: marius at gedmin.as (Marius Gedminas) Date: Fri, 30 Oct 2015 10:06:54 +0200 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: References: Message-ID: <20151030080654.GB10759@platonas> On Thu, Oct 29, 2015 at 05:11:52PM -0400, Donald Stufft wrote: > On October 29, 2015 at 5:09:45 PM, Marcus Smith (qwcode at gmail.com) wrote: > > help me out here... how can we dynamically construct dependencies as we're > > building wheels today? > > I?m not sure I understand the confusion? since a wheel is created by > executing setup.py, you?d just have your build tool dynamically output > different wheels based on the system you?re building on (or whatever > axis is causing the dynamic dependencies). An example is like: > > https://github.com/pypa/twine/blob/a0c87357d9d5d588082c9a59f6efc6f6bc3d3498/setup.py#L28-L31 install_requires = [ "pkginfo >= 1.0", "requests >= 2.3.0", "requests-toolbelt >= 0.4.0", "setuptools >= 0.7.0", ] if sys.version_info[:2] < (2, 7): install_requires += [ "argparse", ] setup( ... install_requires=install_requires, ) But code like this doesn't work! You build a wheel on Python 2.7, you get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel cache, and when you try to install it on Python 2.6, pip tries to use the same wheel, with install_requires computed for Python 2.7 instead of 2.6. Unless you override the wheel dependencies completely in setup.cfg[+]. [*] https://github.com/pypa/twine/blob/master/setup.cfg#L2: [wheel] universal = 1 [+] https://github.com/pypa/twine/blob/master/setup.cfg#L9-L15 [metadata] requires-dist = requests >= 2.3.0 requests-toolbelt >= 0.4.0 pkginfo >= 1.0 setuptools >= 0.7.0 argparse; python_version == '2.6' Marius Gedminas -- Since this protocol deals with Firewalls there are no real security considerations. -- RFC 3093 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 173 bytes Desc: Digital signature URL: From donald at stufft.io Fri Oct 30 07:04:07 2015 From: donald at stufft.io (Donald Stufft) Date: Fri, 30 Oct 2015 07:04:07 -0400 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: <20151030080654.GB10759@platonas> References: <20151030080654.GB10759@platonas> Message-ID: On October 30, 2015 at 4:07:22 AM, Marius Gedminas (marius at gedmin.as) wrote: > > But code like this doesn't work! You build a wheel on Python 2.7, > you > get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel > cache, and when > you try to install it on Python 2.6, pip tries to use the same wheel, > with install_requires computed for Python 2.7 instead of 2.6. > Unless > you override the wheel dependencies completely in setup.cfg[+].? It will work once?https://github.com/pypa/pip/issues/3025?is fixed, which I plan on making sure is in the next release. It of course won?t work for releasing wheels or whatever, but it?s no worse then we are today. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Fri Oct 30 09:00:25 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Oct 2015 13:00:25 +0000 Subject: [Distutils] metadata in sdists (was Re: Second draft of a plan for a new source tree / sdist format) In-Reply-To: References: Message-ID: (Hugely trimmed, because I coldn't find an easy way to pick out the important bits of context, sorry!) On 29 October 2015 at 23:23, Nathaniel Smith wrote: > None of this affects correctness -- it's purely an optimization. But > maybe it's an important optimization in certain specific cases. One concern I have is that it's *not* just an optimisation in some cases. If a build being used to get metadata fails, what will happen then? If you fail the whole install process, then using your scikit-learn case, suppose there are wheels available for older versions of scipy, but none for the latest version (a very common scenario, in my experience, for a period after a new release appears). Then the dependency resolution tries to build the latest version to get the metadata, fails, and things stop. But the older version is actually fine, because the wheel can be used. You could treat build failures as "assume not suitable", but that could result in someone getting an older version when a compile fails, rather than getting the error (which in less complex cases than the above, they might want so they can fix it - e.g. by setting an environment variable they'd forgotten, or by downloading a wheel from a non-PyPI repository like Christoph Gohlke's). So while I follow your explanation for the cases where builds always succeed but might take forever, I'm not so sure your conclusions are right for a mix of wheels for some versions, failing builds and other partially-working scenarios. This case concerns me far more in practice than complex dependency graphs do. Paul From p.f.moore at gmail.com Fri Oct 30 10:13:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Oct 2015 14:13:19 +0000 Subject: [Distutils] A smaller step towards de-specializing setuptools/distutils In-Reply-To: <20151030080654.GB10759@platonas> References: <20151030080654.GB10759@platonas> Message-ID: On 30 October 2015 at 08:06, Marius Gedminas wrote: > But code like this doesn't work! You build a wheel on Python 2.7, you > get a twine-1.6.4-py2.py3-none-any.whl[*] in your pip wheel cache, and when > you try to install it on Python 2.6, pip tries to use the same wheel, > with install_requires computed for Python 2.7 instead of 2.6. Unless > you override the wheel dependencies completely in setup.cfg[+]. But this is simply an incorrect package definition, surely? The setup.cfg override is correct, as would be overriding the wheel to be py27.py3 (with a separate wheel depending on argparse for py26). The original form specifies a universal wheel, but provides metadata that isn't correct for a universal wheel... Maybe it's not as easy as it could be to *do* this, but nevertheless, it is possible to specify statically the correct metadata. Paul From brett at python.org Fri Oct 30 11:29:12 2015 From: brett at python.org (Brett Cannon) Date: Fri, 30 Oct 2015 15:29:12 +0000 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) In-Reply-To: References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: On Thu, 29 Oct 2015 at 22:53 Marcus Smith wrote: > > >> If python-dev ends up adopting GitLab for the main PEPs repo, then we >> should be able to move the whole process there, rather than needing to >> maintain a separate copy. >> > will that be as open as pypa/interoperability-peps? > Proof of concept and final proposal isn't in front of me yet so it isn't known. I think Nick's hope is that it will be as open. -Brett > if it's closed off such that only python devs can log PRs against PEPs > once they're in the system, then that's no fun. > If so, I'd still want to keep pypa/interoperability-peps as the front end > tool for the actual change management. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pezzoni.mario at gmail.com Fri Oct 30 12:47:20 2015 From: pezzoni.mario at gmail.com (Mario Pezzoni) Date: Fri, 30 Oct 2015 17:47:20 +0100 Subject: [Distutils] Packaging shared objects with setuptools Message-ID: Hello, I am wrapping a c++ library with cython. I compile the pyxs and the c++ code with cmake using the cython-cmake-example. If I import the package from the build directory it works flawlessly, if I install through "python setup.py install" into a virtual environment it breaks because the shared objects are not installed (but all the pure .py modules are installed). How do I tell setuptools to install the shared objects? Mario -------------- next part -------------- An HTML attachment was scrubbed... URL: From waynejwerner at gmail.com Sat Oct 31 09:15:18 2015 From: waynejwerner at gmail.com (Wayne Werner) Date: Sat, 31 Oct 2015 08:15:18 -0500 Subject: [Distutils] Brian Goetz - Stewardship: the Sobering Parts In-Reply-To: References: Message-ID: First, do no harm, eh? (only 14 minutes into it so far) On Thu, Oct 29, 2015 at 3:51 PM, Donald Stufft wrote: > This was shown to me today, and I thought I?d share it here since it > pretty much applies 100% to what we?re doing on this list too (other than > the fact the examples are from Java) - > https://www.youtube.com/watch?v=2y5Pv4yN0b0 > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Oct 31 20:52:51 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Nov 2015 01:52:51 +0100 Subject: [Distutils] Removing the aspirational aspects of PEP 426 In-Reply-To: References: Message-ID: On 29 October 2015 at 02:09, Robert Collins wrote: > On 28 October 2015 at 23:39, Nathaniel Smith wrote: >> On Oct 28, 2015 3:25 AM, "Nick Coghlan" wrote: >>> >> [...] >>> From an sdist metadata perspective, though, I think the right thing to do >>> is to descope PEP 426 to just the stuff we *need* for the build system >>> improvements, and defer everything else (e.g. JSON-LD, SPDX, richer >>> dependency semantics, etc) to a future metadata 3.0 proposal (or potentially >>> metadata extensions, or 2.x format updates). >> >> I think PEP 426 is actually orthogonal to these proposals. AFAICT, the only >> reason Robert's PEP as written requires PEP 426 is that he needs a standard >> serializable format to list dependencies... but he actually defines such a >> format about 10 lines above for the static bootstrap-requirements key, i.e. >> a list of specifier strings. So it actually makes more sense to use that for >> dynamic requirements too for internal consistency, and leave PEP 426 out of >> it. > > pip requires: > - distribution name > - install_requires [+extras] > > today. It will want external dependencies in future (and in the spec I > put forward build dependencies would be obtained earlier so could be > skipped). > > I'd rather not invent a *new* format for handling both of these, but > i'm ok if Donald and Nick specifically are. They're probably a good candidate for the version specifier and environment marker treatment: extracting a "dependency specifier" PEP to give us a building block we can use in the higher level specs. Cheers, Nick. > > -Rob > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Sat Oct 31 21:08:32 2015 From: barry at python.org (Barry Warsaw) Date: Sat, 31 Oct 2015 21:08:32 -0400 Subject: [Distutils] Please don't impose additional barriers to participation (was: build system abstraction PEP) References: <85h9lbmvpg.fsf_-_@benfinney.id.au> Message-ID: <20151031210832.1719d821@anarchist.wooz.org> On Oct 29, 2015, at 10:52 PM, Marcus Smith wrote: >> If python-dev ends up adopting GitLab for the main PEPs repo, then we >> should be able to move the whole process there, rather than needing to >> maintain a separate copy. >> >will that be as open as pypa/interoperability-peps? if it's closed off such >that only python devs can log PRs against PEPs once they're in the system, >then that's no fun. If so, I'd still want to keep pypa/interoperability-peps >as the front end tool for the actual change management. The way I believe it would work via GitLab is that anybody can fork the repo, push branches to their fork, and file PRs against the PEPs repo. Pushing to the PEPs repo would be gated via privileged members of the repo's owner, either via git push or button push (i.e. "hey website, I'm chillin' on the beach with my tablet so do the merge for me!") Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: