From freddyrietdijk at fridh.nl Thu Dec 1 04:45:57 2016 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 1 Dec 2016 10:45:57 +0100 Subject: [Distutils] Maintaining a curated set of Python packages Message-ID: Hi, I would like to propose that, as a community, we jointly maintain a curated set of Python packages that are known to work together. These packages would receive security updates for some time and every couple of months a new major release of the curated set comes available. The idea of this is inspired by Haskell LTS, so maybe we should call this PyPI LTS? So why a PyPI LTS? PyPI makes available all versions of packages that were uploaded, and by default installers like pip will try to use the latest available versions of packages, unless told otherwise. With a requirements.txt file (or a future pipfile.lock) and setup.py we can pin as much as we like our requirements of respectively the environment and package requirements, thereby making a more reproducible environment possible and also fixing the API for developers. Pinning requirements is often a manual job, although one could use pip freeze or other tools. A common problem is when two packages in a certain environment require different versions of a package. Having a curated set of packages, developers could be encouraged to test against the latest stable and nightly of the curated package set, thereby increasing compatibility between different packages, something I think we all want. Having a compatible set of packages is not only interesting for developers, but also for downstream distributions. All distributions try to find a set of packages that are working together and release them. This is a lot of work, and I think it would be in everyone's benefit if we try to solve this issue together. A possible solution Downstream, that is developers and distributions, will need a set of packages that are known to work together. At minimum this would consist of, per package, the name of the package and its version, but for reproducibility I would propose adding the filename and hash as well. Because there isn't any reliable method to extract the requirements of a package, I propose also including `setup_requires`, install_requires`, and `tests_require` explicitly. That way, distributions can automatically build recipes for the packages (although non-Python dependencies would still have to be resolved by the distribution). The package set would be released as lts-YYYY-MM-REVISION, and developers can choose to track a specific revision, but would typically be asked to track only lts-YYYY-MM which would resolve to the latest REVISION. Because dependencies vary per Python language version, interpreter, and operating system, we would have to have these sets for each combination and therefore I propose having a source which evaluates to say a TOML/JSON file per version/interpreter/OS. How this source file should be written I don't know; while I think the Nix expression language is an excellent choice for this, it is not possible for everyone to use and therefore likely not an option. Open questions There are still plenty of open questions. - Who decides when a package is updated that would break dependents? This is an issue all distributions face, so maybe we should involve them. - How would this be integrated with pip / virtualenv / pipfile.lock / requirements.txt / setup.py? See e.g. https://github.com/pypa/pipfile/issues/10#issuecomment-262229620 References to Haskell LTS Here are several links to some interesting documents on how Haskell LTS works. - A blog post describing what Haskell LTS is: https://www.fpcomplete.com/blog/2014/12/backporting-bug-fixes - Rules regarding uploading and breaking packages: https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package - The actual LTS files https://github.com/fpco/lts-haskell What do you think of this proposal? Would you be interested in this as developer, or packager? Freddy -------------- next part -------------- An HTML attachment was scrubbed... URL: From shudusmp at gmail.com Thu Dec 1 07:20:18 2016 From: shudusmp at gmail.com (Mashudu Mphohoni) Date: Thu, 1 Dec 2016 14:20:18 +0200 Subject: [Distutils] Python anywhere problems Message-ID: [image: Inline image 1] Hi, Would you please advise me on my python anywhere, It seems to fail to install or upgrade any packages Thanks and regards, Mashudu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 56981 bytes Desc: not available URL: From p.f.moore at gmail.com Fri Dec 2 08:50:54 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Dec 2016 13:50:54 +0000 Subject: [Distutils] Python anywhere problems In-Reply-To: References: Message-ID: On 1 December 2016 at 12:20, Mashudu Mphohoni wrote: > Would you please advise me on my python anywhere, This list isn't for support of services like Python Anywhere. You should ask Python Anywhere for support. if you're using a paid version, you will have a support arrangement. If you're using the free tier, I guess there will be help on the site. A quick search of their site found https://help.pythonanywhere.com/pages/InstallingNewModules which seems like it might be relevant for you. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From rmcgibbo at gmail.com Fri Dec 2 10:33:25 2016 From: rmcgibbo at gmail.com (Robert T. McGibbon) Date: Fri, 2 Dec 2016 10:33:25 -0500 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: Isn't this issue already solved by (and the raison d'?tre of) the multiple third-party Python redistributors, like the various OS package maintainers, Continuum's Anaconda, Enthought Canopy, ActiveState Python, WinPython, etc? [image: Inline image 1] -Robert On Thu, Dec 1, 2016 at 4:45 AM, Freddy Rietdijk wrote: > Hi, > > I would like to propose that, as a community, we jointly maintain a > curated set of Python packages that are known to work together. These > packages would receive security updates for some time and every couple of > months a new major release of the curated set comes available. The idea of > this is inspired by Haskell LTS, so maybe we should call this PyPI LTS? > > So why a PyPI LTS? > > PyPI makes available all versions of packages that were uploaded, and by > default installers like pip will try to use the latest available versions > of packages, unless told otherwise. With a requirements.txt file (or a > future pipfile.lock) and setup.py we can pin as much as we like our > requirements of respectively the environment and package requirements, > thereby making a more reproducible environment possible and also fixing the > API for developers. Pinning requirements is often a manual job, although > one could use pip freeze or other tools. > > A common problem is when two packages in a certain environment require > different versions of a package. Having a curated set of packages, > developers could be encouraged to test against the latest stable and > nightly of the curated package set, thereby increasing compatibility > between different packages, something I think we all want. > > Having a compatible set of packages is not only interesting for > developers, but also for downstream distributions. All distributions try to > find a set of packages that are working together and release them. This is > a lot of work, and I think it would be in everyone's benefit if we try to > solve this issue together. > > A possible solution > > Downstream, that is developers and distributions, will need a set of > packages that are known to work together. At minimum this would consist of, > per package, the name of the package and its version, but for > reproducibility I would propose adding the filename and hash as well. > Because there isn't any reliable method to extract the requirements of a > package, I propose also including `setup_requires`, install_requires`, and > `tests_require` explicitly. That way, distributions can automatically build > recipes for the packages (although non-Python dependencies would still have > to be resolved by the distribution). > > The package set would be released as lts-YYYY-MM-REVISION, and developers > can choose to track a specific revision, but would typically be asked to > track only lts-YYYY-MM which would resolve to the latest REVISION. > > Because dependencies vary per Python language version, interpreter, and > operating system, we would have to have these sets for each combination and > therefore I propose having a source which evaluates to say a TOML/JSON file > per version/interpreter/OS. > How this source file should be written I don't know; while I think the Nix > expression language is an excellent choice for this, it is not possible for > everyone to use and therefore likely not an option. > > Open questions > > There are still plenty of open questions. > > - Who decides when a package is updated that would break dependents? This > is an issue all distributions face, so maybe we should involve them. > - How would this be integrated with pip / virtualenv / pipfile.lock / > requirements.txt / setup.py? See e.g. https://github.com/pypa/ > pipfile/issues/10#issuecomment-262229620 > > References to Haskell LTS > > Here are several links to some interesting documents on how Haskell LTS > works. > - A blog post describing what Haskell LTS is: https://www.fpcomplete. > com/blog/2014/12/backporting-bug-fixes > - Rules regarding uploading and breaking packages: https://github.com/ > fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package > - The actual LTS files https://github.com/fpco/lts-haskell > > > What do you think of this proposal? Would you be interested in this as > developer, or packager? > > > Freddy > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- -Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 51800 bytes Desc: not available URL: From ncoghlan at gmail.com Fri Dec 2 11:23:09 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Dec 2016 02:23:09 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: On 3 December 2016 at 01:33, Robert T. McGibbon wrote: > Isn't this issue already solved by (and the raison d'?tre of) the multiple > third-party Python redistributors, like the various OS package maintainers, > Continuum's Anaconda, Enthought Canopy, ActiveState Python, WinPython, etc? > Yep. Once you start talking content curation, you're in a situation where: - you're providing an ongoing service that will always be needed (the specific packages will change, but the task won't) - you need to commit to a certain level of responsiveness for security issues - you'll generally need to span multiple language ecosystems, not just Python - exactly which packages are interesting will depend on the user audience you're targeting - the tolerance for API breakage will also vary based on the audience you're targeting - you'll often want to be able to carry patches that aren't present in the upstream components - you'll need to decide which target platforms you want to support If a curation community *isn't* doing any of those things, then it isn't adding a lot of value beyond folks just doing DIY integration in their CI system by pinning their dependencies to particular versions. As far as the comments about determining dependencies goes, the way pip does it generally works fine, you just need a sandboxed environment to do the execution, and both redistributors and open source information providers like libraries.io are actively working on automating that process (coping with packages as they exist on PyPI today, rather than relying on the upstream community to change anything about the way Python packaging works). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Fri Dec 2 12:34:20 2016 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Fri, 2 Dec 2016 18:34:20 +0100 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: On Fri, Dec 2, 2016 at 4:33 PM, Robert T. McGibbon wrote: > Isn't this issue already solved by (and the raison d'?tre of) the multiple > third-party Python redistributors, like the various OS package maintainers, > Continuum's Anaconda, Enthought Canopy, ActiveState Python, WinPython, etc? > My intention is not creating yet another distribution. Instead, I want to see if there is interest in the different distributions on sharing some of the burden of curating by bringing up this discussion and seeing what is needed. These distributions have their recipes that allow them to build their packages using their tooling. What I propose is having some of that data community managed so the distributions can use that along with their tooling to build the eventual packages. On Fri, Dec 2, 2016 at 5:23 PM, Nick Coghlan wrote: > On 3 December 2016 at 01:33, Robert T. McGibbon > wrote: > >> Isn't this issue already solved by (and the raison d'?tre of) the >> multiple third-party Python redistributors, like the various OS package >> maintainers, Continuum's Anaconda, Enthought Canopy, ActiveState Python, >> WinPython, etc? >> > > Yep. Once you start talking content curation, you're in a situation where: > > - you're providing an ongoing service that will always be needed (the > specific packages will change, but the task won't) > - you need to commit to a certain level of responsiveness for security > issues > - you'll generally need to span multiple language ecosystems, not just > Python > - exactly which packages are interesting will depend on the user audience > you're targeting > - the tolerance for API breakage will also vary based on the audience > you're targeting > - you'll often want to be able to carry patches that aren't present in the > upstream components > - you'll need to decide which target platforms you want to support > These are interesting issues you bring up here. What I seek is having a set that has per package a version, source, Python dependencies and build system. Other dependencies would be for now left out, unless someone has a good idea how to include those. Distributions can take this curated set and extend the data with their distribution specific things. For example, in Nix we could load such a set, map a function that builds the packages in the set, and override what is passed to the function when necessary (e.g. to add system dependencies, our patches, or how tests are invoked, and so on). Responsiveness is indeed an interesting issue. If there's enough backing, then I imagine security issues will be resolved as fast as they are nowadays by the distributions backing the initiative. > If a curation community *isn't* doing any of those things, then it isn't > adding a lot of value beyond folks just doing DIY integration in their CI > system by pinning their dependencies to particular versions. > > I would imagine that distributions that would support this idea would have a CI tracking packages built using the curated set and the distribution-specific changes. When there's an issue they could fix it at their side, or if it is something that might belong in the curated set, they would report the issue. At some point, when they would freeze, they would pin to a certain YYYY.MM and API breakage should not occur. > As far as the comments about determining dependencies goes, the way pip > does it generally works fine, you just need a sandboxed environment to do > the execution, and both redistributors and open source information > providers like libraries.io are actively working on automating that > process (coping with packages as they exist on PyPI today, rather than > relying on the upstream community to change anything about the way Python > packaging works). > > libraries.io is a very interesting initiative. It seems they scan the contents of the archives and extract dependencies based on what is in the requirements files, which is often more than is actually needed for building and running the package. They would benefit from having a declarative style for the dependencies and build system, but that is another issue (PEP 517 e.g.) than what I bring up here. We also have a tool that runs pip in a sandbox to determine the dependencies, and then provide us with an expression. It works, but it shouldn't be necessary. Freddy -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 2 23:39:01 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Dec 2016 14:39:01 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: On 3 December 2016 at 03:34, Freddy Rietdijk wrote: > > On Fri, Dec 2, 2016 at 4:33 PM, Robert T. McGibbon > wrote: >> >> Isn't this issue already solved by (and the raison d'?tre of) the multiple >> third-party Python redistributors, like the various OS package maintainers, >> Continuum's Anaconda, Enthought Canopy, ActiveState Python, WinPython, etc? > > > My intention is not creating yet another distribution. Instead, I want to > see if there is interest in the different distributions on sharing some of > the burden of curating by bringing up this discussion and seeing what is > needed. These distributions have their recipes that allow them to build > their packages using their tooling. What I propose is having some of that > data community managed so the distributions can use that along with their > tooling to build the eventual packages. There's definitely interest in more automated curation such that publishing through PyPI means you get pre-built binary artifacts and compatibility testing for popular platforms automatically, but the hard part of that isn't really the technical aspects, it's developing a robust funding and governance model for the related sustaining engineering activities. That upstream component level "yes it builds" and "yes it passes its self-tests" data is then useful to redistributors, since it would make it straightforward to filter out releases that don't even build or pass their own tests even before they make it into a downstream review pipeline. > These are interesting issues you bring up here. What I seek is having a set > that has per package a version, source, Python dependencies and build > system. Other dependencies would be for now left out, unless someone has a > good idea how to include those. Distributions can take this curated set and > extend the data with their distribution specific things. For example, in Nix > we could load such a set, map a function that builds the packages in the > set, and override what is passed to the function when necessary (e.g. to add > system dependencies, our patches, or how tests are invoked, and so on). Something that could be useful on that front is to mine the stdlib documentation for "seealso" references to third party libraries and collect them into an automation-friendly reference API. The benefit of that approach is that it: - would be immediately useful in its own right as a "stdlib++" definition - solves the scope problem (the problem tackled has to be common enough to have a default solution in the standard library, but complex enough that there are recommended alternatives) - solves the governance problem (the approval process for new entries is to get them referenced from the relevant stdlib module documentation) > Responsiveness is indeed an interesting issue. If there's enough backing, > then I imagine security issues will be resolved as fast as they are nowadays > by the distributions backing the initiative. Not necessarily, as many of those responsiveness guarantees rely on the ability of redistributors to carry downstream patches, even before there's a corresponding upstream security release. This is especially so for upstream projects that follow an as-needed release model, without much (if any) automation of their publication process. >> If a curation community *isn't* doing any of those things, then it isn't >> adding a lot of value beyond folks just doing DIY integration in their CI >> system by pinning their dependencies to particular versions. > > I would imagine that distributions that would support this idea would have a > CI tracking packages built using the curated set and the > distribution-specific changes. When there's an issue they could fix it at > their side, or if it is something that might belong in the curated set, they > would report the issue. At some point, when they would freeze, they would > pin to a certain YYYY.MM and API breakage should not occur. Yeah, this is effectively what happens already, it's just not particularly visible outside the individual redistributor pipelines. > libraries.io is a very interesting initiative. It seems they scan the > contents of the archives and extract dependencies based on what is in the > requirements files, which is often more than is actually needed for building > and running the package. They would benefit from having a declarative style > for the dependencies and build system, but that is another issue (PEP 517 > e.g.) than what I bring up here. We also have a tool that runs pip in a > sandbox to determine the dependencies, and then provide us with an > expression. It works, but it shouldn't be necessary. Alas, with 94k+ setup.py based packages already in the wild, arbitrary code execution for dependency metadata generation is going to be with us for a while. That said, centralised services like libraries.io should lead to more folks being able to just use their already collected dependency data (even if it isn't the minimal dependency set) and avoid having to generate it themselves. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Dec 7 02:49:00 2016 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Dec 2016 07:49:00 +0000 (UTC) Subject: [Distutils] PyCharm / JetBrains licence References: <1435760091.1537710.1481096940858.ref@mail.yahoo.com> Message-ID: <1435760091.1537710.1481096940858@mail.yahoo.com> Dear Christian, You can access the PyCharm and other JetBrains products licence at the following link: https://account.jetbrains.com/a/53fr5cnq If you don't already have a JetBrains account, you will need to create one to activate the licence. Regards, Vinay Sajip -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Wed Dec 7 06:37:25 2016 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 7 Dec 2016 11:37:25 +0000 (UTC) Subject: [Distutils] JetBrains licences References: <666524866.1897684.1481110645668.ref@mail.yahoo.com> Message-ID: <666524866.1897684.1481110645668@mail.yahoo.com> I accidentally cc-ed this list when sending email to a Python committer about JetBrains licences. Please don't take up one of these licences unless you are a Python committer; I will remove access from any person who is not named in?https://hg.python.org/committers.txt? I apologise for any confusion I have caused. Regards, Vinay Sajip -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Dec 8 10:10:24 2016 From: barry at python.org (Barry Warsaw) Date: Thu, 8 Dec 2016 16:10:24 +0100 Subject: [Distutils] Maintaining a curated set of Python packages References: Message-ID: <20161208161024.682f26dc@presto> On Dec 01, 2016, at 10:45 AM, Freddy Rietdijk wrote: >Having a compatible set of packages is not only interesting for developers, >but also for downstream distributions. All distributions try to find a set >of packages that are working together and release them. This is a lot of >work, and I think it would be in everyone's benefit if we try to solve this >issue together. It's an interesting but difficult problem at the level of PyPI. It's difficult because the definition of compatibility is highly dependent on the consumer's environment. For example, C extension compatibility will depend on the version of libraries available on the platform versions you care about. There are also dependents on Python libraries that you can't capture only in PyPI, e.g. some distro-only package may depend on a particular Python package API. PyPI can't test any of this. And some distros (e.g. Ubuntu) may have multiple dimensions of consumability, e.g. the classic apt packages vs. snaps, primary devel archive vs. stable distro versions vs. backports, etc. Ubuntu has an elaborate automated system for testing some dimension of compatibility issues between packages, not just Python packages. Debian has the same system but isn't gated on the results. Individual distro packages can include a set of tests that are run against the built version of the package (as opposed to tests run at package build time). When a new version of that package is uploaded it ends up in a "proposed" pocket that is generally not installed by users. The proposed new version has its tests run, and any package that depends on that package also has its tests run. Only when all those tests pass is the package automatically promoted to the release pocket, and thus is installable by the general user population. This does a great job within the context of the distro of raising the quality of the archive because obvious incompatibilities (i.e. those for which tests exist) block such promotion. (FWIW, the difference is that Debian doesn't block promotion of packages failing their automated tests, so 1) it provides less value to Debian; 2) we end up inheriting and seeing these problems in Ubuntu and so have to expend downstream effort to fix the failures.) All of this happens automatically within the context of the distro, on multiple architectures, so it provides a lot of value, but I'm not sure how useful it would be higher up the food chain, since those contexts will be different enough to cause both false positives and false negatives. And it does often take quite a bit of focused engineering effort to monitor packages which don't promote (something we want to automate), to actually fix the problems wherever is most appropriate (and as far upstream as possible), and to create meaningful tests of compatibility in the first place (except for default tests such as installability). Still, there may be value in inter-Python package compatibility tests, but it'll take serious engineering effort (i.e. $ and time), ongoing maintenance, ongoing effort to fix problems, and tooling to gate installability of failing packages (with overrides for downstreams which don't care or already expend such effort). Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From wes.turner at gmail.com Thu Dec 8 10:56:38 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Dec 2016 09:56:38 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: On Thursday, December 1, 2016, Freddy Rietdijk wrote: > Hi, > > I would like to propose that, as a community, we jointly maintain a > curated set of Python packages that are known to work together. These > packages would receive security updates for some time and every couple of > months a new major release of the curated set comes available. The idea of > this is inspired by Haskell LTS, so maybe we should call this PyPI LTS? > > So why a PyPI LTS? > > PyPI makes available all versions of packages that were uploaded, and by > default installers like pip will try to use the latest available versions > of packages, unless told otherwise. With a requirements.txt file (or a > future pipfile.lock) and setup.py we can pin as much as we like our > requirements of respectively the environment and package requirements, > thereby making a more reproducible environment possible and also fixing the > API for developers. Pinning requirements is often a manual job, although > one could use pip freeze or other tools. > https://github.com/nvie/pip-tools : - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock) - I can't remember whether pip-compile includes the checksum in the compiled requirements.txt https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode - Current recommendation: sha256 - (These are obviously different for platform-specific wheels and bdists) https://github.com/rbanffy/pip-chill - Unlike pip freeze, pip-chill includes only top-level deps > > > > A common problem is when two packages in a certain environment require > different versions of a package. Having a curated set of packages, > developers could be encouraged to test against the latest stable and > nightly of the curated package set, thereby increasing compatibility > between different packages, something I think we all want. > > Having a compatible set of packages is not only interesting for > developers, but also for downstream distributions. All distributions try to > find a set of packages that are working together and release them. This is > a lot of work, and I think it would be in everyone's benefit if we try to > solve this issue together. > I think conda has already been mentioned. - environment.yml : http://conda.pydata.org/docs/using/envs.html#use-environment-from-file https://conda-forge.github.io - "A community led collection of recipes, build infrastructure and distributions for the conda package manager." - "AppVeyor, CircleCI and TravisCI" > > A possible solution > > Downstream, that is developers and distributions, will need a set of > packages that are known to work together. At minimum this would consist of, > per package, the name of the package and its version, but for > reproducibility I would propose adding the filename and hash as well. > Because there isn't any reliable method to extract the requirements of a > package, I propose also including `setup_requires`, install_requires`, and > `tests_require` explicitly. That way, distributions can automatically build > recipes for the packages (although non-Python dependencies would still have > to be resolved by the distribution). > > The package set would be released as lts-YYYY-MM-REVISION, and developers > can choose to track a specific revision, but would typically be asked to > track only lts-YYYY-MM which would resolve to the latest REVISION. > > > Because dependencies vary per Python language version, interpreter, and > operating system, we would have to have these sets for each combination and > therefore I propose having a source which evaluates to say a TOML/JSON file > per version/interpreter/OS. > How this source file should be written I don't know; while I think the Nix > expression language is an excellent choice for this, it is not possible for > everyone to use and therefore likely not an option. > YAML: environment.yml, meta.yaml - http://conda.pydata.org/docs/building/meta-yaml.html#requirements-section - http://conda.pydata.org/docs/building/meta-yaml.html#test-section Could/would there be a package with an integration test suite in tests/? Practically, a developer would want a subset of the given known-good-set (and then additional packages), so: - fork/copy requirements-YYYY-MM-REV--.txt - #comment out unused deps - add '-r addl-requirements.txt' > > > Open questions > > There are still plenty of open questions. > > - Who decides when a package is updated that would break dependents? This > is an issue all distributions face, so maybe we should involve them. > IDK if e.g. https://requires.io can post to a mailing list? - "Stop wasting your time by manually keeping track of changelogs. Requires.io keeps your python projects secure by monitoring their dependencies." - Source: https://github.com/requires - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin > - How would this be integrated with pip / virtualenv / pipfile.lock / > requirements.txt / setup.py? See e.g. https://github.com/pypa/ > pipfile/issues/10#issuecomment-262229620 > - https://tox.readthedocs.io/en/latest/ - tox.ini - http://doc.devpi.net/latest/ - http://doc.devpi.net/latest/quickstart-releaseprocess.html#devpi-test-testing-an-uploaded-package > > References to Haskell LTS > > Here are several links to some interesting documents on how Haskell LTS > works. > - A blog post describing what Haskell LTS is: https://www.fpcomplete. > com/blog/2014/12/backporting-bug-fixes > - Rules regarding uploading and breaking packages: https://github.com/ > fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package > - The actual LTS files https://github.com/fpco/lts-haskell > > > What do you think of this proposal? Would you be interested in this as > developer, or packager? > Given time to work on this, I'd probably spend it on developing a new or existing (?) comprehensive integration test suite for an already-maintained package set: https://docs.continuum.io/anaconda/pkg-docs https://www.enthought.com/products/canopy/package-index/ https://github.com/audreyr/cookiecutter-pypackage/blob/master/%7B%7Bcookiecutter.project_slug%7D%7D/tests/test_%7B%7Bcookiecutter.project_slug%7D%7D.py > > Freddy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 8 11:05:47 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Dec 2016 10:05:47 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: On Thursday, December 8, 2016, Wes Turner wrote: > > > On Thursday, December 1, 2016, Freddy Rietdijk > wrote: > >> Hi, >> >> I would like to propose that, as a community, we jointly maintain a >> curated set of Python packages that are known to work together. These >> packages would receive security updates for some time and every couple of >> months a new major release of the curated set comes available. The idea of >> this is inspired by Haskell LTS, so maybe we should call this PyPI LTS? >> >> So why a PyPI LTS? >> >> PyPI makes available all versions of packages that were uploaded, and by >> default installers like pip will try to use the latest available versions >> of packages, unless told otherwise. With a requirements.txt file (or a >> future pipfile.lock) and setup.py we can pin as much as we like our >> requirements of respectively the environment and package requirements, >> thereby making a more reproducible environment possible and also fixing the >> API for developers. Pinning requirements is often a manual job, although >> one could use pip freeze or other tools. >> > > https://github.com/nvie/pip-tools : > > - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock) > - I can't remember whether pip-compile includes the checksum in the > compiled requirements.txt > > https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode > > - Current recommendation: sha256 > - (These are obviously different for platform-specific wheels and bdists) > > https://github.com/rbanffy/pip-chill > > - Unlike pip freeze, pip-chill includes only top-level deps > > > >> >> >> >> A common problem is when two packages in a certain environment require >> different versions of a package. Having a curated set of packages, >> developers could be encouraged to test against the latest stable and >> nightly of the curated package set, thereby increasing compatibility >> between different packages, something I think we all want. >> >> Having a compatible set of packages is not only interesting for >> developers, but also for downstream distributions. All distributions try to >> find a set of packages that are working together and release them. This is >> a lot of work, and I think it would be in everyone's benefit if we try to >> solve this issue together. >> > > I think conda has already been mentioned. > > - environment.yml : http://conda.pydata.org/docs/using/envs.html#use- > environment-from-file > > https://conda-forge.github.io > > - "A community led collection of recipes, build infrastructure and > distributions for the conda package manager." > - "AppVeyor, CircleCI and TravisCI" > > >> >> A possible solution >> >> Downstream, that is developers and distributions, will need a set of >> packages that are known to work together. At minimum this would consist of, >> per package, the name of the package and its version, but for >> reproducibility I would propose adding the filename and hash as well. >> Because there isn't any reliable method to extract the requirements of a >> package, I propose also including `setup_requires`, install_requires`, and >> `tests_require` explicitly. That way, distributions can automatically build >> recipes for the packages (although non-Python dependencies would still have >> to be resolved by the distribution). >> >> The package set would be released as lts-YYYY-MM-REVISION, and developers >> can choose to track a specific revision, but would typically be asked to >> track only lts-YYYY-MM which would resolve to the latest REVISION. >> >> >> Because dependencies vary per Python language version, interpreter, and >> operating system, we would have to have these sets for each combination and >> therefore I propose having a source which evaluates to say a TOML/JSON file >> per version/interpreter/OS. >> How this source file should be written I don't know; while I think the >> Nix expression language is an excellent choice for this, it is not possible >> for everyone to use and therefore likely not an option. >> > > YAML: environment.yml, meta.yaml > > - http://conda.pydata.org/docs/building/meta-yaml.html# > requirements-section > - http://conda.pydata.org/docs/building/meta-yaml.html#test-section > > Could/would there be a package with an integration test suite in tests/? > > Practically, a developer would want a subset of the given known-good-set > (and then additional packages), so: > > - fork/copy requirements-YYYY-MM-REV--.txt > - #comment out unused deps > - add '-r addl-requirements.txt' > > >> >> >> Open questions >> >> There are still plenty of open questions. >> >> - Who decides when a package is updated that would break dependents? This >> is an issue all distributions face, so maybe we should involve them. >> > > IDK if e.g. https://requires.io can post to a mailing list? > > - "Stop wasting your time by manually keeping track of changelogs. > Requires.io keeps your python projects secure by monitoring their > dependencies." > - Source: https://github.com/requires > - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin > > > > >> - How would this be integrated with pip / virtualenv / pipfile.lock / >> requirements.txt / setup.py? See e.g. https://github.com/pypa/p >> ipfile/issues/10#issuecomment-262229620 >> > > - https://tox.readthedocs.io/en/latest/ > - tox.ini > - http://doc.devpi.net/latest/ > - http://doc.devpi.net/latest/quickstart-releaseprocess. > html#devpi-test-testing-an-uploaded-package > - Such a known-good-set could be hosted as a package index w/ devpi http://docs.openstack.org/developer/pbr/ - pbr does away with setup.py and install_requires in favor of just requirements.txt > > > >> >> References to Haskell LTS >> >> Here are several links to some interesting documents on how Haskell LTS >> works. >> - A blog post describing what Haskell LTS is: https://www.fpcomplete.com >> /blog/2014/12/backporting-bug-fixes >> - Rules regarding uploading and breaking packages: https://github.com/f >> pco/stackage/blob/master/MAINTAINERS.md#adding-a-package >> - The actual LTS files https://github.com/fpco/lts-haskell >> > Haskell ... pandoc ... https://sites.google.com/site/pydatalog/ > >> >> What do you think of this proposal? Would you be interested in this as >> developer, or packager? >> > > Given time to work on this, I'd probably spend it on developing a new or > existing (?) comprehensive integration test suite for an already-maintained > package set: > > https://docs.continuum.io/anaconda/pkg-docs > > https://www.enthought.com/products/canopy/package-index/ > > https://github.com/audreyr/cookiecutter-pypackage/blob/ > master/%7B%7Bcookiecutter.project_slug%7D%7D/tests/test_ > %7B%7Bcookiecutter.project_slug%7D%7D.py > > >> >> Freddy >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Dec 8 19:15:14 2016 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 9 Dec 2016 00:15:14 +0000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: Message-ID: <20161209001513.GP24597@yuggoth.org> On 2016-12-08 10:05:47 -0600 (-0600), Wes Turner wrote: [...] > http://docs.openstack.org/developer/pbr/ > > - pbr does away with setup.py and install_requires in favor of just > requirements.txt [...] It doesn't entirely "do away with setup.py" (it still relies on a relatively minimal boilerplate setup.py which loads its setuptools entrypoint), but does allow you to basically abstract away most common configuration into declarative setup.cfg and requirements.txt files (similar to some of the pyproject.toml use cases Donald et al have for PEP 518). -- Jeremy Stanley From ncoghlan at gmail.com Thu Dec 8 22:15:01 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Dec 2016 13:15:01 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <20161208161024.682f26dc@presto> References: <20161208161024.682f26dc@presto> Message-ID: Putting the conclusion first, I do see value in better publicising "Recommended libraries" based on some automated criteria like: - recommended in the standard library documentation - available via 1 or more cross-platform commercial Python redistributors - available via 1 or more Linux distro vendors - available via 1 or more web service development platforms That would be a potentially valuable service for folks new to the world of open source that are feeling somewhat overwhelmed by the sheer number of alternatives now available to them. However, I also think that would better fit in with the aims of an open source component tracking community like libraries.io than it does a publisher-centric community like distutils-sig. The further comments below are just a bit more background on why I feel the integration testing aspect of the suggestion isn't likely to be particularly beneficial :) On 9 December 2016 at 01:10, Barry Warsaw wrote: > Still, there may be value in inter-Python package compatibility tests, but > it'll take serious engineering effort (i.e. $ and time), ongoing maintenance, > ongoing effort to fix problems, and tooling to gate installability of failing > packages (with overrides for downstreams which don't care or already expend > such effort). I think this is really the main issue, as both desktop and server environments are moving towards the integrated platform + isolated applications approach popularised by mobile devices. That means we end up with two very different variants of automated integration testing: - the application focused kind offered by the likes of requires.io and pyup.io (i.e. monitor for dependency updates, submit PRs to trigger app level CI) - the platform focused kind employed by distro vendors (testing all the platform components work together, including the app isolation features) The first kind makes sense if you're building something that runs *on* platforms (Docker containers, Snappy or FlatPak apps, web services, mobile apps, etc). The second kind inevitably ends up intertwined with the component review and release engineering systems of the particular platform, so it becomes really hard to collaborate cross-platform outside the context of specific projects like OpenStack that provide clear definitions for "What components do we collectively depend on that we need to test together?" and "What does 'working' mean in the context of this project?". Accordingly, for an initiative like this to be successful, it would need to put some thought up front into the questions of: 1. Who are the intended beneficiaries of the proposal? 2. What problem does it address that will prompt them to contribute time and/or money to solving it? 3. What do we expect people to be able to *stop doing* if the project proves successful? For platform providers, a generic "stdlib++" project wouldn't really reduce the amount of integration testing we'd need to do ourselves (we already don't test arbitrary combinations of dependencies, just the ones we provide at any given point in time). For application and service developers, the approach of pinning dependencies to specific versions and treating updates like any other source code change already works well in most cases. That leaves library and framework developers, who currently tend to adopt the policy of "for each version of Python that we support, we test against the latest versions of our dependencies that were available at the time we ran the test", leaving testing against older versions to platform providers. If there's a key related framework that also provides LTS versions (e.g. Django), then some folks may add that to their test matrix as well. In that context, "Only breaks backwards compatibility for compelling reasons" becomes a useful long term survival trait for libraries and frameworks, as gratuitous breakages are likely to lead to people migrating away from particularly unreliable dependencies. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ben+python at benfinney.id.au Thu Dec 8 22:41:53 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Dec 2016 14:41:53 +1100 Subject: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages) References: <20161209001513.GP24597@yuggoth.org> Message-ID: <857f793h1a.fsf_-_@benfinney.id.au> Jeremy Stanley writes: > [the ?pbr? library] does allow you to basically abstract away most > common configuration into declarative setup.cfg and requirements.txt > files Hmm. That description sounds like a mistaken conflation of two things that should be distinct: * Declaration in Setuptools metadata of what versions of dependencies this distribution is *compatible with*. This purpose is served by Distutils ?install_requires? (and ?test_requires?, etc.). It is characterised by specifying a range of versions for each dependency, for allowing dependency resolvers to have options to choose from. foo >=1.2, <3.0 * Declaration in Pip metadata for what *exact version* of each dependency I want to deploy. This purpose is served by Pip ?requires.txt? input. It is characterised by pinning a *single* version of each dependency, for a deterministic, repeatable deployment. foo == 1.4.7 If we're saying ?pbr? encourages the use of a single set of declarations for those quite different purposes, that sounds like an attractive nuisance. For those who haven't read it, see this post from Donald Stufft for why those purposes need to be kept distinct: There?s a lot of misunderstanding between setup.py and requirements.txt and their roles. A lot of people have felt they are duplicated information and have even created tools to handle this ?duplication?. -- \ ?Corporation, n. An ingenious device for obtaining individual | `\ profit without individual responsibility.? ?Ambrose Bierce, | _o__) _The Devil's Dictionary_, 1906 | Ben Finney From robertc at robertcollins.net Thu Dec 8 23:32:25 2016 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 9 Dec 2016 17:32:25 +1300 Subject: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages) In-Reply-To: <857f793h1a.fsf_-_@benfinney.id.au> References: <20161209001513.GP24597@yuggoth.org> <857f793h1a.fsf_-_@benfinney.id.au> Message-ID: On 9 Dec 2016 4:42 PM, "Ben Finney" wrote: Jeremy Stanley writes: > [the ?pbr? library] does allow you to basically abstract away most > common configuration into declarative setup.cfg and requirements.txt > files Hmm. That description sounds like a mistaken conflation of two things that should be distinct It's not, though the name of the file it looks for is indeed evocative of concrete version lists. Requirements.txt was on the path to being deprecated in order to reduce confusion, but I'm no longer paid to work on that tool chain, so it's now in my lengthy best effort to-do list. Requirements.txt files are mapped directly into install-requires by pbr, and the replacement is to put the same dependencies into setup.cfg. * ... If we're saying ?pbr? encourages the use of a single set of declarations for those quite different purposes, that sounds like an attractive nuisance It doesn't, confusing names aside. We even wrote a tool - generate-constraints to calculate the transitive closure, python version specific as needed, for placing into a constraints / requires file for pip to consume. Rob -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Dec 8 23:45:08 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 8 Dec 2016 23:45:08 -0500 Subject: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages) In-Reply-To: <857f793h1a.fsf_-_@benfinney.id.au> References: <20161209001513.GP24597@yuggoth.org> <857f793h1a.fsf_-_@benfinney.id.au> Message-ID: <34B17AC5-05F3-4CFC-A40A-B54BBAEF2493@stufft.io> > On Dec 8, 2016, at 10:41 PM, Ben Finney wrote: > > For those who haven't read it, see this post from Donald Stufft for why > those purposes need to be kept distinct Somewhat funnily, pbr is what triggered me to actually write that post :) If I recall, at the time it only supported requirements.txt but it?s since then gotten support for putting it in setup.cfg and I think that is preferred now? ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Thu Dec 8 23:50:11 2016 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 9 Dec 2016 17:50:11 +1300 Subject: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages) In-Reply-To: References: <20161209001513.GP24597@yuggoth.org> <857f793h1a.fsf_-_@benfinney.id.au> <34B17AC5-05F3-4CFC-A40A-B54BBAEF2493@stufft.io> Message-ID: On 9 Dec 2016 5:45 PM, "Donald Stufft" wrote: On Dec 8, 2016, at 10:41 PM, Ben Finney wrote: For those who haven't read it, see this post from Donald Stufft for why those purposes need to be kept distinct Somewhat funnily, pbr is what triggered me to actually write that post :) If I recall, at the time it only supported requirements.txt but it?s since then gotten support for putting it in setup.cfg and I think that is preferred now? It always had support for install requires in setup.cfg from its d2to1 heritage. That said early days of pbr were confused about a bunch of things... It's much happier and less layer breaking than it was... :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Dec 9 00:27:43 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Dec 2016 23:27:43 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: On Thursday, December 8, 2016, Nick Coghlan wrote: > Putting the conclusion first, I do see value in better publicising > "Recommended libraries" based on some automated criteria like: > > - recommended in the standard library documentation > - available via 1 or more cross-platform commercial Python redistributors > - available via 1 or more Linux distro vendors > - available via 1 or more web service development platforms > > So these would be attributes tracked by a project maintainer and verified by the known-good-set maintainer? Or? (Again, here I reach for JSONLD. "count n" is only so useful; *which* {[re]distros, platforms, heartfelt testimonials from incredible experts} URLs ) - test coverage - seclist contact info AND procedures - more than one super admin maintainer - what other criteria should/could/can we use to vet open source libraries? > That would be a potentially valuable service for folks new to the > world of open source that are feeling somewhat overwhelmed by the > sheer number of alternatives now available to them. > > However, I also think that would better fit in with the aims of an > open source component tracking community like libraries.io than it > does a publisher-centric community like distutils-sig. IDK if libraries are really in scope for stackshare. The feature upcoming/down voting is pretty cool. https://stackshare.io/python > > The further comments below are just a bit more background on why I > feel the integration testing aspect of the suggestion isn't likely to > be particularly beneficial :) A catch-all for testing bits from application-specific integration test suites could be useful (and would likely require at least docker-compose, dox, kompose for working with actual data stores) > > On 9 December 2016 at 01:10, Barry Warsaw > > wrote: > > Still, there may be value in inter-Python package compatibility tests, > but > > it'll take serious engineering effort (i.e. $ and time), ongoing > maintenance, > > ongoing effort to fix problems, and tooling to gate installability of > failing > > packages (with overrides for downstreams which don't care or already > expend > > such effort). > > I think this is really the main issue, as both desktop and server > environments are moving towards the integrated platform + isolated > applications approach popularised by mobile devices. > > That means we end up with two very different variants of automated > integration testing: > > - the application focused kind offered by the likes of requires.io and > pyup.io (i.e. monitor for dependency updates, submit PRs to trigger > app level CI) > - the platform focused kind employed by distro vendors (testing all > the platform components work together, including the app isolation > features) > > The first kind makes sense if you're building something that runs *on* > platforms (Docker containers, Snappy or FlatPak apps, web services, > mobile apps, etc). > > The second kind inevitably ends up intertwined with the component > review and release engineering systems of the particular platform, so > it becomes really hard to collaborate cross-platform outside the > context of specific projects like OpenStack that provide clear > definitions for "What components do we collectively depend on that we > need to test together?" and "What does 'working' mean in the context > of this project?". > > Accordingly, for an initiative like this to be successful, it would > need to put some thought up front into the questions of: > > 1. Who are the intended beneficiaries of the proposal? > 2. What problem does it address that will prompt them to contribute > time and/or money to solving it? > 3. What do we expect people to be able to *stop doing* if the project > proves successful? > > For platform providers, a generic "stdlib++" project wouldn't really > reduce the amount of integration testing we'd need to do ourselves (we > already don't test arbitrary combinations of dependencies, just the > ones we provide at any given point in time). > > For application and service developers, the approach of pinning > dependencies to specific versions and treating updates like any other > source code change already works well in most cases. > > That leaves library and framework developers, who currently tend to > adopt the policy of "for each version of Python that we support, we > test against the latest versions of our dependencies that were > available at the time we ran the test", leaving testing against older > versions to platform providers. If there's a key related framework > that also provides LTS versions (e.g. Django), then some folks may add > that to their test matrix as well. > > In that context, "Only breaks backwards compatibility for compelling > reasons" becomes a useful long term survival trait for libraries and > frameworks, as gratuitous breakages are likely to lead to people > migrating away from particularly unreliable dependencies. Sometimes, when there are no active maintainers for a feature, it makes sense to deprecate and/or split that functionality / untested cruft out to a different package. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, > Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Dec 9 00:32:35 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 8 Dec 2016 23:32:35 -0600 Subject: [Distutils] Different purposes for Setuptools requirements versus Pip requirements (was: Maintaining a curated set of Python packages) In-Reply-To: <857f793h1a.fsf_-_@benfinney.id.au> References: <20161209001513.GP24597@yuggoth.org> <857f793h1a.fsf_-_@benfinney.id.au> Message-ID: On Thursday, December 8, 2016, Ben Finney > wrote: > Jeremy Stanley writes: > > > [the ?pbr? library] does allow you to basically abstract away most > > common configuration into declarative setup.cfg and requirements.txt > > files > > Hmm. That description sounds like a mistaken conflation of two things > that should be distinct: > > * Declaration in Setuptools metadata of what versions of dependencies > this distribution is *compatible with*. > > This purpose is served by Distutils ?install_requires? (and > ?test_requires?, etc.). It is characterised by specifying a range of > versions for each dependency, for allowing dependency resolvers to > have options to choose from. > > foo >=1.2, <3.0 > > * Declaration in Pip metadata for what *exact version* of each > dependency I want to deploy. > > This purpose is served by Pip ?requires.txt? input. It is > characterised by pinning a *single* version of each dependency, for a > deterministic, repeatable deployment. > > foo == 1.4.7 > > If we're saying ?pbr? encourages the use of a single set of declarations > for those quite different purposes, that sounds like an attractive > nuisance. > > For those who haven't read it, see this post from Donald Stufft for why > those purposes need to be kept distinct: > > There?s a lot of misunderstanding between setup.py and > requirements.txt and their roles. A lot of people have felt they are > duplicated information and have even created tools to handle this > ?duplication?. > > https://github.com/nvie/pip-tools : - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock) > > -- > \ ?Corporation, n. An ingenious device for obtaining individual | > `\ profit without individual responsibility.? ?Ambrose Bierce, | > _o__) _The Devil's Dictionary_, 1906 | > Ben Finney > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Dec 11 00:05:30 2016 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 10 Dec 2016 23:05:30 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: Here are some standardized (conda) package versions: https://github.com/jupyter/docker-stacks/blob/master/scipy-notebook/Dockerfile On Thursday, December 8, 2016, Wes Turner wrote: > > > On Thursday, December 8, 2016, Nick Coghlan > wrote: > >> Putting the conclusion first, I do see value in better publicising >> "Recommended libraries" based on some automated criteria like: >> >> - recommended in the standard library documentation >> - available via 1 or more cross-platform commercial Python redistributors >> - available via 1 or more Linux distro vendors >> - available via 1 or more web service development platforms >> >> > So these would be attributes tracked by a project maintainer and verified > by the known-good-set maintainer? Or? > > (Again, here I reach for JSONLD. "count n" is only so useful; *which* > {[re]distros, platforms, heartfelt testimonials from incredible experts} > URLs ) > > - test coverage > - seclist contact info AND procedures > - more than one super admin maintainer > - what other criteria should/could/can we use to vet open source libraries? > > >> That would be a potentially valuable service for folks new to the >> world of open source that are feeling somewhat overwhelmed by the >> sheer number of alternatives now available to them. >> >> However, I also think that would better fit in with the aims of an >> open source component tracking community like libraries.io than it >> does a publisher-centric community like distutils-sig. > > > IDK if libraries are really in scope for stackshare. The feature > upcoming/down voting is pretty cool. > > https://stackshare.io/python > > >> >> The further comments below are just a bit more background on why I >> feel the integration testing aspect of the suggestion isn't likely to >> be particularly beneficial :) > > > A catch-all for testing bits from application-specific integration test > suites could be useful (and would likely require at least docker-compose, > dox, kompose for working with actual data stores) > > >> >> On 9 December 2016 at 01:10, Barry Warsaw wrote: >> > Still, there may be value in inter-Python package compatibility tests, >> but >> > it'll take serious engineering effort (i.e. $ and time), ongoing >> maintenance, >> > ongoing effort to fix problems, and tooling to gate installability of >> failing >> > packages (with overrides for downstreams which don't care or already >> expend >> > such effort). >> >> I think this is really the main issue, as both desktop and server >> environments are moving towards the integrated platform + isolated >> applications approach popularised by mobile devices. >> >> That means we end up with two very different variants of automated >> integration testing: >> >> - the application focused kind offered by the likes of requires.io and >> pyup.io (i.e. monitor for dependency updates, submit PRs to trigger >> app level CI) >> - the platform focused kind employed by distro vendors (testing all >> the platform components work together, including the app isolation >> features) >> >> The first kind makes sense if you're building something that runs *on* >> platforms (Docker containers, Snappy or FlatPak apps, web services, >> mobile apps, etc). >> >> The second kind inevitably ends up intertwined with the component >> review and release engineering systems of the particular platform, so >> it becomes really hard to collaborate cross-platform outside the >> context of specific projects like OpenStack that provide clear >> definitions for "What components do we collectively depend on that we >> need to test together?" and "What does 'working' mean in the context >> of this project?". >> >> Accordingly, for an initiative like this to be successful, it would >> need to put some thought up front into the questions of: >> >> 1. Who are the intended beneficiaries of the proposal? >> 2. What problem does it address that will prompt them to contribute >> time and/or money to solving it? >> 3. What do we expect people to be able to *stop doing* if the project >> proves successful? >> >> For platform providers, a generic "stdlib++" project wouldn't really >> reduce the amount of integration testing we'd need to do ourselves (we >> already don't test arbitrary combinations of dependencies, just the >> ones we provide at any given point in time). >> >> For application and service developers, the approach of pinning >> dependencies to specific versions and treating updates like any other >> source code change already works well in most cases. >> >> That leaves library and framework developers, who currently tend to >> adopt the policy of "for each version of Python that we support, we >> test against the latest versions of our dependencies that were >> available at the time we ran the test", leaving testing against older >> versions to platform providers. If there's a key related framework >> that also provides LTS versions (e.g. Django), then some folks may add >> that to their test matrix as well. >> >> In that context, "Only breaks backwards compatibility for compelling >> reasons" becomes a useful long term survival trait for libraries and >> frameworks, as gratuitous breakages are likely to lead to people >> migrating away from particularly unreliable dependencies. > > > Sometimes, when there are no active maintainers for a feature, it makes > sense to deprecate and/or split that functionality / untested cruft out to > a different package. > > >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Dec 14 12:41:44 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 14 Dec 2016 09:41:44 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <20161208161024.682f26dc@presto> References: <20161208161024.682f26dc@presto> Message-ID: As pointed out by others, there are external groups doing "curating". conda-forge is one such project, so I'll comment from that perspective: It's difficult because the definition of compatibility is highly dependent on > > the consumer's environment. For example, C extension compatibility will > depend on the version of libraries available on the platform versions you > care > about. Indeed -- which is why Anaconda and conda-forge are built on conda rather than pip -- it is designed to handle these issues. However with the many linux effort, and some efforts to kludge C libs into binary wheels, pypi may just be able to handle more of these issues -- so curating may have it's advantages. As it happens, I recently proposed a version system for conda-forge (much like Anaconda) where you would select a given conda-forge version, and ALL the packages in that version would be expected to work together. the idea is that you don't want one package dependent on libjpeg6 while another depends on libjpeg7, for instance. But the community has decided that rather than try to version the whole system, we instead rely on robust "pinning" -- i.e. version specific dependencies -- package A depends on libjpeg7, not just "libjpeg". Then there are tools that go through all the packages and check for incompatible pinnings, and update an build new versions where required (or at least ping the maintainers of a package that un update is needed) Ubuntu has an elaborate automated system for testing some dimension of > compatibility issues between packages, not just Python packages. Debian > has > the same system but isn't gated on the results. This brings up the larger issue -- PyPi is inherently different than these efforts -- PyPi has always been about each package author maintaining their own package -- Linux distros and conda-forge, and ??? all have a small set of core contributions that do the package maintenance. This is a large effort, and wold be insanely hard with the massive amount of stuff on PyPi.... In fact, I think the kinda-sort curation that comes from individual communities is working remarkably well: the scipy community the django community ... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Dec 14 20:29:42 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 14 Dec 2016 17:29:42 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: > On Dec 14, 2016, at 9:41 AM, Chris Barker wrote: > > As pointed out by others, there are external groups doing "curating". conda-forge is one such project, so I'll comment from that perspective: > > > It's difficult because the definition of compatibility is highly dependent on > the consumer's environment. For example, C extension compatibility will > depend on the version of libraries available on the platform versions you care > about. > > Indeed -- which is why Anaconda and conda-forge are built on conda rather than pip -- it is designed to handle these issues. > > However with the many linux effort, and some efforts to kludge C libs into binary wheels, pypi may just be able to handle more of these issues -- so curating may have it's advantages. I think it's unfair to describe these efforts as a "kludge"; many of the tools developed for manylinux1 et. al. are actually pretty sophisticated tooling with a mature ecosystem approach to library bundling. Personally I have noticed a _massive_ reduction in the support overhead involved in getting new users spun up in the present Python packaging ecosystem. Due to the availability of cross-platform wheels, it's possible to do a LOT more python development without a C compiler than it used to be. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Dec 14 20:50:39 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 14 Dec 2016 17:50:39 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: <-3240261441486494907@unknownmsgid> > > > I think it's unfair to describe these efforts as a "kludge"; I was specifically referring to using wheels to deliver C libs-- pip+wheel were not designed for that. But I don't mean to offend, there has been a lot of great work done, and yes, the situation has much improved as a result. There are times when a kludge is called for! -CHB From ncoghlan at gmail.com Thu Dec 15 00:22:10 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Dec 2016 15:22:10 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: On 15 December 2016 at 03:41, Chris Barker wrote: [Barry wrote] >> Ubuntu has an elaborate automated system for testing some dimension of >> compatibility issues between packages, not just Python packages. Debian >> has >> the same system but isn't gated on the results. > > This brings up the larger issue -- PyPi is inherently different than these > efforts -- PyPi has always been about each package author maintaining their > own package -- Linux distros and conda-forge, and ??? all have a small set > of core contributions that do the package maintenance. Fedora at least has just shy of 1900 people in the "packager" group, so I don't know that "small" is the right word in absolute terms :) However, relatively speaking, even a packager group that size is still an order of magnitude smaller than the 30k+ publishers on PyPI (which is in turn an order of magnitude smaller than the 180k+ registered PyPI accounts) > This is a large > effort, and wold be insanely hard with the massive amount of stuff on > PyPi.... > > In fact, I think the kinda-sort curation that comes from individual > communities is working remarkably well: > > the scipy community > the django community > ... Exactly. Armin Ronacher and a few others have also started a new umbrella group on GitHub, Pallets, collecting together some of the key infrastructure projects in the Flask ecosystem: https://www.palletsprojects.com/blog/hello/ Dell/EMC's John Mark Walker has a recent article about this "downstream distribution" formation process on opensource.com, where it's an emergent phenomenon arising from the needs of people that are consuming open source components to achieve some particular purpose rather than working on them for their own sake: https://opensource.com/article/16/12/open-source-software-supply-chain It's a fairly different activity from pure upstream development - where upstream is a matter of "design new kinds and versions of Lego bricks" (e.g. the Linux kernel, gcc, CPython, PyPI projects), downstream integration is more "define new Lego kits using the already available bricks" (e.g. Debian, Fedora, conda-forge), while commercial product and service development is "We already put the Lego kit together for you, so you can just use it" (e.g. Ubuntu, RHEL, Amazon Linux, ActivePython, Enthought Canopy, Wakari.io). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From freddyrietdijk at fridh.nl Thu Dec 15 07:13:28 2016 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 15 Dec 2016 13:13:28 +0100 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: It's interesting to read about how other distributions upgrade their package sets. In Nixpkgs most packages are updated manually. Some frameworks/languages provide their dependencies declarative, in which case it becomes 'straightforward' to include whole package sets, like in the case of Haskell. Some expressions need to be overridden manually because e.g. they require certain system libraries. It's manual work, but not that much. This is what I would like to see for Python as well. The Python packages we still update manually although we have tools to automate most of it. The reason for not using those tools is because a) it means evaluating or building parts of the packages to get the dependencies and b) too often upstream pins versions of dependencies which turns out to be entirely unnecessary, and would therefore prevent an upgrade. Even so, we have over 1500 Python packages per interpreter version that according to our CI seem to work together. We do only build on 3 architectures (i386, amd64 and darwin/osx). Compatibility with the latter is sometimes an issue because its guessing what Apple has changed when releasing a new version. > I'm not sure how useful it would be higher up the food chain, since those contexts will be > different enough to cause both false positives and false negatives. And it > does often take quite a bit of focused engineering effort to monitor packages > which don't promote (something we want to automate), In my experience the manual work that typically needs to be done is a) making available Python dependencies, b) unpinning versions when unnecessary and report upstream, and c) making sure the package finds the system libraries. Issues with packages that cannot be upgraded because of a version of a system dependency I haven't yet encountered. In my proposal a) and b) would be fixed by the curated package set. > https://github.com/nvie/pip-tools : > - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock) Yep, there are tools, which get the job done when developing and using a set of packages. Now, you want to deploy your app. You can't use your requirements.txt on a Linux distribution because they have a curated set of packages which is typically different from your set (although maybe they do provide tools to package the versions you need). But, you can choose to use a second package manager, pip or conda. System dependencies? That's something conda can somewhat take of. Problem solved you would say, except now you have multiple package managers that you need. > Practically, a developer would want a subset of the given known-good-set (and then additional packages), so: > > - fork/copy requirements-YYYY-MM-REV--.txt > - #comment out unused deps > - add '-r addl-requirements.txt' See the link I shared earlier on how this is already done with Haskell and stack.yaml and how it could be used with `pipfile` https://github.com/pypa/pipfile/issues/10#issuecomment-262229620 > Putting the conclusion first, I do see value in better publicising > "Recommended libraries" based on some automated criteria like: Yes, we should recommend third-party libraries in a trusted place like the documentation of CPython. The amount of packages that are available can be overwhelming. Yet, defining a set of packages that are recommended, and perhaps working together, is still far from defining an exact set of packages that are known to work together, something which I proposed here. > As pointed out by others, there are external groups doing "curating". conda-forge is one such project, so I'll comment from that perspective I haven't used conda in a long time, and conda-forge didn't exist back then. I see versions are pinned, but versions of dependencies sometimes as well. If I choose to install *all* the packages available via conda-forge, will I get a fixed package set, or will the SAT-solver try to find a working set (and possibly fail at it)? I hope it is the former, since if it is the latter then it is not curated in how I meant it. On Thu, Dec 15, 2016 at 6:22 AM, Nick Coghlan wrote: > On 15 December 2016 at 03:41, Chris Barker wrote: > [Barry wrote] > >> Ubuntu has an elaborate automated system for testing some dimension of > >> compatibility issues between packages, not just Python packages. Debian > >> has > >> the same system but isn't gated on the results. > > > > This brings up the larger issue -- PyPi is inherently different than > these > > efforts -- PyPi has always been about each package author maintaining > their > > own package -- Linux distros and conda-forge, and ??? all have a small > set > > of core contributions that do the package maintenance. > > Fedora at least has just shy of 1900 people in the "packager" group, > so I don't know that "small" is the right word in absolute terms :) > > However, relatively speaking, even a packager group that size is still > an order of magnitude smaller than the 30k+ publishers on PyPI (which > is in turn an order of magnitude smaller than the 180k+ registered > PyPI accounts) > > > This is a large > > effort, and wold be insanely hard with the massive amount of stuff on > > PyPi.... > > > > In fact, I think the kinda-sort curation that comes from individual > > communities is working remarkably well: > > > > the scipy community > > the django community > > ... > > Exactly. Armin Ronacher and a few others have also started a new > umbrella group on GitHub, Pallets, collecting together some of the key > infrastructure projects in the Flask ecosystem: > https://www.palletsprojects.com/blog/hello/ > > Dell/EMC's John Mark Walker has a recent article about this > "downstream distribution" formation process on opensource.com, where > it's an emergent phenomenon arising from the needs of people that are > consuming open source components to achieve some particular purpose > rather than working on them for their own sake: > https://opensource.com/article/16/12/open-source-software-supply-chain > > It's a fairly different activity from pure upstream development - > where upstream is a matter of "design new kinds and versions of Lego > bricks" (e.g. the Linux kernel, gcc, CPython, PyPI projects), > downstream integration is more "define new Lego kits using the already > available bricks" (e.g. Debian, Fedora, conda-forge), while commercial > product and service development is "We already put the Lego kit > together for you, so you can just use it" (e.g. Ubuntu, RHEL, Amazon > Linux, ActivePython, Enthought Canopy, Wakari.io). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Dec 15 07:20:37 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Dec 2016 07:20:37 -0500 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: > On Dec 15, 2016, at 7:13 AM, Freddy Rietdijk wrote: > > > Putting the conclusion first, I do see value in better publicising > > "Recommended libraries" based on some automated criteria like: > > Yes, we should recommend third-party libraries in a trusted place like the documentation of CPython. The amount of packages that are available can be overwhelming. Yet, defining a set of packages that are recommended, and perhaps working together, is still far from defining an exact set of packages that are known to work together, something which I proposed here. We could theoretically bake this into PyPI itself, though I?m not sure if that makes sense. We could also probably bake something like ?curated package sets? into PyPI where individual users (or groups of users) can create their own view of PyPI for people to use, while still relying on all of the infrastructure of PyPI. Although I?m not sure that makes any sense either. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu Dec 15 09:35:38 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 15 Dec 2016 06:35:38 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too. Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.) Top-posted from my Windows Phone -----Original Message----- From: "Donald Stufft" Sent: ?12/?15/?2016 4:21 To: "Freddy Rietdijk" Cc: "DistUtils mailing list" ; "Barry Warsaw" Subject: Re: [Distutils] Maintaining a curated set of Python packages On Dec 15, 2016, at 7:13 AM, Freddy Rietdijk wrote: > Putting the conclusion first, I do see value in better publicising > "Recommended libraries" based on some automated criteria like: Yes, we should recommend third-party libraries in a trusted place like the documentation of CPython. The amount of packages that are available can be overwhelming. Yet, defining a set of packages that are recommended, and perhaps working together, is still far from defining an exact set of packages that are known to work together, something which I proposed here. We could theoretically bake this into PyPI itself, though I?m not sure if that makes sense. We could also probably bake something like ?curated package sets? into PyPI where individual users (or groups of users) can create their own view of PyPI for people to use, while still relying on all of the infrastructure of PyPI. Although I?m not sure that makes any sense either. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Dec 15 09:39:40 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Dec 2016 09:39:40 -0500 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> > On Dec 15, 2016, at 9:35 AM, Steve Dower wrote: > > The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too. > > Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.) > Theoretically we could allow people to not just select packages, but also package specifiers for their ?curated package set?, so instead of saying ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like: $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 15 09:57:37 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 08:57:37 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Donald Stufft wrote: > > On Dec 15, 2016, at 9:35 AM, Steve Dower > wrote: > > The "curated package sets" on PyPI idea sounds a bit like Steam's curator > lists, which I like to think of as Twitter for game reviews. You can follow > a curator to see their comments on particular games, and the most popular > curators have their comments appear on the actual listings too. > > Might be interesting to see how something like that worked for PyPI, > though the initial investment is pretty high. (It doesn't solve the > coherent bundle problem either, just the discovery of good libraries > problem.) > > > Theoretically we could allow people to not just select packages, but also > package specifiers for their ?curated package set?, so instead of saying > ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we > really wanted to get slick we could even provide a requirements.txt file > format, and have people able to install the entire set by doing something > like: > > $ pip install -r https://pypi.org/sets/dstufft/ > my-cool-set/requirements.txt > With version control? $ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/abcd123/requirements.txt $ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/v0.0.1/requirements.txt This would be a graph. JSONLD? #PEP426JSONLD: - https://www.google.com/search?q=pep426jsonld - https://github.com/pypa/interoperability-peps/issues/31 With JSONLD, we could merge SoftwarePackage metadata with SoftwarePackageCollection metadata (just throwing some types out there). A http://schema.org/SoftwareApplication is a http://schema.org/CreativeWork . http://schema.org/softwareVersion > > > ? > Donald Stufft > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 15 10:10:56 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 01:10:56 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On 16 December 2016 at 00:39, Donald Stufft wrote: > Theoretically we could allow people to not just select packages, but also > package specifiers for their ?curated package set?, so instead of saying > ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we > really wanted to get slick we could even provide a requirements.txt file > format, and have people able to install the entire set by doing something > like: > > $ pip install -r > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt CurseGaming provide addon managers for a variety of game addons (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is one of the ways they make it useful to have an account on the site even if you don't publish any addons of your own. Even if you don't make them public, you can still use them to sync your addon sets between different machines. In the context of Python, where I can see this kind of thing being potentially useful is for folks to manage package sets that aren't necessarily coupled to any specific project, but match the way they *personally* work. - "These are the packages I like to have installed to --user" - "These are the packages I use to start a CLI app" - "These are the packages I use to start a web app" - etc... It also provides a way for people to vote on projects that's a little more meaningful than stars - projects that appear in a lot of personal stack definitions are likely to be generally valuable (the closest equivalent to that today is mining code repositories like GitHub for requirements.txt files and seeing what people are using that way). So yeah, if folks interested in this were to add it to Warehouse (and hence pypi.org), I think it would definitely be a valuable enhancement to the overall ecosystem. "What needs to be implemented in order to be able to shut down the legacy service at pypi.python.org?" is the *PSF's* focus, but that doesn't mean it needs to be everyone's focus. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Thu Dec 15 10:22:04 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 09:22:04 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Wes Turner wrote: > > > On Thursday, December 15, 2016, Donald Stufft > wrote: > >> >> On Dec 15, 2016, at 9:35 AM, Steve Dower wrote: >> >> The "curated package sets" on PyPI idea sounds a bit like Steam's curator >> lists, which I like to think of as Twitter for game reviews. You can follow >> a curator to see their comments on particular games, and the most popular >> curators have their comments appear on the actual listings too. >> >> Might be interesting to see how something like that worked for PyPI, >> though the initial investment is pretty high. (It doesn't solve the >> coherent bundle problem either, just the discovery of good libraries >> problem.) >> >> >> Theoretically we could allow people to not just select packages, but also >> package specifiers for their ?curated package set?, so instead of saying >> ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we >> really wanted to get slick we could even provide a requirements.txt file >> format, and have people able to install the entire set by doing something >> like: >> >> $ pip install -r https://pypi.org/sets/dstufft/ >> my-cool-set/requirements.txt >> > > With version control? > > $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/abcd123/ > requirements.txt > > > $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/v0.0.1/ > requirements.txt > > > This would be a graph. JSONLD? > #PEP426JSONLD: > - https://www.google.com/search?q=pep426jsonld > - https://github.com/pypa/interoperability-peps/issues/31 > > With JSONLD, we could merge SoftwarePackage metadata with > SoftwarePackageCollection metadata (just throwing some types out there). > > A http://schema.org/SoftwareApplication is a http://schema.org/ > CreativeWork . > > http://schema.org/softwareVersion > > - https://pypi.org/project/ as the canonical project http://schema.org/url - There's almost certainly a transform of TOML to JSONLD (" TOMLLD ") - There is a standardized transform of JSONLD to RDF - YAMLLD is a stricter subset of YAML (because just OrderedDicts >> >> >> ? >> Donald Stufft >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Thu Dec 15 10:22:50 2016 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 15 Dec 2016 16:22:50 +0100 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: > Theoretically we could allow people to not just select packages, but also package specifiers for their ?curated package set?, so instead of saying ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like: $ pip install -r https://pypi.org/sets/dstufft/ my-cool-set/requirements.txt What you want to be able to do is install a set of (abstract) packages based on a curated set of (concrete) packages. Doing that you just need a file with your concrete dependencies, and now we're back at pipfile and https://github.com/pypa/pipfile/issues/10#issuecomment-262229620 On Thu, Dec 15, 2016 at 4:10 PM, Nick Coghlan wrote: > On 16 December 2016 at 00:39, Donald Stufft wrote: > > Theoretically we could allow people to not just select packages, but also > > package specifiers for their ?curated package set?, so instead of saying > > ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we > > really wanted to get slick we could even provide a requirements.txt file > > format, and have people able to install the entire set by doing something > > like: > > > > $ pip install -r > > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt > > CurseGaming provide addon managers for a variety of game addons > (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is > one of the ways they make it useful to have an account on the site > even if you don't publish any addons of your own. Even if you don't > make them public, you can still use them to sync your addon sets > between different machines. > > In the context of Python, where I can see this kind of thing being > potentially useful is for folks to manage package sets that aren't > necessarily coupled to any specific project, but match the way they > *personally* work. > > - "These are the packages I like to have installed to --user" > - "These are the packages I use to start a CLI app" > - "These are the packages I use to start a web app" > - etc... > > It also provides a way for people to vote on projects that's a little > more meaningful than stars - projects that appear in a lot of personal > stack definitions are likely to be generally valuable (the closest > equivalent to that today is mining code repositories like GitHub for > requirements.txt files and seeing what people are using that way). > > So yeah, if folks interested in this were to add it to Warehouse (and > hence pypi.org), I think it would definitely be a valuable enhancement > to the overall ecosystem. "What needs to be implemented in order to be > able to shut down the legacy service at pypi.python.org?" is the > *PSF's* focus, but that doesn't mean it needs to be everyone's focus. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 15 10:25:44 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 01:25:44 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On 16 December 2016 at 00:57, Wes Turner wrote: > This would be a graph. JSONLD? > #PEP426JSONLD: > - https://www.google.com/search?q=pep426jsonld > - https://github.com/pypa/interoperability-peps/issues/31 > > With JSONLD, we could merge SoftwarePackage metadata with > SoftwarePackageCollection metadata (just throwing some types out there). Wes, JSON-LD is a metasystem used for descriptive analytics across mixed datasets, which *isn't a problem we have*. We have full authority over the data formats we care about, and the user needs that matter to distutils-sig are: - publishers of Python packages - consumers of Python packages - maintainers of the toolchain It would *absolutely* make sense for Semantic Web folks to get involved in the libraries.io project (either directly, or by building a separate service backed by the libraries.io data set) and seek to produce a global set of semantically linked data that spans not only dependencies within language ecosystems, but also dependencies between them. It *doesn't* make sense for every single language ecosystem to come up with its own unique spin on how to incorporate software packages into semantic web models, nor does it make sense to try to warp the Python packaging user experience to better meet the needs of taxonomists of knowledge. This answer hasn't changed the last dozen times you've brought up JSON-LD. It isn't *going* to change. So please stop bringing it up. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Thu Dec 15 10:35:40 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 09:35:40 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Nick Coghlan wrote: > On 16 December 2016 at 00:39, Donald Stufft > wrote: > > Theoretically we could allow people to not just select packages, but also > > package specifiers for their ?curated package set?, so instead of saying > > ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we > > really wanted to get slick we could even provide a requirements.txt file > > format, and have people able to install the entire set by doing something > > like: > > > > $ pip install -r > > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt > > CurseGaming provide addon managers for a variety of game addons > (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is > one of the ways they make it useful to have an account on the site > even if you don't publish any addons of your own. Even if you don't > make them public, you can still use them to sync your addon sets > between different machines. > > In the context of Python, where I can see this kind of thing being > potentially useful is for folks to manage package sets that aren't > necessarily coupled to any specific project, but match the way they > *personally* work. > > - "These are the packages I like to have installed to --user" > - "These are the packages I use to start a CLI app" > - "These are the packages I use to start a web app" > - etc... Does a requirements.txt in a {git,} repo solve for this already? A Collection contains (hasPart) CreativeWorks - https://schema.org/Collection - https://schema.org/hasPart RDFa and JSONLD representations do parse as ordered lists. SoftwarePackageCollection SoftwareApplicationCollection > It also provides a way for people to vote on projects that's a little > more meaningful than stars - projects that appear in a lot of personal > stack definitions are likely to be generally valuable (the closest > equivalent to that today is mining code repositories like GitHub for > requirements.txt files and seeing what people are using that way). https://schema.org/InteractionCounter > https://schema.org/UserLikes D: CreativeWork - https://schema.org/interactionCount is now - https://schema.org/interactionStatistic (These are write-heavy features: they would change the database load of Warehouse) > > So yeah, if folks interested in this were to add it to Warehouse (and > hence pypi.org), I think it would definitely be a valuable enhancement > to the overall ecosystem. "What needs to be implemented in order to be > able to shut down the legacy service at pypi.python.org?" is the > *PSF's* focus, but that doesn't mean it needs to be everyone's focus. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, > Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 15 10:38:39 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 09:38:39 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Nick Coghlan wrote: > On 16 December 2016 at 00:57, Wes Turner > wrote: > > This would be a graph. JSONLD? > > #PEP426JSONLD: > > - https://www.google.com/search?q=pep426jsonld > > - https://github.com/pypa/interoperability-peps/issues/31 > > > > With JSONLD, we could merge SoftwarePackage metadata with > > SoftwarePackageCollection metadata (just throwing some types out there). > > Wes, JSON-LD is a metasystem used for descriptive analytics across > mixed datasets, which *isn't a problem we have*. We have full > authority over the data formats we care about, and the user needs that > matter to distutils-sig are: > > - publishers of Python packages > - consumers of Python packages > - maintainers of the toolchain > > It would *absolutely* make sense for Semantic Web folks to get > involved in the libraries.io project (either directly, or by building > a separate service backed by the libraries.io data set) and seek to > produce a global set of semantically linked data that spans not only > dependencies within language ecosystems, but also dependencies between > them. It *doesn't* make sense for every single language ecosystem to > come up with its own unique spin on how to incorporate software > packages into semantic web models, nor does it make sense to try to > warp the Python packaging user experience to better meet the needs of > taxonomists of knowledge. > > This answer hasn't changed the last dozen times you've brought up > JSON-LD. It isn't *going* to change. So please stop bringing it up. No, the problem is the same; and solving it (joining user-specific package metadata with central repository metadata on a common URI) with web standards is the best approach. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, > Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 15 10:57:02 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 09:57:02 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Wes Turner wrote: > > > On Thursday, December 15, 2016, Nick Coghlan > wrote: > >> On 16 December 2016 at 00:39, Donald Stufft wrote: >> > Theoretically we could allow people to not just select packages, but >> also >> > package specifiers for their ?curated package set?, so instead of saying >> > ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we >> > really wanted to get slick we could even provide a requirements.txt file >> > format, and have people able to install the entire set by doing >> something >> > like: >> > >> > $ pip install -r >> > https://pypi.org/sets/dstufft/my-cool-set/requirements.txt >> >> CurseGaming provide addon managers for a variety of game addons >> (Warcraft, Minecraft, etc), and the ability to define "AddOn Packs" is >> one of the ways they make it useful to have an account on the site >> even if you don't publish any addons of your own. Even if you don't >> make them public, you can still use them to sync your addon sets >> between different machines. >> >> In the context of Python, where I can see this kind of thing being >> potentially useful is for folks to manage package sets that aren't >> necessarily coupled to any specific project, but match the way they >> *personally* work. >> >> - "These are the packages I like to have installed to --user" >> - "These are the packages I use to start a CLI app" >> - "These are the packages I use to start a web app" >> - etc... > > > Does a requirements.txt in a {git,} repo solve for this already? > Could you just generate a README.rst for a long_description from requirements.txt (or requirements.in, or pipfile), store that in a {git,} repository, and use the existing pypi release versioning and upload machinery? Or would it be more useful to surface the other graph edges on project pages? - These additional queries would be less burdensome as AJAX requests to JSON REST API views. Project pages could continue to load without waiting for these additional cached edge bundles (and interactionStatistic(s)) to load: - "This SoftwarePackage is in the following n SoftwarePackageCollections, with the following comments and reviews" - "This SoftwarePackage has received n UserLikes (through Warehouse)" > A Collection contains (hasPart) CreativeWorks > > - https://schema.org/Collection > - https://schema.org/hasPart > > RDFa and JSONLD representations do parse as ordered lists. > > SoftwarePackageCollection > SoftwareApplicationCollection > > >> It also provides a way for people to vote on projects that's a little >> more meaningful than stars - projects that appear in a lot of personal >> stack definitions are likely to be generally valuable (the closest >> equivalent to that today is mining code repositories like GitHub for >> requirements.txt files and seeing what people are using that way). > > > https://schema.org/InteractionCounter > https://schema.org/UserLikes > > D: CreativeWork > - https://schema.org/interactionCount is now > - https://schema.org/interactionStatistic > > (These are write-heavy features: they would change the database load of > Warehouse) > > >> >> So yeah, if folks interested in this were to add it to Warehouse (and >> hence pypi.org), I think it would definitely be a valuable enhancement >> to the overall ecosystem. "What needs to be implemented in order to be >> able to shut down the legacy service at pypi.python.org?" is the >> *PSF's* focus, but that doesn't mean it needs to be everyone's focus. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 15 10:58:32 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 01:58:32 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On 16 December 2016 at 01:38, Wes Turner wrote: > On Thursday, December 15, 2016, Nick Coghlan wrote: >> This answer hasn't changed the last dozen times you've brought up >> JSON-LD. It isn't *going* to change. So please stop bringing it up. > > > No, the problem is the same; and solving it (joining user-specific package > metadata with central repository metadata on a common URI) with web > standards is the best approach. Then do what Donald did with crate.io: go develop your own PyPI competitor that uses your new and better approach to prove your point. As things stand, you're just generating noise and link spam on the list, and asking politely for you to stop it doesn't appear to be working Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Dec 15 11:14:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Dec 2016 16:14:11 +0000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On 15 December 2016 at 15:58, Nick Coghlan wrote: > On 16 December 2016 at 01:38, Wes Turner wrote: >> On Thursday, December 15, 2016, Nick Coghlan wrote: >>> This answer hasn't changed the last dozen times you've brought up >>> JSON-LD. It isn't *going* to change. So please stop bringing it up. >> >> >> No, the problem is the same; and solving it (joining user-specific package >> metadata with central repository metadata on a common URI) with web >> standards is the best approach. > > Then do what Donald did with crate.io: go develop your own PyPI > competitor that uses your new and better approach to prove your point. > > As things stand, you're just generating noise and link spam on the > list, and asking politely for you to stop it doesn't appear to be > working Just to add my POV, I also find your posts unhelpful, Wes. There's not enough information for me to evaluate what you say, and you offer no actual solutions to what's being discussed. As Nick says, if you can demonstrate what you're suggesting via a prototype implementation, that might help. But endless posts of links to standards documents that I have neither the time nor the inclination to read are not useful. Paul From wes.turner at gmail.com Thu Dec 15 11:18:47 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 10:18:47 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Nick Coghlan wrote: > On 16 December 2016 at 01:38, Wes Turner > wrote: > > On Thursday, December 15, 2016, Nick Coghlan > wrote: > >> This answer hasn't changed the last dozen times you've brought up > >> JSON-LD. It isn't *going* to change. So please stop bringing it up. > > > > > > No, the problem is the same; and solving it (joining user-specific > package > > metadata with central repository metadata on a common URI) with web > > standards is the best approach. > > Then do what Donald did with crate.io: go develop your own PyPI > competitor that uses your new and better approach to prove your point. > > As things stand, you're just generating noise and link spam on the > list, and asking politely for you to stop it doesn't appear to be > working I've intentionally included the URLs which support my position; none of which do I have any commercial interest in. This is not link spam, this is me explaining your problem to you in order to save time and money. There are your: - existing solutions that work today - types (web standard types) - use cases - server load engineering challenges (web apps must cache) Have a good day. > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, > Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Dec 15 14:13:13 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 13:13:13 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Paul Moore wrote: > On 15 December 2016 at 15:58, Nick Coghlan > wrote: > > On 16 December 2016 at 01:38, Wes Turner > wrote: > >> On Thursday, December 15, 2016, Nick Coghlan > wrote: > >>> This answer hasn't changed the last dozen times you've brought up > >>> JSON-LD. It isn't *going* to change. So please stop bringing it up. > >> > >> > >> No, the problem is the same; and solving it (joining user-specific > package > >> metadata with central repository metadata on a common URI) with web > >> standards is the best approach. > > > > Then do what Donald did with crate.io: go develop your own PyPI > > competitor that uses your new and better approach to prove your point. > > > > As things stand, you're just generating noise and link spam on the > > list, and asking politely for you to stop it doesn't appear to be > > working > > Just to add my POV, I also find your posts unhelpful, Wes. There's not > enough information for me to evaluate what you say, and you offer no > actual solutions to what's being discussed. I could quote myself suggesting solutions in this thread, if you like? > > As Nick says, if you can demonstrate what you're suggesting via a > prototype implementation, that might help. But endless posts of links > to standards documents that I have neither the time nor the > inclination to read are not useful. I'm suggesting that the solution here is to create version-controlled collections of resources with metadata. With schema.org types, those are Collection s of CreativeWork s. (Or, e.g. SoftwareApplicationCollection s of SoftwareApplication s) With Django, one would create a ListView generic view for the given resource type; and then define e.g. an api.py for DRF (Django REST Framework), and an index.py for Haystack (Elastic search). A Pyramid implementation would be similar; with additional tests and code just like Donald. [Links omitted because you can now copy/paste keywords into the search engines yourself; because you're unable to scroll past citations to find the content you're looking for.] DWBP and BP-LD would be useful reading for Warehouse API developers seeking to add additional functionality to support the aforementioned use cases with maximally-compatible linked open data. Again, Have a good day. > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Dec 15 14:50:15 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Dec 2016 19:50:15 +0000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On 15 December 2016 at 19:13, Wes Turner wrote: >> Just to add my POV, I also find your posts unhelpful, Wes. There's not >> enough information for me to evaluate what you say, and you offer no >> actual solutions to what's being discussed. > > > I could quote myself suggesting solutions in this thread, if you like? You offer lots of pointers to information. But that's different. Anyway, let's just agree to differ - I can skip your mails if they aren't helpful to me, and you don't need to bother about the fact that you're not getting your points across to me. Who knows, one day I might find the time to look into JSON-LD, at which point I may or may not understand why you think it's such a useful tool for solving all these problems (in spite of the fact that no-one else seems to think the same...) Paul From glyph at twistedmatrix.com Thu Dec 15 16:14:47 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 15 Dec 2016 13:14:47 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> > On Dec 15, 2016, at 6:39 AM, Donald Stufft wrote: > > >> On Dec 15, 2016, at 9:35 AM, Steve Dower > wrote: >> >> The "curated package sets" on PyPI idea sounds a bit like Steam's curator lists, which I like to think of as Twitter for game reviews. You can follow a curator to see their comments on particular games, and the most popular curators have their comments appear on the actual listings too. >> >> Might be interesting to see how something like that worked for PyPI, though the initial investment is pretty high. (It doesn't solve the coherent bundle problem either, just the discovery of good libraries problem.) >> > > Theoretically we could allow people to not just select packages, but also package specifiers for their ?curated package set?, so instead of saying ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like: > > $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt Can't people already do this by publishing a package that just depends on their whole 'package set'? -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 15 22:48:59 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 13:48:59 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On 16 December 2016 at 05:50, Paul Moore wrote: > On 15 December 2016 at 19:13, Wes Turner wrote: >>> Just to add my POV, I also find your posts unhelpful, Wes. There's not >>> enough information for me to evaluate what you say, and you offer no >>> actual solutions to what's being discussed. >> >> >> I could quote myself suggesting solutions in this thread, if you like? > > You offer lots of pointers to information. But that's different. Exactly. There are *lots* of information processing standards out there, and lots of things we *could* provide natively that simply aren't worth the hassle since folks that care can provide them as "after market addons" for the audiences that considers them relevant. For example, a few things that can matter to different audiences are: - SPDX (Software Package Data Exchange) identifiers for licenses - CPE (Common Product Enumeration) and SWID (Software Identification) tags for published software - DOI (Digital Object Identifier) tags for citation purposes - Common Criteria certification for software supply chains I don't push for these upstream in distutils-sig not because I don't think they're important in general, but because I *don't think they're a priority for distutils-sig*. If you're teaching Python to school students, or teaching engineers and scientists how to better analyse their own data, or building a web service for yourself or your employer, these kinds of things simply don't matter. The end users that care about them are well-positioned to tackle them on their own (or pay other organisations to do it for them), and because they span arbitrary publishing communities anyway, it doesn't really matter all that much if any given publishing community participates directly in the process (the only real beneficiaries are the intermediaries that actively blur the distinctions between the cooperative communities and the recalcitrant ones). > Anyway, let's just agree to differ - I can skip your mails if they > aren't helpful to me, and you don't need to bother about the fact that > you're not getting your points across to me. I consider it fairly important that we have a reasonably common understanding of the target userbase for direct consumption of PyPI data, and what we expect to be supplied as third party services. It's also important that we have a shared understanding of how to constructively frame proposals for change. For the former, the Semantic Web, and folks that care about Semantic Web concepts like "Linked Data" in the abstract sense are not part of our primary audience. We don't go out of our way to make their lives difficult, but "it makes semantic analysis easier" also isn't a compelling rationale for change. For the latter, some variants of constructive proposals look like: - "this kind of user has this kind of problem and this proposed solution will help mitigate it this way (and, by the way, here's an existing standard we can use)" - "this feature exists in , it's really valuable to users for , how about we offer it by default?" - "I wrote for myself, and I think it would also help others for , can you help me make it more widely known and available?" They don't look like "Here's a bunch of technologies and organisations that exist on the internet that may in some way potentially be relevant to the management of a software distribution network", and nor does it look like "This data modeling standard exists, so we should use it, even though it doesn't actually simplify our lives or our users' lives in any way, and in fact makes them more complicated". > Who knows, one day I > might find the time to look into JSON-LD, at which point I may or may > not understand why you think it's such a useful tool for solving all > these problems (in spite of the fact that no-one else seems to think > the same...) I *have* looked at JSON-LD (based primarily on Wes's original suggestions), both from the perspective of the Python packaging ecosystem specifically, as well as my day job working on software supply chain management. My verdict was that for managing a dependency graph implementation, it ends up in the category of technologies that qualify as "interesting, but not helpful". In many ways, it's the urllib2 of data linking - just as urllib2 gives you a URL handling framework which you can configure to handle HTTP rather than just providing a HTTP-specific interface the way requests does [1], JSON-LD gives you a data linking framework, which you can then use to define links between your data, rather than just linking the data directly in a domain-appropriate fashion. Using a framework for the sake of using a framework rather than out of a genuine engineering need doesn't tend to lead to good software systems. Wes seems to think that my perspective on this is born out of ignorance, so repeatedly bringing it up may make me change my point of view. However, our problems haven't changed, and the nature and purpose of JSON-LD haven't changed, so it really won't - the one thing that will change my mind is demonstrated popularity and utility of a service that integrates raw PyPI data with JSON-LD and schema.org. Hence my suggestions (made with varying degrees of politeness) to go build a dependency analysis service that extracts the dependency trees from libraries.io, maps them to schema.org concepts in a way that makes sense, and then demonstrate what that makes possible that can't be done with the libraries.io data directly. Neither Wes nor anyone else needs anyone's permission to go do that, and it will be far more enjoyable for all concerned than the status quo where Wes is refusing to take "No, we've already looked at JSON-LD, and we believe it adds needless complexity for no benefit that we care about" for an answer by continuing to post about it *here* rather than either venting his frustrations about our collective lack of interest somewhere else, or else channeling that frustration into building the system he wishes existed. Cheers, Nick. [1] http://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Dec 15 23:18:45 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 14:18:45 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> Message-ID: On 16 December 2016 at 07:14, Glyph Lefkowitz wrote: > On Dec 15, 2016, at 6:39 AM, Donald Stufft wrote: > Theoretically we could allow people to not just select packages, but also > package specifiers for their ?curated package set?, so instead of saying > ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we > really wanted to get slick we could even provide a requirements.txt file > format, and have people able to install the entire set by doing something > like: > > $ pip install -r https://pypi.org/sets/dstufft/ > my-cool-set/requirements.txt > > > Can't people already do this by publishing a package that just depends on > their whole 'package set'? > Technically, sure, but it adds a lot of overhead. The closest equivalent right now would be maintaining a source control repo somewhere with various requirements files in it. However, at an ecosystem level, that doesn't have the same user experience impact. The idea of building this into PyPI itself would be to *reshape the learning curve of how people learn about dependency management as they're introduced to Python*. Going back to the CurseGaming example, I actually use the free version of their client to manage the Warcraft addons on my gaming PC. The basic usage model is really simple and (not coincidentally) very similar to the way the Add-on manager works in Firefox and other GUI apps with integrated plugin managers: - you have an "Installed" tab for the addons you have installed - when you start the client, it checks for updates for all your installed addons and the out of date ones gain an "Update" button - there's a separate tab where you can search all the available addons and install new ones I've never used any of Curse's other clients (like the Minecraft or Kerbal Space Program ones), but I assume they operate in a similar way. The paid tier of the Curse Client, and the account sync feature of Firefox, then offer the ability to synchronize your installed addons across machines. (There are also a lot of similarities between this model and the way mobile app stores work) A comparable UX for Python/PyPI/pip would focus less on the library-and-application development cases (where the presence of source control is assumed), and more on the ad hoc scripting and learning-to-program use cases, where you're typically more interested in "--user" installations and the question of which parts of the Python ecosystem are just an import away than you are in reproducability and maintainability. The ecosystem level learning curve then becomes: - did you know you can back up your list of user installed packages to PyPI? - did you know you can use PyPI to sync your user installs between systems? - did you know you can use PyPI to categorise your user installs and share them with others? - OK, now it's time to start learning about version control, virtual environments and automated testing It wouldn't necessarily make sense to bake this *directly* into Warehouse, and the Mozilla folks responsible for Firefox Sync could no doubt offer real word guidance on the infrastructure and support needed to operate a service like that at scale, but the core concept of allowing package management to be introduced independently of both version control and virtual environments sounds potentially valuable to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Dec 15 23:29:25 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 15 Dec 2016 20:29:25 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> Message-ID: <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> > On Dec 15, 2016, at 8:18 PM, Nick Coghlan wrote: > > On 16 December 2016 at 07:14, Glyph Lefkowitz > wrote: >> On Dec 15, 2016, at 6:39 AM, Donald Stufft > wrote: >> Theoretically we could allow people to not just select packages, but also package specifiers for their ?curated package set?, so instead of saying ?requests?, you could say ?requests~=2.12? or ?requests==2.12.2?. If we really wanted to get slick we could even provide a requirements.txt file format, and have people able to install the entire set by doing something like: >> >> $ pip install -r https://pypi.org/sets/dstufft/my-cool-set/requirements.txt > Can't people already do this by publishing a package that just depends on their whole 'package set'? > > Technically, sure, but it adds a lot of overhead. The closest equivalent right now would be maintaining a source control repo somewhere with various requirements files in it. > > However, at an ecosystem level, that doesn't have the same user experience impact. The idea of building this into PyPI itself would be to *reshape the learning curve of how people learn about dependency management as they're introduced to Python*. > > Going back to the CurseGaming example, I actually use the free version of their client to manage the Warcraft addons on my gaming PC. The basic usage model is really simple and (not coincidentally) very similar to the way the Add-on manager works in Firefox and other GUI apps with integrated plugin managers: > > - you have an "Installed" tab for the addons you have installed > - when you start the client, it checks for updates for all your installed addons and the out of date ones gain an "Update" button > - there's a separate tab where you can search all the available addons and install new ones > > I've never used any of Curse's other clients (like the Minecraft or Kerbal Space Program ones), but I assume they operate in a similar way. > > The paid tier of the Curse Client, and the account sync feature of Firefox, then offer the ability to synchronize your installed addons across machines. (There are also a lot of similarities between this model and the way mobile app stores work) > > A comparable UX for Python/PyPI/pip would focus less on the library-and-application development cases (where the presence of source control is assumed), and more on the ad hoc scripting and learning-to-program use cases, where you're typically more interested in "--user" installations and the question of which parts of the Python ecosystem are just an import away than you are in reproducability and maintainability. > > The ecosystem level learning curve then becomes: > > - did you know you can back up your list of user installed packages to PyPI? > - did you know you can use PyPI to sync your user installs between systems? > - did you know you can use PyPI to categorise your user installs and share them with others? > - OK, now it's time to start learning about version control, virtual environments and automated testing > > It wouldn't necessarily make sense to bake this *directly* into Warehouse, and the Mozilla folks responsible for Firefox Sync could no doubt offer real word guidance on the infrastructure and support needed to operate a service like that at scale, but the core concept of allowing package management to be introduced independently of both version control and virtual environments sounds potentially valuable to me. Yeah, I think that this focus on curating packages on PyPI is reminiscent about the old yarn about looking for lost keys under the streetlight because it's dark everywhere else. We're all familiar with web services and data formats, so we want to somehow have a data format or a web service be the answer to this problem. But I don't believe that's where the problem is. ("this problem" being "let's make it easy and fun to a) bootstrap a common Python experimentation environment across multiple machines and b) _know that you have to do that_") At the beginning of your story you mentioned the GUI client - that is the missing piece ;). I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment... User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-). Users might even figure out this sort of stuff for themselves if they are given a discoverable API for things like search and installation of packages. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Dec 15 23:33:32 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 15 Dec 2016 23:33:32 -0500 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: <000FADA8-2D13-48D6-AF4C-02BA4302CD34@stufft.io> > On Dec 15, 2016, at 11:29 PM, Glyph Lefkowitz wrote: > > User-curated package sets strikes me as the _lowest_ priority feature out of all of those I don?t think anyone in the PyPA is planning on working on this currently. It was a possible idea that was spawned from this thread. However the nature of volunteer OSS is that volunteer time is not fungible and if someone feels particularly enthused about this idea they are free to pursue it. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Dec 15 23:35:05 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 15 Dec 2016 20:35:05 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <000FADA8-2D13-48D6-AF4C-02BA4302CD34@stufft.io> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> <000FADA8-2D13-48D6-AF4C-02BA4302CD34@stufft.io> Message-ID: <15EB5833-56A9-4495-91A5-1816C40C4658@twistedmatrix.com> > On Dec 15, 2016, at 8:33 PM, Donald Stufft wrote: > > >> On Dec 15, 2016, at 11:29 PM, Glyph Lefkowitz > wrote: >> >> User-curated package sets strikes me as the _lowest_ priority feature out of all of those > > > > I don?t think anyone in the PyPA is planning on working on this currently. It was a possible idea that was spawned from this thread. However the nature of volunteer OSS is that volunteer time is not fungible and if someone feels particularly enthused about this idea they are free to pursue it. I did very consciously choose my words there: "strikes me as", not "is" ;-) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Dec 16 00:20:42 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 23:20:42 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> Message-ID: On Thursday, December 15, 2016, Nick Coghlan wrote: > On 16 December 2016 at 05:50, Paul Moore > wrote: > > On 15 December 2016 at 19:13, Wes Turner > wrote: > >>> Just to add my POV, I also find your posts unhelpful, Wes. There's not > >>> enough information for me to evaluate what you say, and you offer no > >>> actual solutions to what's being discussed. > >> > >> > >> I could quote myself suggesting solutions in this thread, if you like? > > > > You offer lots of pointers to information. But that's different. > > Exactly. There are *lots* of information processing standards out > there, and lots of things we *could* provide natively that simply > aren't worth the hassle since folks that care can provide them as > "after market addons" for the audiences that considers them relevant. > > For example, a few things that can matter to different audiences are: > > - SPDX (Software Package Data Exchange) identifiers for licenses > - CPE (Common Product Enumeration) and SWID (Software Identification) > tags for published software > - DOI (Digital Object Identifier) tags for citation purposes > - Common Criteria certification for software supply chains These are called properties with RDFS. It takes very little effort to add additional properties. If the unqualified attribute is not listed in a JSONLD @context, it can still be added by specifying a URI > > I don't push for these upstream in distutils-sig not because I don't > think they're important in general, but because I *don't think they're > a priority for distutils-sig*. If you're teaching Python to school > students, or teaching engineers and scientists how to better analyse > their own data, or building a web service for yourself or your > employer, these kinds of things simply don't matter. #31 lists a number of advantages. OTOMH, CVE security reports could be linked to the project/package URI (and thus displayed along with the project detail page) > > The end users that care about them are well-positioned to tackle them > on their own (or pay other organisations to do it for them), and > because they span arbitrary publishing communities anyway, it doesn't > really matter all that much if any given publishing community > participates directly in the process (the only real beneficiaries are > the intermediaries that actively blur the distinctions between the > cooperative communities and the recalcitrant ones). Linked Data minimizes > > > Anyway, let's just agree to differ - I can skip your mails if they > > aren't helpful to me, and you don't need to bother about the fact that > > you're not getting your points across to me. > > I consider it fairly important that we have a reasonably common > understanding of the target userbase for direct consumption of PyPI > data, and what we expect to be supplied as third party services. It's > also important that we have a shared understanding of how to > constructively frame proposals for change. When I can afford the time, I'll again take a look at fixing the metadata specification once and for all by (1) defining an @context for the existing metadata, and (2) producing an additional pydist.jsonld TODO metadata document (because the releases are currently keyed by version), and (3) adding the model attribute and view to Warehouse. > > For the former, the Semantic Web, and folks that care about Semantic > Web concepts like "Linked Data" in the abstract sense are not part of > our primary audience. We don't go out of our way to make their lives > difficult, but "it makes semantic analysis easier" also isn't a > compelling rationale for change. Unfortunately, you types are not well-versed in the problems that Linked Data solves: it's all your data in your schema in your database; and URIs are far less useful than RAM-local references (pointers). See: BP-LD > > For the latter, some variants of constructive proposals look like: > > - "this kind of user has this kind of problem and this proposed > solution will help mitigate it this way (and, by the way, here's an > existing standard we can use)" > - "this feature exists in , it's really > valuable to users for , how about we offer it by > default?" > - "I wrote for myself, and I think it would also help others > for , can you help me make it more widely known and > available?" One could stuff additional metadata in # comments of a requirements.txt, but that would be an ad-hoc parsing scheme with a SPOF tool dependency. > They don't look like "Here's a bunch of technologies and organisations > that exist on the internet that may in some way potentially be > relevant to the management of a software distribution network", and > nor does it look like "This data modeling standard exists, so we > should use it, even though it doesn't actually simplify our lives or > our users' lives in any way, and in fact makes them more complicated". Those badges we all (!) add to our README.rst long_descriptions point to third-party services with lots of potentially structured linked data that is very relevant to curating a collection of resources: test coverage, build stats, discoverable documentation which could be searched en-masse, security vulnerability reports, downstream packages; but they're unfortunately just links; whereas they could be edges that other tools could make use of. > > > Who knows, one day I > > might find the time to look into JSON-LD, at which point I may or may > > not understand why you think it's such a useful tool for solving all > > these problems (in spite of the fact that no-one else seems to think > > the same...) It would be logically fallacious of me to, without an understanding of a web standard graph representation format, suggest that it's not sufficient (or ideally-suited) for these very use cases. #31 TODO somewhat laboriously lists ROI; Though I haven't yet had the time for an impact study. > > I *have* looked at JSON-LD (based primarily on Wes's original > suggestions), both from the perspective of the Python packaging > ecosystem specifically, as well as my day job working on software > supply chain management. I recognize your expertise and your preference for given Linux distributions. I can tell you that, while many of the linked data examples describe social graph applications regarding Bob and Alice, there are very many domains where Linked Data is worth learning: medicine (research, clinical), open government data (where tool-dependence is a no-no and a lost opportunity). When you have data in lots of different datasets, it really starts to make sense to: - use URIs as keys - use URIs as column names - recognize that you're just reimplementing graph semantics which are already well-solved (RDF, RDFS, OWL, and now JSONLD because JS) > > My verdict was that for managing a dependency graph implementation, it > ends up in the category of technologies that qualify as "interesting, > but not helpful". In many ways, it's the urllib2 of data linking - > just as urllib2 gives you a URL handling framework which you can > configure to handle HTTP rather than just providing a HTTP-specific > interface the way requests does [1], JSON-LD gives you a data linking > framework, which you can then use to define links between your data, > rather than just linking the data directly in a domain-appropriate > fashion. Using a framework for the sake of using a framework rather > than out of a genuine engineering need doesn't tend to lead to good > software systems. Interesting analogy. urllib, urlparse, urllib2, urllib3, requests; and now we have cert hostname checking. SemWeb standards are also layered. There were other standards for triples in JSON that do still exist, but none could map an existing JSON document to RDF with such flexibility. Followed by a truism. > > Wes seems to think that my perspective on this is born out of > ignorance, so repeatedly bringing it up may make me change my point of > view. However, our problems haven't changed, and the nature and > purpose of JSON-LD haven't changed, so it really won't - the one thing > that will change my mind is demonstrated popularity and utility of a > service that integrates raw PyPI data with JSON-LD and schema.org. > > Hence my suggestions (made with varying degrees of politeness) to go > build a dependency analysis service that extracts the dependency trees > from libraries.io, maps them to schema.org concepts in a way that > makes sense, and then demonstrate what that makes possible that can't > be done with the libraries.io data directly. A chicken-and-egg problem, ironically. There are many proprietary solutions for aggregating software quality information; all of which must write parsers and JOIN logic for each packaging ecosystem's ad-hoc partial graph implementations. When the key of a thing is a URI, other datasets which reference the same URI just magically join together: The justifying metadata for a package in a curated collection could just join with the actual package metadata (and the aforementioned datasources) > > Neither Wes nor anyone else needs anyone's permission to go do that, > and it will be far more enjoyable for all concerned than the status > quo where Wes is refusing to take "No, we've already looked at > JSON-LD, and we believe it adds needless complexity for no benefit > that we care about" for an answer by continuing to post about it > *here* rather than either venting his frustrations about our > collective lack of interest somewhere else, or else channeling that > frustration into building the system he wishes existed. If you've never written an @context for an existing JSON schema, I question both your assessment of complexity and your experience with sharing graph data with myriad applications; But that's irrelevant, Because here all I think I need is a table of dataset-local autoincrement IDs and some columns And ALTER TABLE migrations, And then someone else can write a parser for the schema I expose with my JSON REST API, So that I can JOIN this data with other useful datasets (In order to share a versioned Collection of CreativeWorks which already have URIs; Because I'm unsatisfied with requirements.txt Because it's line-based, And I can't just add additional attributes, And there's no key because indexes and editable URLs stuffed with checksums and eggwheel names, Oh and the JSON metadata specification is fixed and doesn't support URI attribute names So I can't just add additional attributes or values from a controlled vocabulary as-needed, Unless it's ReStructuredText-come-HTML (now with pypi:readme) Is that in PEP form? Someone should really put together an industry-group to produce some UML here, So our tools can talk, And JOIN on URIs With versions, platforms, and custom URI schemes, In order to curate a collection of packages, As a team, With group permissions, according to structured criteria (and comments!) Because "ld-signatures" A #LinkedMetaAnalyses (#LinkedReproducibility) application would similarly support curation of resources with URIs and defined criteria in order to elicit redundant expert evaluations of CreativeWorks (likely with JSONLD). In light of the reception here, that may be a better use of resources. I've said my piece, I'll leave you guys to find a solution to this use case which minimizes re-work and maximizes data integration potential. [Pip checksum docs, JSONLD] https://github.com/pypa/interoperability-peps/issues/31#issuecomment-160970112 > Cheers, > Nick. > > [1] http://www.curiousefficiency.org/posts/2016/08/what- > problem-does-it-solve.html > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, > Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 16 00:23:59 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 15:23:59 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: On 16 December 2016 at 14:29, Glyph Lefkowitz wrote: > On Dec 15, 2016, at 8:18 PM, Nick Coghlan wrote: > > At the beginning of your story you mentioned the GUI client - *that* is > the missing piece ;). I've been saying for years that we need a Python.app > that lets you easily bootstrap all this stuff: walk you through installing > C dev tools if your packages need them, present a GUI search interface to > finding packages, present a normal "file->open" dialog for selecting a > location for a new virtualenv, automatically pop open a terminal, launching > a Jupyter notebook whose kernel is pointed at said environment... > It isn't really, as we started looking at this for IDLE, and the entire current UX is just fundamentally beginner hostile: - virtual environments are hard - requirements files are hard - knowing what packages are trustworthy and worth your time is hard - limiting students to a set of "known safe" packages is hard - components that assume command line use are hard They're especially hard if the only way to distribute a fix is to release an entire new edition of CPython rather than having IDLE talk to a (preferably configurable) backend cloud service for updated instructions. So there's a reason so many learning and even full development environments are moving online - they let the service provider deal with all the hassles of providing an appropriately configured environment, while the students can focus on learning how to code, and the developers can focus on defining their application logic. However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience: - their default focus is on managing one environment per device - they both may require environment restarts for changes to take effect - they both reference an at least somewhat moderated back end (by Curse in the Curse client case, by Mozilla in the Firefox case) - they both incorporate popularity metrics and addon ratings into the client experience Mobile app store clients also share those four characteristics (where "number of installations" and "star ratings" are critically important to search rankings, but gaming the latter is mitigated by hiding the "Write a review" feature if you haven't actually installed the app anywhere) > User-curated package sets strikes me as the _lowest_ priority feature out > of all of those, if we are ordering by priority to deliver a good user > experience. I know "steam curators" have been brought up before - but > we're talking about adding curators (one of my least favorite features of > Steam, for what it's worth) before we've added "install game" ;-). > In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide). > Users might even figure out this sort of stuff for themselves if they are > given a discoverable API for things like search and installation of > packages. > That sounds a bit like agreement that we're still missing some of the backend pieces needed to make a beginner-friendly client really viable :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Dec 16 00:33:08 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 15 Dec 2016 23:33:08 -0600 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> Message-ID: On Saturday, December 10, 2016, Wes Turner wrote: > Here are some standardized (conda) package versions: https://github.com/ > jupyter/docker-stacks/blob/master/scipy-notebook/Dockerfile > > IDK how they choose packages - what "criteria for inclusion" - for the kaggle/docker-python Dockerfile: https://github.com/Kaggle/docker-python/blob/master/Dockerfile (Ubuntu, Conda, *, TPOT) > On Thursday, December 8, 2016, Wes Turner > wrote: > >> >> >> On Thursday, December 8, 2016, Nick Coghlan wrote: >> >>> Putting the conclusion first, I do see value in better publicising >>> "Recommended libraries" based on some automated criteria like: >>> >>> - recommended in the standard library documentation >>> - available via 1 or more cross-platform commercial Python redistributors >>> - available via 1 or more Linux distro vendors >>> - available via 1 or more web service development platforms >>> >>> >> So these would be attributes tracked by a project maintainer and verified >> by the known-good-set maintainer? Or? >> >> (Again, here I reach for JSONLD. "count n" is only so useful; *which* >> {[re]distros, platforms, heartfelt testimonials from incredible experts} >> URLs ) >> >> - test coverage >> - seclist contact info AND procedures >> - more than one super admin maintainer >> - what other criteria should/could/can we use to vet open source >> libraries? >> >> >>> That would be a potentially valuable service for folks new to the >>> world of open source that are feeling somewhat overwhelmed by the >>> sheer number of alternatives now available to them. >>> >>> However, I also think that would better fit in with the aims of an >>> open source component tracking community like libraries.io than it >>> does a publisher-centric community like distutils-sig. >> >> >> IDK if libraries are really in scope for stackshare. The feature >> upcoming/down voting is pretty cool. >> >> https://stackshare.io/python >> >> >>> >>> The further comments below are just a bit more background on why I >>> feel the integration testing aspect of the suggestion isn't likely to >>> be particularly beneficial :) >> >> >> A catch-all for testing bits from application-specific integration test >> suites could be useful (and would likely require at least docker-compose, >> dox, kompose for working with actual data stores) >> >> >>> >>> On 9 December 2016 at 01:10, Barry Warsaw wrote: >>> > Still, there may be value in inter-Python package compatibility tests, >>> but >>> > it'll take serious engineering effort (i.e. $ and time), ongoing >>> maintenance, >>> > ongoing effort to fix problems, and tooling to gate installability of >>> failing >>> > packages (with overrides for downstreams which don't care or already >>> expend >>> > such effort). >>> >>> I think this is really the main issue, as both desktop and server >>> environments are moving towards the integrated platform + isolated >>> applications approach popularised by mobile devices. >>> >>> That means we end up with two very different variants of automated >>> integration testing: >>> >>> - the application focused kind offered by the likes of requires.io and >>> pyup.io (i.e. monitor for dependency updates, submit PRs to trigger >>> app level CI) >>> - the platform focused kind employed by distro vendors (testing all >>> the platform components work together, including the app isolation >>> features) >>> >>> The first kind makes sense if you're building something that runs *on* >>> platforms (Docker containers, Snappy or FlatPak apps, web services, >>> mobile apps, etc). >>> >>> The second kind inevitably ends up intertwined with the component >>> review and release engineering systems of the particular platform, so >>> it becomes really hard to collaborate cross-platform outside the >>> context of specific projects like OpenStack that provide clear >>> definitions for "What components do we collectively depend on that we >>> need to test together?" and "What does 'working' mean in the context >>> of this project?". >>> >>> Accordingly, for an initiative like this to be successful, it would >>> need to put some thought up front into the questions of: >>> >>> 1. Who are the intended beneficiaries of the proposal? >>> 2. What problem does it address that will prompt them to contribute >>> time and/or money to solving it? >>> 3. What do we expect people to be able to *stop doing* if the project >>> proves successful? >>> >>> For platform providers, a generic "stdlib++" project wouldn't really >>> reduce the amount of integration testing we'd need to do ourselves (we >>> already don't test arbitrary combinations of dependencies, just the >>> ones we provide at any given point in time). >>> >>> For application and service developers, the approach of pinning >>> dependencies to specific versions and treating updates like any other >>> source code change already works well in most cases. >>> >>> That leaves library and framework developers, who currently tend to >>> adopt the policy of "for each version of Python that we support, we >>> test against the latest versions of our dependencies that were >>> available at the time we ran the test", leaving testing against older >>> versions to platform providers. If there's a key related framework >>> that also provides LTS versions (e.g. Django), then some folks may add >>> that to their test matrix as well. >>> >>> In that context, "Only breaks backwards compatibility for compelling >>> reasons" becomes a useful long term survival trait for libraries and >>> frameworks, as gratuitous breakages are likely to lead to people >>> migrating away from particularly unreliable dependencies. >> >> >> Sometimes, when there are no active maintainers for a feature, it makes >> sense to deprecate and/or split that functionality / untested cruft out to >> a different package. >> >> >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Dec 16 05:57:23 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 16 Dec 2016 02:57:23 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: > On Dec 15, 2016, at 9:23 PM, Nick Coghlan wrote: > > On 16 December 2016 at 14:29, Glyph Lefkowitz > wrote: >> On Dec 15, 2016, at 8:18 PM, Nick Coghlan > wrote: > > At the beginning of your story you mentioned the GUI client - that is the missing piece ;). I've been saying for years that we need a Python.app that lets you easily bootstrap all this stuff: walk you through installing C dev tools if your packages need them, present a GUI search interface to finding packages, present a normal "file->open" dialog for selecting a location for a new virtualenv, automatically pop open a terminal, launching a Jupyter notebook whose kernel is pointed at said environment... > > It isn't really, as we started looking at this for IDLE, and the entire current UX is just fundamentally beginner hostile: > > - virtual environments are hard > - requirements files are hard > - knowing what packages are trustworthy and worth your time is hard > - limiting students to a set of "known safe" packages is hard > - components that assume command line use are hard > > They're especially hard if the only way to distribute a fix is to release an entire new edition of CPython rather than having IDLE talk to a (preferably configurable) backend cloud service for updated instructions. > > So there's a reason so many learning and even full development environments are moving online - they let the service provider deal with all the hassles of providing an appropriately configured environment, while the students can focus on learning how to code, and the developers can focus on defining their application logic. None of what you're saying is wrong here, so I don't want to disagree. But, I think this is just one perspective; i.e. moving to a cloud environment is one approach to providing a more circumscribed environment, but embracing endpoint sandboxing is another. For example, learning how to use Xcode is a fundamentally different (and easier!) sort of experience than learning the traditional UNIX development pipeline, due in large part to the fact that it provides a unified, discoverable interface. This is despite the fact that Xcode projects are actually substantially more complex than their UNIX-y equivalents, due to the high levels of coupling and complexity in the way that you have to interface with certain system services (signing with entitlements, bundle metadata, etc). You still have to retrieve many resources from the cloud - simulators, documentation, SDKs - but the UI tells you that you need those things, and straightforwardly automates the process of getting them. Everything else that goes into a development project is not "environment setup", but a part of the Xcode project itself. Similarly, version control (a git repository) is nearly implicitly a part of the project. It's tricky to even create one without a VCS backing it any more. Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early? This might all be way too much work, but I think it's important to remember it's possible. > However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience: > > - their default focus is on managing one environment per device In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing... > - they both may require environment restarts for changes to take effect ... one could just put a little blinking red light on any jupyter windows whose kernels need to be restarted :) ... > - they both reference an at least somewhat moderated back end (by Curse in the Curse client case, by Mozilla in the Firefox case) > - they both incorporate popularity metrics and addon ratings into the client experience I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :). > Mobile app store clients also share those four characteristics (where "number of installations" and "star ratings" are critically important to search rankings, but gaming the latter is mitigated by hiding the "Write a review" feature if you haven't actually installed the app anywhere) > > User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-). > > In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide). I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important? > Users might even figure out this sort of stuff for themselves if they are given a discoverable API for things like search and installation of packages. > > That sounds a bit like agreement that we're still missing some of the backend pieces needed to make a beginner-friendly client really viable :) Oops. I meant to say "UI". "beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a limiting factor in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with. But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 16 08:07:22 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Dec 2016 23:07:22 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: On 16 December 2016 at 20:57, Glyph Lefkowitz wrote: > > Anyhow, Xcode is far from perfect - many of the places it touches the UNIX > pipeline are extremely sharp edges you can easily impale yourself on (and > don't get me started about codesigning) - but it nevertheless points at a > different potential direction. For example; why expose the concept of a > "virtual environment" directly at all? "New Project" could just create a > requirements.txt and a setup.py for you, alongside a git repo and a > virtualenv for that project. Or, the UI could be geared towards setting up > a tox.ini rather than a virtualenv, and run everything through tox so it's > in an isolated environment with defined requirements. This is a best > practice anyway so why not make it easier to start early? > > This might all be way too much work, but I think it's important to > remember it's possible. > Yeah, I think we agree more than we disagree here. The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*. XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS. Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers. That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project. [1] http://rodeo.yhat.com/docs/ [2] http://codewith.mu/ > However, the reason I brought up the Curse and Firefox GUI examples was to > emphasise the problems they hide from the default rich client experience: > > - their default focus is on managing one environment per device > > > In the analogous Python tool, one could replace "per device" with "per > project" - and perhaps have a "default project" so something useful could > happen even before you've decided what you're doing... > But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need. I thought this thread was already interminable, I look forward to reading > the never-ending rest of it now that you've raised the grim spectre of the > PyPI user-ratings feature from the dead :). > All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline. > User-curated package sets strikes me as the _lowest_ priority feature out >> of all of those, if we are ordering by priority to deliver a good user >> experience. I know "steam curators" have been brought up before - but >> we're talking about adding curators (one of my least favorite features of >> Steam, for what it's worth) before we've added "install game" ;-). >> > > In many educational contexts, adding "install game" without support for > institutional curators of some kind is a complete non-starter (even if > those curators are a collaborative community like a Linux distribution, > there's still more accountability than software publishing sites like PyPI > tend to provide). > > > I initially wanted to disagree when I read this, but I'm not actually sure > what educational contexts you're talking about, and why "accountability" is > important? > Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers. Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that. > "beginner" is a direction, and not a fixed position; many people more > "beginner" than the current audience could be well-served by a discoverable > initial project-creation and REPL UI. While I don't doubt that some > backend pieces might help (although I still don't see how the one being > discussed would), I also think that it would be very hard to say that the > back-end is a *limiting factor* in UX improvement for the Python > onboarding process; the front end could move quite a bit up the value chain > without touching any of the various backends it would need to interact with. > > But of course, if I really wanted to make this point, I'd just write it; > dstufft is certainly right that volunteer time is not fungible. If I'm > lucky, I'll have the time to do that at some point, since my efforts to > convince someone else that this is the high-value target have been failing > for some years now ;). > I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :) Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became: 1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command Cheers, Nick. [1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Dec 16 08:51:39 2016 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Dec 2016 13:51:39 +0000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: I'm also a visual studio code fan. It is the first editor I've tried that feels lightweight like Vim but has the power of many plugins. That, and the text rendering is excellent. https://pypi.python.org/pypi/Stallion is a lovely GUI package manager. One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py. On Fri, Dec 16, 2016 at 8:07 AM Nick Coghlan wrote: > On 16 December 2016 at 20:57, Glyph Lefkowitz > wrote: > > > Anyhow, Xcode is far from perfect - many of the places it touches the UNIX > pipeline are extremely sharp edges you can easily impale yourself on (and > don't get me started about codesigning) - but it nevertheless points at a > different potential direction. For example; why expose the concept of a > "virtual environment" directly at all? "New Project" could just create a > requirements.txt and a setup.py for you, alongside a git repo and a > virtualenv for that project. Or, the UI could be geared towards setting up > a tox.ini rather than a virtualenv, and run everything through tox so it's > in an isolated environment with defined requirements. This is a best > practice anyway so why not make it easier to start early? > > This might all be way too much work, but I think it's important to > remember it's possible. > > > Yeah, I think we agree more than we disagree here. The main thing is that > one of the key ways newcomer-friendly environments make themselves more > approachable is to *constrain choice*. > > XCode usability benefits from being Apple-centric. Ditto for Visual Studio > and MS. > > Linux and Python, by contrast, were both born out of a DIY culture where > folks being free to choose their own tools was initially perceived solely > as a highly desirable feature, rather than as a potential barrier to entry > for newcomers. > > That means there's an argument to be made that something like YHat's Rodeo > [1] might be a better starting point for data analytics in Python than > jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] > exists as a dedicated tool for folks learning Python by way of the > micro:bit project. > > [1] http://rodeo.yhat.com/docs/ > [2] http://codewith.mu/ > > However, the reason I brought up the Curse and Firefox GUI examples was to > emphasise the problems they hide from the default rich client experience: > > - their default focus is on managing one environment per device > > > In the analogous Python tool, one could replace "per device" with "per > project" - and perhaps have a "default project" so something useful could > happen even before you've decided what you're doing... > > > But we've immediately bumped the complexity level up in doing so, and it's > a level of complexity that many people initially spending all of their > development time on a single project may not need. > > I thought this thread was already interminable, I look forward to reading > the never-ending rest of it now that you've raised the grim spectre of the > PyPI user-ratings feature from the dead :). > > > All the arguments against integrating user ratings into a service that's > focused on lowering barriers to publication still hold, so I'm really just > noting that that decision to create a friendlier publishing environment > *does* introduce some additional constraints elsewhere in the distribution > pipeline. > > User-curated package sets strikes me as the _lowest_ priority feature out > of all of those, if we are ordering by priority to deliver a good user > experience. I know "steam curators" have been brought up before - but > we're talking about adding curators (one of my least favorite features of > Steam, for what it's worth) before we've added "install game" ;-). > > > In many educational contexts, adding "install game" without support for > institutional curators of some kind is a complete non-starter (even if > those curators are a collaborative community like a Linux distribution, > there's still more accountability than software publishing sites like PyPI > tend to provide). > > > I initially wanted to disagree when I read this, but I'm not actually sure > what educational contexts you're talking about, and why "accountability" is > important? > > > Schools, mainly. Lots of administrators are still scared of the internet, > so one of the attractions of things like Raspberry Pi is that the software > updates come from Debian rather than directly from the software publishers. > > Sometimes you can get away with "What the bureaucracy doesn't know won't > hurt it", but it's more convenient when teachers don't have to do that. > > > "beginner" is a direction, and not a fixed position; many people more > "beginner" than the current audience could be well-served by a discoverable > initial project-creation and REPL UI. While I don't doubt that some > backend pieces might help (although I still don't see how the one being > discussed would), I also think that it would be very hard to say that the > back-end is a *limiting factor* in UX improvement for the Python > onboarding process; the front end could move quite a bit up the value chain > without touching any of the various backends it would need to interact with. > > But of course, if I really wanted to make this point, I'd just write it; > dstufft is certainly right that volunteer time is not fungible. If I'm > lucky, I'll have the time to do that at some point, since my efforts to > convince someone else that this is the high-value target have been failing > for some years now ;). > > > I'll confess that one of my ulterior motives for encouraging computing > teachers to engage more directly with the upstream Python community is that > I kinda hope we'll eventually run into one that either decides none of the > current editors are good enough and creates their own, or else decides that > "create the introductory editor that you wish you had when you started > learning to program" might make a good collaborative student project :) > > Alternatively, I've recently started using Visual Studio Code as my editor > for work [1], and it seems likely that would be hackable enough for someone > to create a plugin that bootstrapped a complete Python toolchain such that > the bootstrapping flow became: > > 1. Install VSCode > 2. Install the "New to Python" plugin > 3. Run the plugin's "Setup Python Environment" command > > Cheers, > Nick. > > [1] Oh the irony that the first cross-platform editor I've tried that I > actually think looks nice and find pleasant to use on Fedora was released > by Microsoft :) > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Fri Dec 16 09:15:57 2016 From: steve.dower at python.org (Steve Dower) Date: Fri, 16 Dec 2016 06:15:57 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: "Alternatively, I've recently started using Visual Studio Code as my editor for work ..." FWIW, the long-term Python story in VSCode is currently (largely) one of my responsibilities, and that bootstrapping flow is exactly one of the pieces I really want to put in. Unfortunately, nobody has let me have any engineers to implement it yet :( (mostly for big-company-politics reasons, rather than hiring trouble) Top-posted from my Windows Phone -----Original Message----- From: "Nick Coghlan" Sent: ?12/?16/?2016 5:08 To: "Glyph Lefkowitz" Cc: "Barry Warsaw" ; "DistUtils mailing list" Subject: Re: [Distutils] Maintaining a curated set of Python packages On 16 December 2016 at 20:57, Glyph Lefkowitz wrote: Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early? This might all be way too much work, but I think it's important to remember it's possible. Yeah, I think we agree more than we disagree here. The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*. XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS. Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers. That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project. [1] http://rodeo.yhat.com/docs/ [2] http://codewith.mu/ However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience: - their default focus is on managing one environment per device In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing... But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need. I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :). All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline. User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-). In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide). I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important? Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers. Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that. "beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a limiting factor in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with. But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;). I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :) Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became: 1. Install VSCode 2. Install the "New to Python" plugin 3. Run the plugin's "Setup Python Environment" command Cheers, Nick. [1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Dec 16 13:26:51 2016 From: brett at python.org (Brett Cannon) Date: Fri, 16 Dec 2016 18:26:51 +0000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: If people are serious about trying to prototype this stuff then the easiest way might be coming up with shell scripts that do the prompting if it's faster to iterate that way than doing a full-blown GUI. Now that WIndows 10 has WSL/Bash it means for the first time all 3 major OSs have a common shell people can work from. You could even go as far as making the shell scripts be Cookiecutter templates such that people can experiment with things being included/left out (e.g. an instructor wants to require Python 3.5, no git, and have people work from a virtual environment and so they generate the shell script everyone is told to run to get things going/verify the student's system is set up properly). On Fri, 16 Dec 2016 at 05:52 Daniel Holth wrote: > I'm also a visual studio code fan. It is the first editor I've tried that > feels lightweight like Vim but has the power of many plugins. That, and the > text rendering is excellent. > > https://pypi.python.org/pypi/Stallion is a lovely GUI package manager. > > One possibility to consider is that virtualenv itself is a bad idea. Why > should the Python interpreter executable, rather than the program being > run, determine the set of packages that is available for import? It is > confusing and inconvenient to have to deal with environments at all. Yes, > even if you are using a helper. Maybe there can be a better way to manage > dependencies that is not completely disjoint from setup.py. > That just sounds like node_modules/ and I personally don't want to go down that route. If you view the interpreter as another component of an app then the disconnect doesn't seem so nutty (at least in my head; at that point it's just another /usr/local to me). -Brett > > On Fri, Dec 16, 2016 at 8:07 AM Nick Coghlan wrote: > > On 16 December 2016 at 20:57, Glyph Lefkowitz > wrote: > > > Anyhow, Xcode is far from perfect - many of the places it touches the UNIX > pipeline are extremely sharp edges you can easily impale yourself on (and > don't get me started about codesigning) - but it nevertheless points at a > different potential direction. For example; why expose the concept of a > "virtual environment" directly at all? "New Project" could just create a > requirements.txt and a setup.py for you, alongside a git repo and a > virtualenv for that project. Or, the UI could be geared towards setting up > a tox.ini rather than a virtualenv, and run everything through tox so it's > in an isolated environment with defined requirements. This is a best > practice anyway so why not make it easier to start early? > > This might all be way too much work, but I think it's important to > remember it's possible. > > > Yeah, I think we agree more than we disagree here. The main thing is that > one of the key ways newcomer-friendly environments make themselves more > approachable is to *constrain choice*. > > XCode usability benefits from being Apple-centric. Ditto for Visual Studio > and MS. > > Linux and Python, by contrast, were both born out of a DIY culture where > folks being free to choose their own tools was initially perceived solely > as a highly desirable feature, rather than as a potential barrier to entry > for newcomers. > > That means there's an argument to be made that something like YHat's Rodeo > [1] might be a better starting point for data analytics in Python than > jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] > exists as a dedicated tool for folks learning Python by way of the > micro:bit project. > > [1] http://rodeo.yhat.com/docs/ > [2] http://codewith.mu/ > > However, the reason I brought up the Curse and Firefox GUI examples was to > emphasise the problems they hide from the default rich client experience: > > - their default focus is on managing one environment per device > > > In the analogous Python tool, one could replace "per device" with "per > project" - and perhaps have a "default project" so something useful could > happen even before you've decided what you're doing... > > > But we've immediately bumped the complexity level up in doing so, and it's > a level of complexity that many people initially spending all of their > development time on a single project may not need. > > I thought this thread was already interminable, I look forward to reading > the never-ending rest of it now that you've raised the grim spectre of the > PyPI user-ratings feature from the dead :). > > > All the arguments against integrating user ratings into a service that's > focused on lowering barriers to publication still hold, so I'm really just > noting that that decision to create a friendlier publishing environment > *does* introduce some additional constraints elsewhere in the distribution > pipeline. > > User-curated package sets strikes me as the _lowest_ priority feature out > of all of those, if we are ordering by priority to deliver a good user > experience. I know "steam curators" have been brought up before - but > we're talking about adding curators (one of my least favorite features of > Steam, for what it's worth) before we've added "install game" ;-). > > > In many educational contexts, adding "install game" without support for > institutional curators of some kind is a complete non-starter (even if > those curators are a collaborative community like a Linux distribution, > there's still more accountability than software publishing sites like PyPI > tend to provide). > > > I initially wanted to disagree when I read this, but I'm not actually sure > what educational contexts you're talking about, and why "accountability" is > important? > > > Schools, mainly. Lots of administrators are still scared of the internet, > so one of the attractions of things like Raspberry Pi is that the software > updates come from Debian rather than directly from the software publishers. > > Sometimes you can get away with "What the bureaucracy doesn't know won't > hurt it", but it's more convenient when teachers don't have to do that. > > > "beginner" is a direction, and not a fixed position; many people more > "beginner" than the current audience could be well-served by a discoverable > initial project-creation and REPL UI. While I don't doubt that some > backend pieces might help (although I still don't see how the one being > discussed would), I also think that it would be very hard to say that the > back-end is a *limiting factor* in UX improvement for the Python > onboarding process; the front end could move quite a bit up the value chain > without touching any of the various backends it would need to interact with. > > But of course, if I really wanted to make this point, I'd just write it; > dstufft is certainly right that volunteer time is not fungible. If I'm > lucky, I'll have the time to do that at some point, since my efforts to > convince someone else that this is the high-value target have been failing > for some years now ;). > > > I'll confess that one of my ulterior motives for encouraging computing > teachers to engage more directly with the upstream Python community is that > I kinda hope we'll eventually run into one that either decides none of the > current editors are good enough and creates their own, or else decides that > "create the introductory editor that you wish you had when you started > learning to program" might make a good collaborative student project :) > > Alternatively, I've recently started using Visual Studio Code as my editor > for work [1], and it seems likely that would be hackable enough for someone > to create a plugin that bootstrapped a complete Python toolchain such that > the bootstrapping flow became: > > 1. Install VSCode > 2. Install the "New to Python" plugin > 3. Run the plugin's "Setup Python Environment" command > > Cheers, > Nick. > > [1] Oh the irony that the first cross-platform editor I've tried that I > actually think looks nice and find pleasant to use on Fedora was released > by Microsoft :) > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Dec 16 15:40:14 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 16 Dec 2016 12:40:14 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: <7722218D-F361-4C30-AE0C-E94FD965DFF2@twistedmatrix.com> > On Dec 16, 2016, at 5:51 AM, Daniel Holth wrote: > > I'm also a visual studio code fan. It is the first editor I've tried that feels lightweight like Vim but has the power of many plugins. That, and the text rendering is excellent. > > https://pypi.python.org/pypi/Stallion is a lovely GUI package manager. There's a lot to like here - no disrespect to the Stallion team - but it's worth remembering this lesson from Havoc Pennington: https://blog.ometer.com/2016/01/26/the-dangerous-ui-team/ The major reason UI is important for this use-case - onboarding of new people to Python programming - is to give them discoverability on terms they're already familiar with. That means that the first "UI" element has to be a cross-platform UI bundle. Stallion is still a thing you have to install (and from what I can see, a thing you have to install into a virtualenv?) > One possibility to consider is that virtualenv itself is a bad idea. Why should the Python interpreter executable, rather than the program being run, determine the set of packages that is available for import? It is confusing and inconvenient to have to deal with environments at all. Yes, even if you are using a helper. Maybe there can be a better way to manage dependencies that is not completely disjoint from setup.py. I can see why you'd say that, but I disagree. I think the name "virtualenv" is really confusing, but the general idea of "it's the interpreter and not the app" is a very powerful concept because you can run a REPL (or a notebook, or a debugger, or a doc generator, or any other dev tool) in the same context as your application code, without actually loading or executing any specific thing from your application code. Virtualenv also lets you easily control which Python version or interpreter (hello, pypy!) is being used in each context. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Dec 16 15:55:48 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 16 Dec 2016 12:55:48 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: > On Dec 16, 2016, at 5:07 AM, Nick Coghlan wrote: > > On 16 December 2016 at 20:57, Glyph Lefkowitz > wrote: > > Anyhow, Xcode is far from perfect - many of the places it touches the UNIX pipeline are extremely sharp edges you can easily impale yourself on (and don't get me started about codesigning) - but it nevertheless points at a different potential direction. For example; why expose the concept of a "virtual environment" directly at all? "New Project" could just create a requirements.txt and a setup.py for you, alongside a git repo and a virtualenv for that project. Or, the UI could be geared towards setting up a tox.ini rather than a virtualenv, and run everything through tox so it's in an isolated environment with defined requirements. This is a best practice anyway so why not make it easier to start early? > > This might all be way too much work, but I think it's important to remember it's possible. > > Yeah, I think we agree more than we disagree here. Quite. But the devil's in the details :). > The main thing is that one of the key ways newcomer-friendly environments make themselves more approachable is to *constrain choice*. I think framing this as "constraint" is a little misleading. In a sense it is a constraint, but a better way to think of it is: provide a reasonable default. Right now, the "default UI" that most users get is a bare bash prompt where commands like 'pip install' fail with an error for no reason they can discern. They can still choose to inject a different tool at any point in the process (after all, we're talking about frontends which create existing concepts like virtualenvs and package installations) if they so choose; they just get a default that does something - anything - useful. > XCode usability benefits from being Apple-centric. Ditto for Visual Studio and MS. > > Linux and Python, by contrast, were both born out of a DIY culture where folks being free to choose their own tools was initially perceived solely as a highly desirable feature, rather than as a potential barrier to entry for newcomers. > > That means there's an argument to be made that something like YHat's Rodeo [1] might be a better starting point for data analytics in Python than jumping straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a dedicated tool for folks learning Python by way of the micro:bit project. > > [1] http://rodeo.yhat.com/docs/ > [2] http://codewith.mu/ Minor point - nobody should use Mu yet, at least not on the mac: https://github.com/mu-editor/mu/issues/206 :) More significantly, I think any text editor will do just fine (as long as it's not Emacs or Vim) - I've had great success with http://www.barebones.com/products/textwrangler/ , and even Notepad will do in a pinch. There are already pretty good integration points where editors can be told to open specific files. One of my frustrations with the educational ecosystem is the focus on the (quite difficult) problem of providing students with a fully integrated text editing / script running / debugging environment, rather than figuring out how to orchestrate and launch the quite powerful and sophisticated tools we already have. >> However, the reason I brought up the Curse and Firefox GUI examples was to emphasise the problems they hide from the default rich client experience: >> >> - their default focus is on managing one environment per device > > In the analogous Python tool, one could replace "per device" with "per project" - and perhaps have a "default project" so something useful could happen even before you've decided what you're doing... > > But we've immediately bumped the complexity level up in doing so, and it's a level of complexity that many people initially spending all of their development time on a single project may not need. I think we're underestimating potential programming students. The idea of managing multiple documents is likely something they're familiar with from word processing apps. If not, then fine - we can start them off with a default document. > I thought this thread was already interminable, I look forward to reading the never-ending rest of it now that you've raised the grim spectre of the PyPI user-ratings feature from the dead :). > > All the arguments against integrating user ratings into a service that's focused on lowering barriers to publication still hold, so I'm really just noting that that decision to create a friendlier publishing environment *does* introduce some additional constraints elsewhere in the distribution pipeline. >> User-curated package sets strikes me as the _lowest_ priority feature out of all of those, if we are ordering by priority to deliver a good user experience. I know "steam curators" have been brought up before - but we're talking about adding curators (one of my least favorite features of Steam, for what it's worth) before we've added "install game" ;-). >> >> In many educational contexts, adding "install game" without support for institutional curators of some kind is a complete non-starter (even if those curators are a collaborative community like a Linux distribution, there's still more accountability than software publishing sites like PyPI tend to provide). > > I initially wanted to disagree when I read this, but I'm not actually sure what educational contexts you're talking about, and why "accountability" is important? > > Schools, mainly. Lots of administrators are still scared of the internet, so one of the attractions of things like Raspberry Pi is that the software updates come from Debian rather than directly from the software publishers. > > Sometimes you can get away with "What the bureaucracy doesn't know won't hurt it", but it's more convenient when teachers don't have to do that. I think it's very good that we have people focused on institutional concerns. My focus is somewhere else - peer learning. I want students to be able to take what they've learned in class and easily apply it in a personally useful context. And frankly, I think this is useful to a huge audience of people who aren't exactly "students" - people who already sort of know how computers work, who spend all day long sitting in Excel and writing macros (because the UI to do so is discoverable!) and are keen to learn something better to actually do tasks that they have in their job, not just to learn to code. > "beginner" is a direction, and not a fixed position; many people more "beginner" than the current audience could be well-served by a discoverable initial project-creation and REPL UI. While I don't doubt that some backend pieces might help (although I still don't see how the one being discussed would), I also think that it would be very hard to say that the back-end is a limiting factor in UX improvement for the Python onboarding process; the front end could move quite a bit up the value chain without touching any of the various backends it would need to interact with. > > But of course, if I really wanted to make this point, I'd just write it; dstufft is certainly right that volunteer time is not fungible. If I'm lucky, I'll have the time to do that at some point, since my efforts to convince someone else that this is the high-value target have been failing for some years now ;). > > I'll confess that one of my ulterior motives for encouraging computing teachers to engage more directly with the upstream Python community is that I kinda hope we'll eventually run into one that either decides none of the current editors are good enough and creates their own, or else decides that "create the introductory editor that you wish you had when you started learning to program" might make a good collaborative student project :) I think we're going to remain stuck in our current quagmire until we let go of the idea of writing a text editor :). An all-singing all-dancing integrated development environment would be great. But editing text is a problem that's already solvable with a huge plethora of tools, most of them free. The tricky part where everybody gets stuck is making 'pip' or 'python' in a command-line window do anything useful at all. > Alternatively, I've recently started using Visual Studio Code as my editor for work [1], and it seems likely that would be hackable enough for someone to create a plugin that bootstrapped a complete Python toolchain such that the bootstrapping flow became: > > 1. Install VSCode > 2. Install the "New to Python" plugin > 3. Run the plugin's "Setup Python Environment" command > > Cheers, > Nick. > > [1] Oh the irony that the first cross-platform editor I've tried that I actually think looks nice and find pleasant to use on Fedora was released by Microsoft :) Turnabout is fair play - maybe 5 years from now we'll all be lamenting the terrible state of Python on UNIX and wishing it was as well-supported as Windows :). -g -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Dec 16 16:14:30 2016 From: dholth at gmail.com (Daniel Holth) Date: Fri, 16 Dec 2016 21:14:30 +0000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <7722218D-F361-4C30-AE0C-E94FD965DFF2@twistedmatrix.com> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> <7722218D-F361-4C30-AE0C-E94FD965DFF2@twistedmatrix.com> Message-ID: On Fri, Dec 16, 2016 at 3:40 PM Glyph Lefkowitz wrote: > On Dec 16, 2016, at 5:51 AM, Daniel Holth wrote: > > I'm also a visual studio code fan. It is the first editor I've tried that > feels lightweight like Vim but has the power of many plugins. That, and the > text rendering is excellent. > > https://pypi.python.org/pypi/Stallion is a lovely GUI package manager. > > > There's a lot to like here - no disrespect to the Stallion team - but it's > worth remembering this lesson from Havoc Pennington: > https://blog.ometer.com/2016/01/26/the-dangerous-ui-team/ > > The *major* reason UI is important for this use-case - onboarding of new > people to Python programming - is to give them *discoverability* on terms > they're already familiar with. That means that the first "UI" element has > to be a cross-platform UI bundle. Stallion is still a thing you have to > install (and from what I can see, a thing you have to install into a > virtualenv?) > > One possibility to consider is that virtualenv itself is a bad idea. Why > should the Python interpreter executable, rather than the program being > run, determine the set of packages that is available for import? It is > confusing and inconvenient to have to deal with environments at all. Yes, > even if you are using a helper. Maybe there can be a better way to manage > dependencies that is not completely disjoint from setup.py. > > > I can see why you'd say that, but I disagree. I think the *name* "virtualenv" > is really confusing, but the general idea of "it's the interpreter and not > the app" is a very powerful concept because you can run a REPL (or a > notebook, or a debugger, or a doc generator, or any other dev tool) in the > same *context* as your application code, without actually loading or > executing any specific thing from your application code. Virtualenv also > lets you easily control which Python version or interpreter (hello, pypy!) > is being used in each context. > My point is really that virtualenv both causes and solves problems. There's jaraco's thing I was trying to remember https://pypi.org/project/rwt/ which is a creative alternative to virtualenv for some situations. I wish there was a smoother path between virtualenv develop-against-pypi-libraries and end-user application deployment. Stallion has always looked cool but it hasn't been updated in a few years, I admit I never used it "for real". Don't know what it would it take to make a useful GUI package manager. Interactive dependency conflict resolution graphs? Able to install into remote environments? Maybe the command line version will always be easier to use. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Dec 16 16:38:39 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 17 Dec 2016 10:38:39 +1300 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: <58545EDF.7010905@canterbury.ac.nz> Glyph Lefkowitz wrote: > > "New Project" could > just create a requirements.txt and a setup.py for you, alongside a git > repo and a virtualenv for that project. Or, the UI could be geared > towards setting up a tox.ini rather than a virtualenv, and run > everything through tox so it's in an isolated environment with defined > requirements. I'd be very interested in something like this. I'm not a big fan of IDEs generally, but one feature I do appreciate greatly is having a one-button "build" process that creates a distributable app bundled with everything it needs, and be assured it will work on someone else's machine. That's currently rather difficult to do with Python in any but the simplest cases, even for a single platform. Cross-platform is even worse. +1 on providing some tools to make it easier. -- Greg From ncoghlan at gmail.com Sat Dec 17 02:29:48 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Dec 2016 17:29:48 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <7722218D-F361-4C30-AE0C-E94FD965DFF2@twistedmatrix.com> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> <7722218D-F361-4C30-AE0C-E94FD965DFF2@twistedmatrix.com> Message-ID: On 17 December 2016 at 06:40, Glyph Lefkowitz wrote: > On Dec 16, 2016, at 5:51 AM, Daniel Holth wrote: > > One possibility to consider is that virtualenv itself is a bad idea. Why > should the Python interpreter executable, rather than the program being > run, determine the set of packages that is available for import? It is > confusing and inconvenient to have to deal with environments at all. Yes, > even if you are using a helper. Maybe there can be a better way to manage > dependencies that is not completely disjoint from setup.py. > > > I can see why you'd say that, but I disagree. I think the *name* "virtualenv" > is really confusing, but the general idea of "it's the interpreter and not > the app" is a very powerful concept because you can run a REPL (or a > notebook, or a debugger, or a doc generator, or any other dev tool) in the > same *context* as your application code, without actually loading or > executing any specific thing from your application code. Virtualenv also > lets you easily control which Python version or interpreter (hello, pypy!) > is being used in each context. > I'll also note that VSCode's Python plugin will find virtual environments that are located inside the project directory by default. That approach of "the virtualenv is inside the project directory" is probably a decent pattern to recommend as a default, since it aligns with the way a lot of IDEs (including VSCode itself) already work. When you use that model, rather than being something you have to think about explicitly, the "Python virtual environment" just becomes an implementation detail of how the IDE manages your application dependencies, with the added bonus that *if you want to*, you can re-use that environment independently of both the application *and* the IDE. And while I personally prefer to keep the notion of "project" and "environment" more explicitly separate (i.e. I have an M:N mapping between a collection of virtualenvs centrally managed with vex and the various projects in my devel, fedoradevel and rhdevel folders, hence [1]), I believe that level of complexity in a local dev setup isn't really normal even for experienced programmers, let alone folks that are still early in the learning process. Cheers, Nick. [1] https://github.com/DonJayamanne/pythonVSCode/issues/569 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From elvis.stansvik at orexplore.com Tue Dec 20 03:54:52 2016 From: elvis.stansvik at orexplore.com (Elvis Stansvik) Date: Tue, 20 Dec 2016 09:54:52 +0100 Subject: [Distutils] nspkg.pth files break $PYTHONPATH overrides Message-ID: Hi all, Sorry to bump this 2 year+ old thread. This recently caused some serious head scratching. Barry's excellent analysis of what happens seems spot on on to me. Does anyone know if this issue was ever "resolved" or turned into a bug report? I can see that the Debian packaging of the lazr.* packages that Barry used as example still has this in their debian/rules: override_dh_auto_install: dh_auto_install find debian/python*-lazr.* -name '*.pth' -delete I.e. they undo the "accidental" installation of nspkg.pth files that install_egg_info caused. I'm doing the same in our packages right now, but it feels like a band-aid solution. In my case it's a bunch of company internal packages that all install under an "orexplore" namespace package. I'm actually distributing with a __init__.py for the namespace package and using namespace_packages= in setup.py, so it's not PEP 420 from the get go. It only becomes a PEP 420 when the debhelper has its way with it (stripping the __init__.py, but problematically also installing the nspkg.pth). Many thanks in advance. Elvis On Mar 24, 2014, at 5:48 PM, Barry Warsaw wrote: > Apologies for cross-posting, but this intersects setuptools and the import > system, and I wanted to be sure it reached the right audience. > > A colleague asked me why a seemingly innocent and common use case for > developing local versions of system installed packages wasn't working, and I > was quite perplexed. As I dug into the problem, more questions than answers > came up. I finally (think! I) figured out what is happening, but not so much > as to why, or what can/should be done about it. > > This person had a local checkout of a package's source, where the package was > also installed into the system Python. He wanted to be able to set > $PYTHONPATH so that the local package wins when he tries to import it. E.g.: > > % PYTHONPATH=`pwd`/src python3 > > but this didn't work because despite the setting of PYTHONPATH, the system > version of the package was always found first. The package in question is > lazr.uri, although other packages with similar layouts will also suffer the > same problem, which prevents an easy local development of a newer version of > the package, aside from being a complete head-scratcher. > > The lazr.uri package is intended to be a submodule of the lazr namespace > package. As such, the lazr/__init__.py has the old style way of declaring a > namespace package: > > try: > import pkg_resources > pkg_resources.declare_namespace(__name__) > except ImportError: > import pkgutil > __path__ = pkgutil.extend_path(__path__, __name__) > > and its setup.py declares a namespace package: > > setup( > name='lazr.uri', > version=__version__, > namespace_packages=['lazr'], > ... > > One of the things that the Debian "helper" program does when it builds a > package for the archive is call `$python setup.py install_egg_info`. It's > this command that breaks $PYTHONPATH overriding. > > install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file, > in which it finds the string 'lazr', and it proceeds to write a > lazr-uri-1.0.3-py3.4-nspkg.pth file. This causes other strange and unexpected > things to happen: > > % python3 > Python 3.4.0 (default, Mar 22 2014, 22:51:25) > [GCC 4.8.2] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> sys.modules['lazr'] > >>>> sys.modules['lazr'].__path__ > ['/usr/lib/python3/dist-packages/lazr'] > > It's completely weird that sys.modules would contain a key for 'lazr' when > that package was never explicitly imported. Even stranger, because a fake > module object is stuffed into sys.modules via the .pth file, tracing imports > with -v gives you no clue as to what's happening. And while > sys.modules['lazr'] has an __path__, it has no other attributes. > > I really don't understand what the purpose of the nspkg.pth file is, > especially for Python 3 namespace packages. > > Here's what the nspkg.pth file contains: > > import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) > > The __path__ value is important here because even though you've never > explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the > existing lazr module object's __path__ takes over, and thus the system > lazr.uri package is found even though both lazr/ and lazr/uri/ should have > been found earlier on sys.path (yes, sys.path looks exactly as expected). > > So the presence of the nspkg.pth file breaks $PYTHONPATH overriding. That > seems bad. ;) > > If you delete the nspkg.path file, then things work as expected, but even this > is a little misleading! > > I think the Debian helper is running install_egg_info as a way to determine > what namespace packages are defined, so that it can actually *remove* the > parent's __init__.py file and use PEP 420 style namespace packages. In fact, > in the Debian python3-lazr.uri binary package, you find no system > lazr/__init__.py file. This is why removing the nspkg.pth file works. > > So I thought, why not conditionally define setup(..., namespace_packages) only > for Python 2? This doesn't work because the Debian helper will see that no > namespace packages are defined, and thus it will leave the original > lazr/__init__.py file in place. This then breaks $PYTHONPATH overriding too > because of __path__ extension of the pre-PEP 420 code only *appends* the local > development path. IOW, the system import path is the first element of a > 2-element list on lazr.__path__. While the local import path is the second > element, in this case too the local import fails. > > It seems like what you want for Python 3 (and we're talking >= 3.2 here) is > for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and > let PEP 420 do it's thing. In fact if you set things up this way, $PYTHONPATH > overriding works exactly as expected. > > Because I don't know why install_egg_info is installing the nspkg.pth file, I > don't know which component needs to be changed: > > * Change setuptools install_egg_info command to not install an nspkg.pth file > even for namespace_package declare packages, at least under Python 3. > This behavior seems pretty nasty all by itself because it magically and > untraceably installs stripped down module objects in sys.modules when > Python first scans the import path. > > * Change the Debian helper to remove the nspkg.pth file, or not call > install_egg_info *and* continue to remove /__init__.py in Python 3 > so as to take advantage of PEP 420. It's nice to know that PEP 420 > actually represents something sane. :) > > For added bonus, we have this additional oddity: > > % PYTHONPATH=`pwd`/src python3 > Python 3.4.0 (default, Mar 22 2014, 22:51:25) > [GCC 4.8.2] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> sys.modules['lazr'] > >>>> sys.modules['lazr'].__path__ > ['/usr/lib/python3/dist-packages/lazr'] >>>> import lazr.uri >>>> lazr.uri.__file__ > '/usr/lib/python3/dist-packages/lazr/uri/__init__.py' >>>> sys.modules['lazr'] > >>>> sys.modules['lazr'].__path__ > ['/home/barry/projects/ubuntu/lazruri/trusty/src/lazr', '/usr/lib/python3/dist-packages/lazr'] > > > Notice how importing lazr.uri *replaces* sys.modules['lazr'] with the local > development one, even though it still imports lazr.uri from the system path. > I'm not exactly sure how this happens, but I've traced that to > _LoaderBasics.exec_module()'s call of _call_with_frames_removed(), which > exec's lazr.uri's code object into that module's __dict__. Nothing in > lazr/uri/__init__.py should be doing that, afaict from both visual inspection > of the code and disassembling the compiled code object. > > Hopefully I've explained the situation correctly and lucidly. Below I'll > describe how to set up a reproducible environment on a Debian machine. > Thoughts and comments are welcome! > > Cheers, > -Barry > > % sudo apt-get install python3-lazr.uri > % cd tmp > % bzr branch lp:lazr.uri trunk > % cd trunk > % PYTHONPATH=`pwd`/src python3 > (Then try things at the Python prompt from above.) > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From prometheus235 at gmail.com Wed Dec 21 15:29:42 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Wed, 21 Dec 2016 14:29:42 -0600 Subject: [Distutils] Can't upload sdist: "File already exists" Message-ID: I have a little package "huffman" where I build an sdist and wheel (python setup.py sdist bdist_wheel) and both seem to get built and can install fine. I can't seem to upload both to PyPI because the "File already exists": $ twine upload dist/* Uploading distributions to https://upload.pypi.org/legacy/ Uploading huffman-0.1.2-py2.py3-none-any.whl Uploading huffman-0.1.2.tar.gz HTTPError: 400 Client Error: File already exists. for url: https://upload.pypi.org/legacy/ Subsequent call to upload *just* the tarball fails the same way. I can't see an sdist anywhere, and uploading it via the website or twine just tells me it's already there...somehow. Asking pip to try to give it to me fails though (the binary works, however): $ pip download --no-cache-dir --only-binary :all: huffman==0.1.2 Collecting huffman==0.1.2 Downloading huffman-0.1.2-py2.py3-none-any.whl Saved ./huffman-0.1.2-py2.py3-none-any.whl Successfully downloaded huffman $ pip download --no-cache-dir --no-binary :all: huffman==0.1.2 Collecting huffman==0.1.2 Could not find a version that satisfies the requirement huffman==0.1.2 (from versions: ) No matching distribution found for huffman==0.1.2 Am I missing something? I am as sure as I can be that I didn't upload it twice; I bumped my version up one because I figured that may have been it. "twine register" that some guides mention just gets shot down with an HTTP "410 [...] simply upload the file" Cheers, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Dec 21 17:54:11 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Dec 2016 14:54:11 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: On Thu, Dec 15, 2016 at 8:29 PM, Glyph Lefkowitz wrote: > At the beginning of your story you mentioned the GUI client - *that* is > the missing piece ;). I've been saying for years that we need a Python.app > that lets you easily bootstrap all this stuff: walk you through installing > C dev tools if your packages need them, present a GUI search interface to > finding packages, present a normal "file->open" dialog for selecting a > location for a new virtualenv, automatically pop open a terminal, launching > a Jupyter notebook whose kernel is pointed at said environment... > Anaconda provides something like this -- personally, I;m a command lien geek, so have no idea how much or if it's any good. But might be worth a look if you're interested. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Dec 21 18:08:34 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 21 Dec 2016 15:08:34 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: On Fri, Dec 16, 2016 at 5:51 AM, Daniel Holth wrote: > One possibility to consider is that virtualenv itself is a bad idea. Why > should the Python interpreter executable, rather than the program being > run, determine the set of packages that is available for import? > well, way back when, som eof us suggestted that pyton have pacakge version mangement built in to import: import this_package>=2.1 or whatever. At that time, the pyGTK and wxPython projects had done a role-your-own version of this. wxPython's was: import wxversion wxversion.select('2.3') import wx kind a kludgy, but it worked. However, Guido, among others was pretty adamant that this was NOT python's responsibility. Then, along came setuptools that kinda-sorta provided something like that, and then virtualenv -- and the rest is history. I now use conda, which provides environments that manage python itself, other C libs, etc, and it works pretty well. And there are utilities that let you run a script in a given environment: https://github.com/pelson/conda-execute (and maybe others) So that does kinda pass the responsibility to the app. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Dec 22 10:49:38 2016 From: brett at python.org (Brett Cannon) Date: Thu, 22 Dec 2016 15:49:38 +0000 Subject: [Distutils] Can't upload sdist: "File already exists" In-Reply-To: References: Message-ID: Because you already uploaded a wheel for version 0.1.2 you can't upload any other files for that version, else people could accidentally upload e.g. an sdist with code different from what was already uploaded in the wheel. If you want an sdist then I would do another release as version 0.1.2post1 with the wheel and sdist (or whatever the proper post release version format is; on my phone so a pain to look up right now). On Wed, Dec 21, 2016, 12:30 Nick Timkovich, wrote: > I have a little package "huffman" where I build an sdist and wheel (python > setup.py sdist bdist_wheel) and both seem to get built and can install > fine. I can't seem to upload both to PyPI because the "File already exists": > > $ twine upload dist/* > Uploading distributions to https://upload.pypi.org/legacy/ > Uploading huffman-0.1.2-py2.py3-none-any.whl > Uploading huffman-0.1.2.tar.gz > HTTPError: 400 Client Error: File already exists. for url: > https://upload.pypi.org/legacy/ > > Subsequent call to upload *just* the tarball fails the same way. I can't > see an sdist anywhere, and uploading it via the website or twine just tells > me it's already there...somehow. Asking pip to try to give it to me fails > though (the binary works, however): > > $ pip download --no-cache-dir --only-binary :all: huffman==0.1.2 > Collecting huffman==0.1.2 > Downloading huffman-0.1.2-py2.py3-none-any.whl > Saved ./huffman-0.1.2-py2.py3-none-any.whl > Successfully downloaded huffman > > > $ pip download --no-cache-dir --no-binary :all: huffman==0.1.2 > Collecting huffman==0.1.2 > Could not find a version that satisfies the requirement huffman==0.1.2 > (from versions: ) > No matching distribution found for huffman==0.1.2 > > Am I missing something? I am as sure as I can be that I didn't upload it > twice; I bumped my version up one because I figured that may have been it. > "twine register" that some guides mention just gets shot down with an HTTP > "410 [...] simply upload the file" > > Cheers, > Nick > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Thu Dec 22 11:04:55 2016 From: prometheus235 at gmail.com (Nick Timkovich) Date: Thu, 22 Dec 2016 11:04:55 -0500 Subject: [Distutils] Can't upload sdist: "File already exists" In-Reply-To: References: Message-ID: So upload the sdist first in the future? What's the difference between uploading an sdist second with code that differs from what's in the wheel and vice versa, is PyPI doing some checking? Generally, is it OK (recommended/socially acceptable) to upload universal wheels only or is the sdist still nice to have available? I guess maybe for pip, but it considering universal, pure-Python wheels as "binaries" seems quirky. On Thu, Dec 22, 2016 at 10:49 AM, Brett Cannon wrote: > Because you already uploaded a wheel for version 0.1.2 you can't upload > any other files for that version, else people could accidentally upload > e.g. an sdist with code different from what was already uploaded in the > wheel. If you want an sdist then I would do another release as version > 0.1.2post1 with the wheel and sdist (or whatever the proper post release > version format is; on my phone so a pain to look up right now). > > On Wed, Dec 21, 2016, 12:30 Nick Timkovich, > wrote: > >> I have a little package "huffman" where I build an sdist and wheel >> (python setup.py sdist bdist_wheel) and both seem to get built and can >> install fine. I can't seem to upload both to PyPI because the "File already >> exists": >> >> $ twine upload dist/* >> Uploading distributions to https://upload.pypi.org/legacy/ >> Uploading huffman-0.1.2-py2.py3-none-any.whl >> Uploading huffman-0.1.2.tar.gz >> HTTPError: 400 Client Error: File already exists. for url: >> https://upload.pypi.org/legacy/ >> >> Subsequent call to upload *just* the tarball fails the same way. I can't >> see an sdist anywhere, and uploading it via the website or twine just tells >> me it's already there...somehow. Asking pip to try to give it to me fails >> though (the binary works, however): >> >> $ pip download --no-cache-dir --only-binary :all: huffman==0.1.2 >> Collecting huffman==0.1.2 >> Downloading huffman-0.1.2-py2.py3-none-any.whl >> Saved ./huffman-0.1.2-py2.py3-none-any.whl >> Successfully downloaded huffman >> >> >> $ pip download --no-cache-dir --no-binary :all: huffman==0.1.2 >> Collecting huffman==0.1.2 >> Could not find a version that satisfies the requirement huffman==0.1.2 >> (from versions: ) >> No matching distribution found for huffman==0.1.2 >> >> Am I missing something? I am as sure as I can be that I didn't upload it >> twice; I bumped my version up one because I figured that may have been it. >> "twine register" that some guides mention just gets shot down with an HTTP >> "410 [...] simply upload the file" >> >> Cheers, >> Nick >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Thu Dec 22 11:54:54 2016 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 22 Dec 2016 10:54:54 -0600 Subject: [Distutils] Can't upload sdist: "File already exists" In-Reply-To: References: Message-ID: On Thu, Dec 22, 2016 at 9:49 AM, Brett Cannon wrote: > Because you already uploaded a wheel for version 0.1.2 you can't upload any > other files for that version, else people could accidentally upload e.g. an > sdist with code different from what was already uploaded in the wheel. If > you want an sdist then I would do another release as version 0.1.2post1 with > the wheel and sdist (or whatever the proper post release version format is; > on my phone so a pain to look up right now). I'm pretty sure that's not correct. Twine is written to specifically upload the wheel first because PyPI will extract metadata from that and display it on the page. It won't do that if the sdist is uploaded first. I'm not able to reproduce the behaviour Nick is seeing. My only guess is that something changed in Warehouse or the file existed, was deleted, and is now being re-uploaded with the same version. That's not something Warehouse or PyPI allows anymore (republishing with the same version) From glyph at twistedmatrix.com Thu Dec 22 15:32:43 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 22 Dec 2016 12:32:43 -0800 Subject: [Distutils] Can't upload sdist: "File already exists" In-Reply-To: References: Message-ID: <7238F292-D507-4C2B-ACA6-8C93B1ACB7A2@twistedmatrix.com> > On Dec 22, 2016, at 8:54 AM, Ian Cordasco wrote: > > On Thu, Dec 22, 2016 at 9:49 AM, Brett Cannon wrote: >> Because you already uploaded a wheel for version 0.1.2 you can't upload any >> other files for that version, else people could accidentally upload e.g. an >> sdist with code different from what was already uploaded in the wheel. If >> you want an sdist then I would do another release as version 0.1.2post1 with >> the wheel and sdist (or whatever the proper post release version format is; >> on my phone so a pain to look up right now). > > I'm pretty sure that's not correct. Twine is written to specifically > upload the wheel first because PyPI will extract metadata from that > and display it on the page. It won't do that if the sdist is uploaded > first. > > I'm not able to reproduce the behaviour Nick is seeing. My only guess > is that something changed in Warehouse or the file existed, was > deleted, and is now being re-uploaded with the same version. That's > not something Warehouse or PyPI allows anymore (republishing with the > same version) Similarly; I've *definitely* done sdist releases after wheel releases of the same version in the past. -glyph From donald at stufft.io Thu Dec 22 15:41:41 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 22 Dec 2016 15:41:41 -0500 Subject: [Distutils] Can't upload sdist: "File already exists" In-Reply-To: <7238F292-D507-4C2B-ACA6-8C93B1ACB7A2@twistedmatrix.com> References: <7238F292-D507-4C2B-ACA6-8C93B1ACB7A2@twistedmatrix.com> Message-ID: <6D5351B4-3122-4748-8322-43FE15099888@stufft.io> There is a fairly new restriction that you can only have *one* sdist per release now. That should not apply at all to Wheels and if it is, then it is a bug. I can?t reproduce this issue though, so I?m going to guess there was some bit of confusion about exact errors here. If someone actually cannot upload 1 sdist + N wheels, please leave the state as is and tell me. > On Dec 22, 2016, at 3:32 PM, Glyph Lefkowitz wrote: > >> >> On Dec 22, 2016, at 8:54 AM, Ian Cordasco wrote: >> >> On Thu, Dec 22, 2016 at 9:49 AM, Brett Cannon wrote: >>> Because you already uploaded a wheel for version 0.1.2 you can't upload any >>> other files for that version, else people could accidentally upload e.g. an >>> sdist with code different from what was already uploaded in the wheel. If >>> you want an sdist then I would do another release as version 0.1.2post1 with >>> the wheel and sdist (or whatever the proper post release version format is; >>> on my phone so a pain to look up right now). >> >> I'm pretty sure that's not correct. Twine is written to specifically >> upload the wheel first because PyPI will extract metadata from that >> and display it on the page. It won't do that if the sdist is uploaded >> first. >> >> I'm not able to reproduce the behaviour Nick is seeing. My only guess >> is that something changed in Warehouse or the file existed, was >> deleted, and is now being re-uploaded with the same version. That's >> not something Warehouse or PyPI allows anymore (republishing with the >> same version) > > Similarly; I've *definitely* done sdist releases after wheel releases of the same version in the past. > > -glyph > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 23 02:15:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Dec 2016 17:15:34 +1000 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: On 22 December 2016 at 09:08, Chris Barker wrote: > And there are utilities that let you run a script in a given environment: > > https://github.com/pelson/conda-execute > > (and maybe others) > https://pypi.python.org/pypi/pipsi (pip Script Installer) creates a dedicated venv for the module and its dependencies, and then adds symlinks from ~/.local/bin to any scripts installed into the venv's bin directory. As Armin notes in the README, it's a really nice way to handle utilities that happen to be written in Python and published via PyPI, without having them impact any other aspect of your system. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Dec 23 16:23:01 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 23 Dec 2016 13:23:01 -0800 Subject: [Distutils] Maintaining a curated set of Python packages In-Reply-To: References: <20161208161024.682f26dc@presto> <1D7B1CC3-CA85-4A84-B049-1EA375A25C7B@stufft.io> <9CA02846-360B-4D2F-B089-E353069A1978@twistedmatrix.com> <4E6AADD2-2B2A-4B5F-A770-4CEC5F97FD7F@twistedmatrix.com> Message-ID: <55C689A3-8F2D-42E0-B628-6A50E33CF34A@twistedmatrix.com> > On Dec 22, 2016, at 11:15 PM, Nick Coghlan wrote: > > On 22 December 2016 at 09:08, Chris Barker > wrote: > And there are utilities that let you run a script in a given environment: > > https://github.com/pelson/conda-execute > > (and maybe others) > > https://pypi.python.org/pypi/pipsi (pip Script Installer) creates a dedicated venv for the module and its dependencies, and then adds symlinks from ~/.local/bin to any scripts installed into the venv's bin directory. As Armin notes in the README, it's a really nice way to handle utilities that happen to be written in Python and published via PyPI, without having them impact any other aspect of your system. I just wanted to echo that this is a great tool, and it teaches really good habits (i.e. don't install your general-purpose python tools into project-specific virtual environments). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Dec 25 19:36:52 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 26 Dec 2016 11:36:52 +1100 Subject: [Distutils] Can't upload sdist: "File already exists" References: Message-ID: <8537hby197.fsf@benfinney.id.au> Nick Timkovich writes: > Generally, is it OK (recommended/socially acceptable) to upload > universal wheels only or is the sdist still nice to have available? I > guess maybe for pip, but it considering universal, pure-Python wheels > as "binaries" seems quirky. None the less, the Wheel format *is* a format intended for installation (so a ?binary? format in that terminology). A wheel is not suitable for making changes and re-building from source. So the source distribution still needs to be uploaded as well, to support that important freedom for the recipient. -- \ ?Liberty, n. One of imagination's most precious possessions.? | `\ ?Ambrose Bierce, _The Devil's Dictionary_, 1906 | _o__) | Ben Finney