[Distutils] Maintaining a curated set of Python packages

[Wes Turner]

On Thursday, December 8, 2016, Wes Turner <wes.turner at gmail.com> wrote:

>
>
> On Thursday, December 1, 2016, Freddy Rietdijk <freddyrietdijk at fridh.nl
> <javascript:_e(%7B%7D,'cvml','freddyrietdijk at fridh.nl');>> wrote:
>
>> Hi,
>>
>> I would like to propose that, as a community, we jointly maintain a
>> curated set of Python packages that are known to work together. These
>> packages would receive security updates for some time and every couple of
>> months a new major release of the curated set comes available. The idea of
>> this is inspired by Haskell LTS, so maybe we should call this PyPI LTS?
>>
>> So why a PyPI LTS?
>>
>> PyPI makes available all versions of packages that were uploaded, and by
>> default installers like pip will try to use the latest available versions
>> of packages, unless told otherwise. With a requirements.txt file (or a
>> future pipfile.lock) and setup.py we can pin as much as we like our
>> requirements of respectively the environment and package requirements,
>> thereby making a more reproducible environment possible and also fixing the
>> API for developers. Pinning requirements is often a manual job, although
>> one could use pip freeze or other tools.
>>
>
> https://github.com/nvie/pip-tools :
>
> - requirements.in -> pip-compile -> requirements.txt (~pipfile.lock)
> - I can't remember whether pip-compile includes the checksum in the
> compiled requirements.txt
>
> https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
>
> - Current recommendation: sha256
> - (These are obviously different for platform-specific wheels and bdists)
>
> https://github.com/rbanffy/pip-chill
>
> - Unlike pip freeze, pip-chill includes only top-level deps
>
>
>
>>
>>
>>
>> A common problem is when two packages in a certain environment require
>> different versions of a package. Having a curated set of packages,
>> developers could be encouraged to test against the latest stable and
>> nightly of the curated package set, thereby increasing compatibility
>> between different packages, something I think we all want.
>>
>> Having a compatible set of packages is not only interesting for
>> developers, but also for downstream distributions. All distributions try to
>> find a set of packages that are working together and release them. This is
>> a lot of work, and I think it would be in everyone's benefit if we try to
>> solve this issue together.
>>
>
> I think conda has already been mentioned.
>
> - environment.yml : http://conda.pydata.org/docs/using/envs.html#use-
> environment-from-file
>
> https://conda-forge.github.io
>
> - "A community led collection of recipes, build infrastructure and
> distributions for the conda package manager."
> - "AppVeyor, CircleCI and TravisCI"
>
>
>>
>> A possible solution
>>
>> Downstream, that is developers and distributions, will need a set of
>> packages that are known to work together. At minimum this would consist of,
>> per package, the name of the package and its version, but for
>> reproducibility I would propose adding the filename and hash as well.
>> Because there isn't any reliable method to extract the requirements of a
>> package, I propose also including `setup_requires`, install_requires`, and
>> `tests_require` explicitly. That way, distributions can automatically build
>> recipes for the packages (although non-Python dependencies would still have
>> to be resolved by the distribution).
>>
>> The package set would be released as lts-YYYY-MM-REVISION, and developers
>> can choose to track a specific revision, but would typically be asked to
>> track only lts-YYYY-MM which would resolve to the latest REVISION.
>>
>>
>> Because dependencies vary per Python language version, interpreter, and
>> operating system, we would have to have these sets for each combination and
>> therefore I propose having a source which evaluates to say a TOML/JSON file
>> per version/interpreter/OS.
>> How this source file should be written I don't know; while I think the
>> Nix expression language is an excellent choice for this, it is not possible
>> for everyone to use and therefore likely not an option.
>>
>
> YAML: environment.yml, meta.yaml
>
> - http://conda.pydata.org/docs/building/meta-yaml.html#
> requirements-section
> - http://conda.pydata.org/docs/building/meta-yaml.html#test-section
>
> Could/would there be a package with an integration test suite in tests/?
>
> Practically, a developer would want a subset of the given known-good-set
> (and then additional packages), so:
>
> - fork/copy requirements-YYYY-MM-REV--<OSNAMEVER>.txt
> - #comment out unused deps
> - add '-r addl-requirements.txt'
>
>
>>
>>
>> Open questions
>>
>> There are still plenty of open questions.
>>
>> - Who decides when a package is updated that would break dependents? This
>> is an issue all distributions face, so maybe we should involve them.
>>
>
> IDK if e.g. https://requires.io can post to a mailing list?
>
> - "Stop wasting your time by manually keeping track of changelogs.
> Requires.io keeps your python projects secure by monitoring their
> dependencies."
> - Source: https://github.com/requires
>   - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin
>
>
>
>
>> - How would this be integrated with pip / virtualenv / pipfile.lock /
>> requirements.txt / setup.py? See e.g. https://github.com/pypa/p
>> ipfile/issues/10#issuecomment-262229620
>>
>
> - https://tox.readthedocs.io/en/latest/
>   - tox.ini
> - http://doc.devpi.net/latest/
>   - http://doc.devpi.net/latest/quickstart-releaseprocess.
> html#devpi-test-testing-an-uploaded-package
>

- Such a known-good-set could be hosted as a package index w/ devpi

http://docs.openstack.org/developer/pbr/

- pbr does away with setup.py and install_requires in favor of just
requirements.txt


>
>
>
>>
>> References to Haskell LTS
>>
>> Here are several links to some interesting documents on how Haskell LTS
>> works.
>> - A blog post describing what Haskell LTS is: https://www.fpcomplete.com
>> /blog/2014/12/backporting-bug-fixes
>> - Rules regarding uploading and breaking packages: https://github.com/f
>> pco/stackage/blob/master/MAINTAINERS.md#adding-a-package
>> - The actual LTS files https://github.com/fpco/lts-haskell
>>
>
Haskell ... pandoc ... https://sites.google.com/site/pydatalog/


>
>>
>> What do you think of this proposal? Would you be interested in this as
>> developer, or packager?
>>
>
> Given time to work on this, I'd probably spend it on developing a new or
> existing (?) comprehensive integration test suite for an already-maintained
> package set:
>
> https://docs.continuum.io/anaconda/pkg-docs
>
> https://www.enthought.com/products/canopy/package-index/
>
> https://github.com/audreyr/cookiecutter-pypackage/blob/
> master/%7B%7Bcookiecutter.project_slug%7D%7D/tests/test_
> %7B%7Bcookiecutter.project_slug%7D%7D.py
>
>
>>
>> Freddy
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161208/5d2c8afa/attachment.html>