[Distutils] Immutable Files on PyPI

[M.-A. Lemburg]

Thanks for the confirmation that my interpretation was wrong. This
makes things a lot better :-)

More below...

On 29.09.2014 11:39, Nick Coghlan wrote:
> On 29 Sep 2014 18:49, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>
>> You are missing out on cases, where the release process causes files to
>> be omitted, human errors where packagers forget to apply changes to
>> e.g. documentation files, version files, change logs, etc., where
>> packagers want to add information that doesn't affect the software
>> itself, but meta information included in the distribution files.
>>
>> Such changes often do not affect the software itself, and so are not
>> detected by software tests.
> 
> Fixing such packaging errors is the primary intended use case of the "post"
> field in PEP 440. Alternatively, many projects will just spin a new release
> that addresses those issues, just as they would for any other bug.
> 
> Both of those approaches have the advantage of letting users (especially
> those operating mirrors, or downloading tarballs and feeding them into a
> separate redistribution system) easily tell whether or not they have the
> fixed version.

I don't see how that would help. AFAIU, PyPI would regard a "3.1.4.post1"
as new release and so invalidate all other already uploaded distribution
files rather than just regard the fixed upload as update
to the "3.1.4" release.

If we could get a widely adopted notion of build numbers into the
tools that would help a lot.

Installers and PyPI would then regard "3.1.4-1" as belonging to
release "3.1.4", but being a more current build as a distribution
file carrying "3.1.4" in its file name.

This would also have to work for all files uploaded for a release,
not only for some distribution formats, of course, including source
release files, Windows installers, egg files, etc.

I'd have to run some experiments, but don't believe that setuptools
and associated tools support this at the moment.

>> If I understand you correctly, you are essentially suggesting that it
>> becomes impossible to ever delete anything uploaded to PyPI, i.e.
>> turning PyPI into a WORM.
> 
> No, deletion remains supported. The only capability being removed is silent
> substitution of hosted files with different ones bearing the same name.
> 
> So if, for example, release "3.1.4" had a packaging error, then deleting it
> would still be possible, but the replacement would need to be something
> like "3.1.4.post1" or "3.1.5", rather than being permitted to reuse the
> "3.1.4" name.

So just to summarize: the proposal is to turn PyPI into a WORM,
but at least it's still possible to remove distribution files.

> The external hosting support is then the mechanism by which authors can
> retain complete and total control over their release process. That approach
> avoids all licensing concerns (including those related to US export
> controls), as well as ensuring they have the ability to silently change the
> contents of previously released files if they choose to do so (although, as
> noted above, actually doing so may break installation for anyone installing
> with peep, which checks file hashes based on the first version downloaded).

You're regularly bringing up this argument.

Let's just be fair here: external hosting of packages has been made so
user unfriendly in recent pip releases, that this has pretty much
become a non-option for anyone who wants to create a user friendly
package installation environment.

pip is unfortunately using the same kind of
--no-one-will-want-to-use-this-option-because-its-too-long
approach as setuptools/easy_install has done in the past to force
people into installing packages as eggs rather than installing
the packages in the standard write to site-packages dir way.

The end result is the same: users will not want to go
through those extra hoops and thus packages not hosted on PyPI
itself will be regarded as broken (because they don't install using
the standard method; not because they are really broken).

This is what I'm trying to address in discussions like these all
along. PyPI has a responsibility not only for the part consuming
part of the Python community, but also for the creating part of it.

While PyPI is great for indexing packages, it's not necessarily
the best answer for hosting the distribution files and I believe
we should open up some more to allow for making it possible to
offer the same kind of user experience while not making pypi.python.org
the only source of distribution files.

In the Linux world, this already works great by having multiple
repos which you can switch on/off easily.

For eGenix, we've found a way to work around the PyPI constraints:

http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/

which addresses our user's problems. Still, we'd much rather use
standard ways of working *with* PyPI rather than work around it.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2014-09-30: Python Meeting Duesseldorf ...                         today

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/