[Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

[Donald Stufft]

On May 8, 2014, at 11:21 AM, R. David Murray <rdmurray at bitdance.com> wrote:

> On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft <donald at stufft.io> wrote:
>> Most users are not going to care up until the point where the external server
>> is unavailable, and then they care a whole lot. On the tin it sounds reasonable
>> to just download the external file if the server is up however we've done
>> that for a long time and the end result has been end user pain.
>> 
>> Now requiring someone to add a flag in order to download an externally hosted
>> file is also end user pain. The difference between the two pains is when they
>> happen. The requiring a flag pain happens at the point of decision, when you
>> decide to make your deployment depend on something hosted externally. The 
>> default to allow pain happens sometime in the future, when you may or may not
>> have any idea why suddenly your installs aren't working (and when you look,
>> PyPI is up so you're really very confused why this particular file doesn't
>> work). Even worse is the case when a project has some old files, but the newer
>> versions aren't hosted and suddenly people are getting very old releases which
>> is even more confusing to the end users.
> 
> Ah, I understand now.
> 
> Your perspective is as someone who is using pip for *deployment*.

Deployment, or any kind of situation where you want to have a reproducible
build. Generally via deployment yes.

> 
> I'm speaking from a python+plus+pip end-user perspective, which is going
> to be even more common now that it is part of the Python distribution.
> 
> I'm not sure how you reconcile these two worlds.  I would venture to
> suggest that if you are using it for deployment you really ought to
> be using a local package repository[*], not the global one; but, as
> someone observed, the sensible thing to do and what people actually
> do are often very different; and, since I haven't done this particular
> kind of deployment scenario in Python myself, there may be reasons
> I'm missing.

People simply don’t do this[1]. Especially in a world with things like Heroku
existing which makes it stupid simple to use pip to install from PyPI but
installing from your own server requires standing up some infrastructure.

> 
> However, your last mention of "end users" confuses me.  Why are "end
> users" getting old packages in a deployment situation?  Isn't it the
> developer/operations people (and the latter would, I assume, control
> the 'external packages' flag) who would be seeing that?  Maybe you mean
> something by deployment different from how I use the word?

Someone using pip, this may be a developer who is just trying to recreate
their production environment locally, this may be someone using chef/puppet
to automate installing via pip, this may be someone pushing to Heroku.

The old versions thing is more that it’s really confusing when you type
``pip install foo`` on a monday and get 2.0, and ``pip install foo`` on weds
and get 0.4.

> 
> --David
> 
> [*] I found it *such* a pain to do this for perl/cpan.  I have a
> project for a customer where I have to do this, and the hoops I had
> to jump through to get a reliable deployment (where packages wouldn't
> be unexpectedly upgraded under my feet) were nasty.  (This was several
> years ago and I haven't revisited it, so maybe things have gotten better,
> or I just missed something.)
> 
> I haven't had to do it for python yet, oddly enough, so I don't know
> how hard it is for Python.

For Python with pip you can use a requirements.txt file to create a set of
dependencies that are pinned to exact versions like:

foo==2.0
bar==2.3

And pip will (theoretically, our dep solving is real bad ATM) install exactly
those versions from your index server. Generally this means PyPI which
means the author can delete the version and upload a new file with the
same version number. However it’s also trivial to stand up your own
server. It can be as easy as pointing nginx/Apache at a static directory with
autoindex = True. (See: https://wheels.caremad.io/).

On top of that there is peep which adds a secure message digest on it to
make sure that the author/index didn’t swap things out on you, and there
is some discussion about how best to add that to pip itself.

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/3812d75d/attachment.sig>