[Distutils] option #1 plus download_url scraping

[Donald Stufft]

On Jun 5, 2013, at 3:11 PM, Barry Warsaw <barry at python.org> wrote:

> On Jun 05, 2013, at 02:47 PM, Donald Stufft wrote:
> 
>> I'm really just trying to get a sense of your workflow to see if I can make
>> any changes to improve the process for it.
>> 
>> One of the big problems with download_url is that the data in setup.py is
>> used in (and influences the content of) the final dist file. This means that
>> inside of a setup.py you won't know what the hash of the final file is. So
>> it's difficult for a setup.py based workflow with external urls to provide
>> md5 sums for the files which means that pip and friends can't verify that no
>> body modified the download in transit.
> 
> Let me explain what I (used to) do, and I'll let you decide whether anything
> needs to change. ;)
> 
> When I've finally got my vcs into a releasable state, I'll generally do:
> 
> $ python setup.py sdist upload -s
> 
> As you know,  this will create the tarball and signature file in dist, and
> upload everything nicely to the Cheeseshop.  At this point, I go to my
> project's Launchpad page and push the big "I made a release" button.  This
> fiddles some state on my project page, and it allows me to upload files
> attached to that particular release.  The nice thing is that I can just upload
> the dist/*.tar.gz and dist/*.asc  to add the tarball and signature to the
> Launchpad download page.  E.g.
> 
> https://launchpad.net/flufl.enum
> 
> and
> 
> https://launchpad.net/flufl.enum/+download
> 
> The url is predictable (which is good because it also has to play nicely with
> Debian watch files), with option #3, I just added the index page to
> download_url and let clients scrape it.  You'll see that it contains links to
> the md5 checksum and the locally generated signature.
> 
> There must be some value to also allowing folks to download from Launchpad, as
> shown by the 1055 downloads of flufl.enum.  Where are the PyPI download stats?
> 
> -Barry
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

Ah ok! I understand what you're trying to do now, thanks :)

Right now download counts are disabled on PyPI due to some issues with the script that integrates them pegging the CPU, and then the CDN. But prior to that flufl.enum had 28196 downloads from PyPI (total across all versions).

So Launchpad doesn't provide the md5 sums in a way that the tools will be able to process them, however you actually got lucky in that both your download url, and the files themselves are available via verifiable SSL so they aren't insecure if someone is using pip 1.3+ (and maybe newer easy_install? not sure the state of SSL outside of pip).

I think the downloads you see are either people manually downloading it, or tools that don't prefer PyPI hosted urls that just happened to pick the launchpad url. I think for this the best option is to just continue uploading everything to PyPI and switch to #1 (which I think I saw you did). While launchpad is verifiable via SSL, and is unlikely to have bad uptime I don't think it provides any benefit for the folks installing your package so there's not much of a reason to keep it around on your /simple/ page.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130605/3b217640/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130605/3b217640/attachment.pgp>