[Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

Mon Mar 11 12:32:25 CET 2013

On Mar 11, 2013, at 5:23 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:

> On 11.03.2013 09:18, Lennart Regebro wrote:
>> On Mon, Mar 11, 2013 at 9:06 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>>> But this isn't necessarily true, there is another solution: mirror your requirements locally.
>> 
>> I do that. This is not a solution, because your requirements yesterday
>> is not your requirements tomorrow.
>> 
>>> Is it even clear why numerous archives aren't hosted on PyPI?
>> 
>> No, the only one that has mentioned why is Marc-André, I think, whose
>> eGenix packages are distributed as binary packages for loads of
>> different platforms. It's unclear to me if all these binary packages
>> should be uploaded to PyPI, and it is also unclear to me why they
>> can't be, it seems to be mostly a case of it being too much work.
> 
> I've listed all the reasons in one of the previous emails:
> 
> http://mail.python.org/pipermail/catalog-sig/2013-March/005502.html
> 
> Others will likely have additional reasons, like e.g.
> 
> * the PyPI uploads not being compatible to their release process

I've offered to donate my free time to anyone whose release process actually dictates they can't upload to PyPI to move to one that is as close as possible to their current solution while also not requiring external hosting.

> 
> * not knowing that it's possible to host files on PyPI - after
>  all it's an *index*, not a repository :-)

I know your joking but if this is an actual limiting factor my next proposal will be to change the name >:].

> 
> * still believing that PyPI is an unreliable hosting provider
>  due the many downtimes and problems it had in the past - which
>  is no longer true today
> 
> * not wanting to host and maintain files in several different
>  places

Publish their own repository, put instructions on their PyPI page how to add it to a potential users deploy.

> 
> * not wanting to host release files at all, i.e. have people
>  check out the version from a repository instead of doing
>  the download, unzip, install dance

So don't? Include instructions. No one's proposal prevents people from listing projects on PyPI that aren't hosted there. It just means that if you don't want to host your things on PyPI you'll need to provide instructions for getting your files.

> 
> * not wanting to separate associated library or product
>  code from the Python wrapper code (think e.g. the
>  Python interface for subversion)

Same answer as before, either separate it or provide instructions on your PyPI index page.

> 
> * not being allowed to upload files to external servers
>  by company policy, or having to deal with a company
>  policy that makes this difficult/unattractive

Again with include instructions in the PyPI description.

> 
> * having issues with the added latency of PyPI downloads compared
>  to a simple file based index hosted on a company web server

This seems backwards. If they are upset with the latency why aren't they just installing directly from the index on the company web server? Why are they hitting PyPI at all?

> 
> * having a strong need to know the number of downloads per
>  package and associated statistics such as downloads per
>  country, per year/month/day/hour

Daily stats are published per filename. Doesn't include breakdowns per country though. I will fight for any statistic people actually want that doesn't expose sensitive information. (No IP addresses etc. Countries are fine etc.).

> 
> * not wanting to give up access to the download log files

That runs counter to privacy concerns. If this is an actual blocker then I suggest they run their own index again.

> 
> * having a requirement to restrict downloads on a per country
>  basis, e.g. for export controlled software or software which
>  may not be imported/used in certain countries

Don't host the files on PyPI, publish instructions for installing your software on PyPI.

> 
> * having PyPI not provide the technical means to host the
>  release files, e.g. due to the releases using a format
>  which is not supported by PyPI (e.g. all the ActiveState
>  packages - http://code.activestate.com/pypm/)

Open a discussion here about including your format, open a ticket tracker about including your format, submit a PR about including your format, host your own repository if it makes sense for your format (See active state again).

> 
> * user experience/support issues:
>  if the package has external dependencies,
>  or needs special setup, it may provide a better user experience
>  to host the Python wrapper on the same page as the dependencies
>  and instructions on how to install them; rather than having
>  them on PyPI which lets people believe that a simple
>  "pip install something" will get them a working setup

So don't host the files on PyPI, include your instructions on PyPI.

> 
> Those are just a few things that come to mind. I'm sure there
> are more issues that keep authors from uploading their
> packages to PyPI.
> 
> Overall, I think we should encourage people to make their
> code available through PyPI and make it attractive to them,
> but keep the possibility to continue using external hosting
> platforms, should they run into issues that PyPI cannot
> solve for them.

This is a nice thought, but it doesn't work in practice because of the "Weakest Link Theory". Basically you're only as strong as your weakest link. The weakest link is any external package.

> 
>> He also mentioned the big Python distributions eGenix does as being
>> too large for PyPI, but I don't really see the point of uploading
>> Python distributions to PyPI, they can't be installed with Python
>> installers anyway.
> 
> Not sure what you mean here.
> 
> PyPI is also used to index Python projects which are not Python
> packages to be installed by pip/easy_install/etc.

That's fine. No one's saying you can't list a package on PyPI that doesn't include files. Just the external links won't be available on /simple/.

> 
> Some of those may also want to
> 
>>> IMHO it would be better to remove barriers than force projects to host files on PyPI.
>> 
>> Nobody has really been able to point out any real barriers, so we
>> don't know what they are or if they exist.
> 
> Again, please see the email where I listed the ones affecting
> at least eGenix.
> 
> Most of those can be addressed in one way or another, e.g.
> by having PyPI cache the files, provide access to the download
> counts by country, provide a way to host separate indexes for
> UCS2/UCS4 egg files, etc.
> 
> The only issues that need more investigation are the PyPI license
> terms and the general issue of not being able to host export
> regulated files on PyPI.
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Mar 11 2013)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130311/cec5d687/attachment-0001.pgp>