[Distutils] easy_install installing beta version of psycopg2

[P.J. Eby]

At 12:01 AM 2/17/2011 +0000, Daniele Varrazzo wrote:
>On Wed, Feb 16, 2011 at 8:40 PM, P.J. Eby <pje at telecommunity.com> wrote:
> > At 05:47 PM 2/16/2011 +0000, Daniele Varrazzo wrote:
> >>
> >> Do I, as a packager, have the possibility to say "what I have specified
> >> on PyPI as stable release is exactly what I mean"?
> >
> > If you don't want easy_install to find it, don't list it on the pages
> > referred to in your "Home Page" or "Download URL" on PyPI.  Easy_install
> > reads those pages because many package authors do not provide directly
> > usable URLs or packages on PyPI.
>
>I'm sorry, it is obvious that I have not spent so much time into this
>problem as the designer of this feature. But it still don't get the
>rationale behind discarding available, non-ambiguous metadata in
>favour of screen scraping.

When easy_install was first written, PyPI didn't even support 
*uploading*.  And the quality of available metadata on PyPI is still 
quite sketchy -- many packages will have only one file uploaded for 
an outdated version, but still  have good downloads on their home 
pages or download URLs.

While I realize this can be an inconvenience for the people who DO 
have good metadata, the truth is that there are many packages for 
which "unstable" version numbers are in fact the *best* download 
choice for many users.  Without a uniform versioning system that 
*every* package adheres to (and PEP 386 isn't actually sufficient for 
this yet -- something more like "semantic versioning" is needed), 
there's no way to easily set a policy across projects for "how stable 
a version do you want to download?"

In general, the practice for most projects is to simply publish 
unstable, "don't download this unless you really mean it" versions 
via "development" links, such as links to svn or other 
repositories.  These links don't look like downloads to easy_install, 
except for the #egg=project-version tag that tells it how to 
interpret them, and saying '#egg=myproject-beta' suffices to ensure 
that only a specific installation request for 'myproject==beta' will 
follow the link.

(Unfortunately, this tag does *not* currently override easy_install's 
native interpretation of the link, so tacking '#egg=psycopg-beta' on 
the end of your download links won't stop them being detected as 
future versions.  See below for other workarounds.)


>I don't think it is fair to ask to stop listing the beta from the
>homepage and the download page of the 2-pages website of the module:
>how should I advertise there is a beta around and testing is welcome?

You can make a direct download link, or make the filename not one 
that easy_install will recognize.  For example, if you rename the 
downloads to "beta-psycopg-whatever.*", or use a URL that redirects, 
like /beta-download, that's then redirected by the web server to the 
appropriate file location.  easy_install only looks at links that 
*appear* to be distutils-or-setuptools-generated archives for the 
desired project.

Another alternative is to block easy_install from parsing your home 
page or download links, by simply providing those links inline in 
your PyPI project description, and *removing* the specific fields 
from the current release and all previous releases stored on 
PyPI.  Still another would be to block the 'setuptools' and 
'distribute' user agents from your website, returning 404s.


>the shortcomings of a package manager

Well, technically, this'd be a feature.  Granted, it's only a feature 
for users of projects whose maintainers are *not* keeping a 
well-groomed PyPI page.  ;-)  I guess it is a shortcoming in the 
sense that there ought to be a way to stop it from using this 
feature.  In retrospect, the #egg=proj-ver feature should override 
easy_install's native interpretation of a URL, rather than just 
adding to it, and I think I would be justified in considering this a 
bug worthy of fixing.

Unfortunately, even if I fixed that today, it wouldn't have ANY 
effect on 99% of the field installations of any Python package 
management tools: there are still people using 4 or 5 year old 
versions of easy_install, and a lot of people use Distribute (via 
their OS install), which is a year behind the setuptools trunk on 
various things.  Most other Python package management solutions are 
based on top of easy_install in one way or another, as well.

Pip is the main package manager that uses its own link-finding 
algorithm, but it only supports source installation 
AFAIK.  Distutils2 uses a link-finding algorithm that was lifted 
pretty much verbatim from easy_install, though I think there may have 
been some additions to it since I last looked at it.


> > You can prevent this as a package author, by specifying zip_safe=False in
> > your setup() script.
>
>psycopg2 is not zip_safe, as it contains a C extension.

C extensions don't make a package not-zip-safe - they just mean that 
if you install it zipped, there has to be an egg cache.  The cache is 
used to unzip the C extension.  If you want to force easy_install to 
unzip your package from the get-go, then specifying zip_safe=False 
explicitly is required.  (Of course, a user can still download the 
egg manually or override this on the command line, but that's their 
problem in that case.)