[Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm

[M.-A. Lemburg]

On 12.03.2013 22:26, PJ Eby wrote:
> On Tue, Mar 12, 2013 at 3:59 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 12.03.2013 19:15, M.-A. Lemburg wrote:
>>> I've run into a weird issue with easy_install, that I'm trying to solve:
>>>
>>> If I place two files named
>>>
>>> egenix_mxodbc_connect_client-2.0.2-py2.6.egg
>>> egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip
>>>
>>> into the same directory and let easy_install running on Linux
>>> scan this, it considers the second file for Windows as best
>>> match.
>>>
>>> Is the algorithm used for determining the best match documented
>>> somewhere ?
>>>
>>> I've had a look at the implementation, but this left me rather
>>> clueless.
>>>
>>> I thought that setuptools would prefer the .egg file over
>>> the prebuilt .zip file - binary files being easier to install
>>> than "source" files.
>>
>> After some experiments, I found that the follow change
>> in filename (swapping platform and python version, in addition
>> to use '-' instead of '.) works:
>>
>> egenix-mxodbc-connect-client-2.0.2-py2.6-win32.prebuilt.zip
>>
>> OTOH, this one doesn't (notice the difference ?):
>>
>> egenix-mxodbc-connect-client-2.0.2.py2.6-win32.prebuilt.zip
>>
>> The logic behind all this looks rather fragile to me.
> 
> easy_install only guarantees sane version parsing for distribution
> files built using setuptools' naming algorithms.  If you use
> distutils, it can only make guesses, because the distutils does not
> have a completely unambiguous file naming scheme.  And if you are
> naming the files by hand, God help you.  ;-)

The problem appears to be a bug in setuptools' package_index.py.

The function interpret_distro_name() creates a set of possible
separations of the found name into project name and version.

It does find the right separation, but for some reason, the
code using that function does not check the found project
names against the project name the user is trying to install,
but simply takes the last entry of the list returned by the
above function.

As a result, easy_install downloads and tries to install
project files that don't match the project name in some
cases.

Here's another example where it fails (say you're on a x64 Linux box):

# easy_install egenix-pyopenssl

As example, say it finds these distribution files:

    'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-linux-x86_64-prebuilt.zip',
    'egenix_pyopenssl-0.13.1.1.0.1.5-py2.7-linux-x86_64.egg',
    'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-macosx-10.5-x86_64-prebuilt.zip',
    'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs4-macosx-10.5-x86_64-prebuilt.zip',

It then creates different interpretations of those names, puts
them in a list and sorts them. Here's the end of that list:

egenix-pyopenssl; 0.13.1.1.0.1.5 <<-- this would be the correct .egg file
egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-linux-x86-64-prebuilt
egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-macosx-10.5-x86-64-prebuilt
egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt
egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs2-macosx; 10.5-x86-64-prebuilt
egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx; 10.5-x86-64-prebuilt

It picks the last entry, which would be for a project called
"egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx" - not the one
the user searched.

I'm trying to find a way to get it to use the correct .egg file
The .egg files does have precedence over the other files, since
easy_install regards them as source files with lower precedence.

This is important, because the /simple/ index page will have links
not only to .egg files, but also to our prebuilt .zip files,
which use a source file compatible setup.py interface.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 14 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/