From a.badger at gmail.com  Sat Sep  1 00:26:33 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Fri, 31 Aug 2007 15:26:33 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070829164031.24C963A4072@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
Message-ID: <46D89599.4010707@gmail.com>

Phillip J. Eby wrote:
> At 01:14 AM 8/29/2007 -0400, Ignacio Vazquez-Abrams wrote:
>> I have some packages that I'm using the following with:
>>
>> %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT
>>
>> I now need to install multiple versions of one of the packages and so
>> need to disable the implicit --single-version-externally-managed. How
>> can I do so and still use setup.py install?
> 
> You can't.  You can use something like this, though, to make an RPM 
> that installs a non-active egg:
> 
>     setup.py easy_install --prefix=$RPM_BUILD_ROOT/usr -mx .
> 
Hmmm... So I've been experimenting a bit with this to make some new
guidelines for actually utilizing setuptools to make rpm packages [1]_
and found something a little strange.

I'm using sqlalchemy0.3 and sqlalchemy0.4 as my test case.  Install
sqlalchemy0.3 using::
  easy_install --prefix=$RPM_BUILD_ROOT/usr -m dist/SQLAlchemy-0.3*.egg

Then install sqlalchemy0.4 using::
  python setup.py install --root=$RPM_BUILD_ROOT%{python_sitelib}

This creates the following in site_packages::
  SQLAlchemy-0.3.10-py2.5.egg:
    EGG-INFO  sqlalchemy
  SQLAlchemy-0.4.0beta4-py2.5.egg-info:
    dependency_links.txt  entry_points.txt  PKG-INFO  SOURCES.txt
    top_level.txt
  sqlalchemy (0.4.0)

At this point I expect import sqlalchemy to import the 0.4 version and
pkg_resources to give me access to 0.3.10 or 0.4.0beta4 depending on how
I specify my versions.  But in fact, 0.4.0 is not available::

  pkg_require >=0.3,<0.4.0beta1...  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
in require
      needed = self.resolve(parse_requirements(requirements))
    File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
in resolve
      raise VersionConflict(dist,req) # XXX put more info here
  pkg_resources.VersionConflict: (SQLAlchemy 0.4.0beta4
(/usr/lib/python2.5/site-packages),
Requirement.parse('SQLAlchemy>=0.3,<0.4.0beta1'))

This also happens when I use *.pth files instead of installing 0.4 as an
active egg.

Is this a bug?  I can get my desired behaviour by installing both 0.3
and 0.4 as inactive eggs.  Then manually copying/symlinking the 0.4
version into site-packages.

I'll attach a small script I use to test this.

.. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-sql.sh
Type: application/x-shellscript
Size: 919 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070831/73e2d6f1/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 251 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070831/73e2d6f1/attachment.pgp 

From pje at telecommunity.com  Sat Sep  1 01:33:22 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 31 Aug 2007 19:33:22 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D89599.4010707@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
Message-ID: <20070831233159.B10803A40AE@sparrow.telecommunity.com>

At 03:26 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
>I'm using sqlalchemy0.3 and sqlalchemy0.4 as my test case.  Install
>sqlalchemy0.3 using::
>   easy_install --prefix=$RPM_BUILD_ROOT/usr -m dist/SQLAlchemy-0.3*.egg
>
>Then install sqlalchemy0.4 using::
>   python setup.py install --root=$RPM_BUILD_ROOT%{python_sitelib}
>
>This creates the following in site_packages::
>   SQLAlchemy-0.3.10-py2.5.egg:
>     EGG-INFO  sqlalchemy
>   SQLAlchemy-0.4.0beta4-py2.5.egg-info:
>     dependency_links.txt  entry_points.txt  PKG-INFO  SOURCES.txt
>     top_level.txt
>   sqlalchemy (0.4.0)
>
>At this point I expect import sqlalchemy to import the 0.4 version and
>pkg_resources to give me access to 0.3.10 or 0.4.0beta4 depending on how
>I specify my versions.  But in fact, 0.4.0 is not available::
>
>   pkg_require >=0.3,<0.4.0beta1...  Traceback (most recent call last):
>     File "<string>", line 1, in <module>
>     File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
>in require
>       needed = self.resolve(parse_requirements(requirements))
>     File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
>in resolve
>       raise VersionConflict(dist,req) # XXX put more info here
>   pkg_resources.VersionConflict: (SQLAlchemy 0.4.0beta4
>(/usr/lib/python2.5/site-packages),
>Requirement.parse('SQLAlchemy>=0.3,<0.4.0beta1'))
>
>This also happens when I use *.pth files instead of installing 0.4 as an
>active egg.

I'm not sure what you mean by using .pth files here.  If you did the 
easy_install without the '-m', that would make the 0.3 version the 
active (default) version, though, and it would override the 
default-active 0.4 version.


>Is this a bug?

Not really.  Once pkg_resources has been imported, it's too late to 
request an inactive version that conflicts with an active version of 
the same project, as the default working set has already been 
configured by that time.

However, if you create a project that depends on the 0.3 SQLAlchemy, 
and run a script from that project, it will see the correct 
version.  pkg_resources can resolve the version conflict 
automatically under those circumstances.

Anyway, the point here is that a default version is a default 
version, period.  The only way it can be overridden is by a 
conflicting requirement defined for a script.


From a.badger at gmail.com  Sat Sep  1 03:56:00 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Fri, 31 Aug 2007 18:56:00 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070831233159.B10803A40AE@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
Message-ID: <46D8C6B0.8060702@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 03:26 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
>>
>> At this point I expect import sqlalchemy to import the 0.4 version and
>> pkg_resources to give me access to 0.3.10 or 0.4.0beta4 depending on how
>> I specify my versions.  But in fact, 0.4.0 is not available::
>>
>>   pkg_require >=0.3,<0.4.0beta1...  Traceback (most recent call last):
>>     File "<string>", line 1, in <module>
>>     File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
>> in require
>>       needed = self.resolve(parse_requirements(requirements))
>>     File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
>> in resolve
>>       raise VersionConflict(dist,req) # XXX put more info here
>>   pkg_resources.VersionConflict: (SQLAlchemy 0.4.0beta4
>> (/usr/lib/python2.5/site-packages),
>> Requirement.parse('SQLAlchemy>=0.3,<0.4.0beta1'))
>>
>> This also happens when I use *.pth files instead of installing 0.4 as an
>> active egg.
> 
> I'm not sure what you mean by using .pth files here.  If you did the
> easy_install without the '-m', that would make the 0.3 version the
> active (default) version, though, and it would override the
> default-active 0.4 version.
> 
I tried manually creating a .pth file that lists one or the other of the
eggs.

> 
>> Is this a bug?
> 
> Not really.  Once pkg_resources has been imported, it's too late to
> request an inactive version that conflicts with an active version of the
> same project, as the default working set has already been configured by
> that time.
>
What determines if a version is active or inactive?

> However, if you create a project that depends on the 0.3 SQLAlchemy, and
> run a script from that project, it will see the correct version. 
> pkg_resources can resolve the version conflict automatically under those
> circumstances.
>
If I understand you correctly, this is exactly the situation that is
causing the error above.  I guess I should have been more explicit in
what was being invoked to cause that traceback.  It's the second test in
my test script:

python -c 'import pkg_resources;
pkg_resources.require("SQLAlchemy>=0.3,<0.4.0be
ta1"); import sqlalchemy; print sqlalchemy.__version__'


> Anyway, the point here is that a default version is a default version,
> period.  The only way it can be overridden is by a conflicting
> requirement defined for a script.
> 
1) Is the default version different from the active version?  If so how?
2) Is what I'm doing above with pkg_resources.require() what you mean by
specifying a conflicting requirement?
2b) If so, is it a bug that it doesn't work?

Thanks,
- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2MawX6yAic2E7kgRAuGgAJ4nEa7ZOk/hZ0a0m89tz8hKDiKH0wCeLCgd
rtXcPuxOT5HPrT7bmEHrQRc=
=Kxbb
-----END PGP SIGNATURE-----

From a.badger at gmail.com  Sat Sep  1 06:32:37 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Fri, 31 Aug 2007 21:32:37 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D8C6B0.8060702@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com>
Message-ID: <46D8EB65.8080406@gmail.com>

[Resending with gzipped complete filelist as the previous try was too
large to send to the list]

Just to illustrate what I'm trying to achieve.  I've updated the Fedora
Packaging Guidelines[1]_ to allow two versions of a package to coexist.
 I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and
the output of the test-sql.sh script using this procedure.  The end
result is what we want but the build step to get there seem a tad
fragile and kludgey.  Since these are going to become guidelines for all
of our python packages, I'd like to know if either: 1) there's a better
way to do this or 2) the results I'm achieving are not expected and
could disappear with a random upgrade of setuptools.

.. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs

Build
-----

sqlalchemy-0.3 compat package::
  CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py bdist_egg
  mkdir -p %{python_sitelib}
  easy_install -m --prefix $RPM_BUILD_ROOT%{_usr} dist/*.egg


sqlalchemy-0.4 default package::
  CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py bdist_egg
  mkdir -p %{python_sitelib}
  easy_install -m --prefix %{_usr} --always-unzip dist/*.egg
  cd %{python_sitelib}/%{srcname}-%{version}%{betaver}-py%{pyver}.egg
  mv sqlalchemy ..
  ln -s ../sqlalchemy .

The compat package is pretty straighforward.  However, building the
default package seems overly complex.  It seems like we should be able
to do this::
  CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build
  %{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT

But that yields the tracebacks when using pkg_resource.require() to try
to run 0.3.

truncated filelist
------------------
Full filelist attached.  These are the main toplevel directories to show
where the important pieces are.  The sqlalchemy directories all contain
a version of the python module.  (SQLAlchemy-0.4.egg/sqlalchemy is
actually a symlink to site-packages/sqlalchemy but that doesn't matter.
   Those can be reversed or they can be copies with the same results).

  site-packages/SQLAlchemy-0.3.10-py2.5.egg
  site-packages/SQLAlchemy-0.3.10-py2.5.egg/EGG-INFO
  site-packages/SQLAlchemy-0.3.10-py2.5.egg/sqlalchemy
  site-packages/SQLAlchemy-0.4.0beta4-py2.5.egg
  site-packages/SQLAlchemy-0.4.0beta4-py2.5.egg/EGG-INFO
  site-packages/SQLAlchemy-0.4.0beta4-py2.5.egg/sqlalchemy
  site-packages/sqlalchemy


test-sql.sh output
------------------
import sqlalchemy...  0.4.0beta4
pkg_require >=0.3,<0.4.0beta1...  0.3.10
pkg_require...  0.4.0beta4
pkg_require >=0.3...  0.4.0beta4
pkg_require <= 0.4.10...  0.4.0beta4
pkg_require <=0.3.12...  0.3.10

-Toshio

-------------- next part --------------
A non-text attachment was scrubbed...
Name: sqlalchemy.lst.gz
Type: application/x-gzip
Size: 2755 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070831/e568b0ec/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 251 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070831/e568b0ec/attachment.pgp 

From pje at telecommunity.com  Sat Sep  1 19:58:51 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 01 Sep 2007 13:58:51 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D8C6B0.8060702@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com>
Message-ID: <20070901175642.90FA23A40A0@sparrow.telecommunity.com>

At 06:56 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
>Phillip J. Eby wrote:
> > At 03:26 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
> >>
> >> At this point I expect import sqlalchemy to import the 0.4 version and
> >> pkg_resources to give me access to 0.3.10 or 0.4.0beta4 depending on how
> >> I specify my versions.  But in fact, 0.4.0 is not available::
> >>
> >>   pkg_require >=0.3,<0.4.0beta1...  Traceback (most recent call last):
> >>     File "<string>", line 1, in <module>
> >>     File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
> >> in require
> >>       needed = self.resolve(parse_requirements(requirements))
> >>     File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
> >> in resolve
> >>       raise VersionConflict(dist,req) # XXX put more info here
> >>   pkg_resources.VersionConflict: (SQLAlchemy 0.4.0beta4
> >> (/usr/lib/python2.5/site-packages),
> >> Requirement.parse('SQLAlchemy>=0.3,<0.4.0beta1'))
> >>
> >> This also happens when I use *.pth files instead of installing 0.4 as an
> >> active egg.
> >
> > I'm not sure what you mean by using .pth files here.  If you did the
> > easy_install without the '-m', that would make the 0.3 version the
> > active (default) version, though, and it would override the
> > default-active 0.4 version.
> >
>I tried manually creating a .pth file that lists one or the other of the
>eggs.

That won't work to make the egg the default version.  You have to 
have easy_install generate the .pth file, so that the necessary magic 
is included.

Normally, paths in .pth files are added to the *end* of sys.path, 
which means the single-version egg will take 
precedence.  easy_install adds special incantations to its .pth files 
so that the eggs it installs have higher precedence than everything else.


>What determines if a version is active or inactive?


Whether it's listed on sys.path.  Your single-version egg's location 
was site-packages, and thus it's always on sys.path.


> > However, if you create a project that depends on the 0.3 SQLAlchemy, and
> > run a script from that project, it will see the correct version.
> > pkg_resources can resolve the version conflict automatically under those
> > circumstances.
> >
>If I understand you correctly, this is exactly the situation that is
>causing the error above.  I guess I should have been more explicit in
>what was being invoked to cause that traceback.  It's the second test in
>my test script:
>
>python -c 'import pkg_resources;
>pkg_resources.require("SQLAlchemy>=0.3,<0.4.0be
>ta1"); import sqlalchemy; print sqlalchemy.__version__'

Please reread what I said above: "if you create a *project* ... and 
run a script *from that project*".  That has nothing to do with what 
you're trying here, where you neither have a project nor a script 
generated from that project.  (You don't even have a script!)

The script wrappers generated by easy_install have code that handles 
this conflict issue.  Your test does not.


> > Anyway, the point here is that a default version is a default version,
> > period.  The only way it can be overridden is by a conflicting
> > requirement defined for a script.
> >
>1) Is the default version different from the active version?  If so how?

A default version is the version that's active by default.  :)  If 
it's listed in a .pth *generated by easy_install*, it will be the 
default version.  Otherwise, if it's a single-version egg on 
sys.path, it will be the default version.  Otherwise, if it's listed 
in a handmade (non-magic) .pth file, it will be the default 
version.  If it's somehow otherwise put on sys.path, it will be the 
default version.  If none of the above apply, there is no default version.


>2) Is what I'm doing above with pkg_resources.require() what you mean by
>specifying a conflicting requirement?

No.


From pje at telecommunity.com  Sat Sep  1 20:07:57 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 01 Sep 2007 14:07:57 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D8E880.5070204@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
Message-ID: <20070901180530.881D03A4092@sparrow.telecommunity.com>

At 09:20 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
>Just to illustrate what I'm trying to achieve.  I've updated the Fedora
>Packaging Guidelines[1]_ to allow two versions of a package to coexist.
>  I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and
>the output of the test-sql.sh script using this procedure.  The end
>result is what we want but the build step to get there seem a tad
>fragile and kludgey.  Since these are going to become guidelines for all
>of our python packages, I'd like to know if either: 1) there's a better
>way to do this or 2) the results I'm achieving are not expected and
>could disappear with a random upgrade of setuptools.
>
>.. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs

Here's the thing: if you want multiple installed versions of a 
package, *and* you want there to be a default version, then you 
*have* to have easy_install (or zc.buildout, or some similar tool) 
generate the startup scripts for anything that wants to use a 
non-default version.

This is true irrespective of the formats of the versions involved, 
whether they're zipfiles, directories, single-version, or 
whatever.  It's just the nature of the beast, due to the fact that 
there is a global 'working_set' that lists the projects that are 
currently on sys.path, and it is initialized when pkg_resources is 
imported.  (And currently, there is no way to *remove* a distribution 
from the working set.)

Thus, if you want multiple versions *and* want to be able to select a 
version after pkg_resources has been imported, you *cannot* have a 
default version.  In other words, the egg must not be on sys.path 
when pkg_resources is imported.  Then, pkg_resources can locate the 
desired version and add it.

(By the way: "on sys.path" means, "is importable", not "is in a 
directory on sys.path".  An .egg file or directory in site-packages 
is not "on sys.path" unless its filename is actually listed in 
sys.path.  A single-version egg in site-packages, however, *is* on sys.path).


From a.badger at gmail.com  Sat Sep  1 21:29:54 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 12:29:54 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070901175642.90FA23A40A0@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com>
	<20070901175642.90FA23A40A0@sparrow.telecommunity.com>
Message-ID: <46D9BDB2.80307@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 06:56 PM 8/31/2007 -0700, Toshio Kuratomi wrote:

>> I tried manually creating a .pth file that lists one or the other of the
>> eggs.
> 
> That won't work to make the egg the default version.  You have to have
> easy_install generate the .pth file, so that the necessary magic is
> included.
> 
> Normally, paths in .pth files are added to the *end* of sys.path, which
> means the single-version egg will take precedence.  easy_install adds
> special incantations to its .pth files so that the eggs it installs have
> higher precedence than everything else.
>
Good to know.  It doesn't affect what I tested as I used a manually
created .pth file *instead* of using single-version-externally-managed.
 So the eggs were arranged as:
  SQLAlchemy-0.4[...].egg/sqlalchemy
  SQLAlchemy-0.3[...].egg/sqlalchemy
  sqlalchemy.pth (containing the path into one of the eggs)

I'd rather do without .pth's in the Guidelines though, as they seem to
duplicate what can already be achieved by installing one egg as
single-version.

[...]
The rest of this is miscommunication based on my using terms
incorrectly.  I'll reply to the other message with something meaningful
now that I understand:

active version -- egg on sys.path

inactive version -- egg cannot be found by python as it is not on sys.path

default version -- version of a module that comes first on sys.path and
therefore will be selected from a bare import

project -- setuptools managed project that uses requires.txt to manage
conflicting versions.

If I've still got those wrong, let me know :-)

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2b2xX6yAic2E7kgRAg4MAJ9114KrfVWHxBa3MdglZPMUqoOTJACgiDKm
7BG+jEMItLfGplZ2rffDoGQ=
=7IK3
-----END PGP SIGNATURE-----

From a.badger at gmail.com  Sat Sep  1 22:38:20 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 13:38:20 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070901180530.881D03A4092@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
Message-ID: <46D9CDBC.4030504@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Well, I found a bug.  I haven't coded a solution yet so I don't know
precisely what fixing it will do for the rest of our conversation yet::

    def insert_on(self, path, loc = None):
        """Insert self.location in path before its nearest parent
directory"""
        [...Place loc in path...]
        # p is the spot where we found or inserted loc; now remove
duplicates
        while 1:
            try:
                np = npath.index(nloc, p+1)
            except ValueError:
                break
            else:
                del npath[np], path[np]
                p = np  # ha!

This code creates an unordered list because you might have already had
to place something after one of the duplicate locations that this code
is removing.  We need to do a sort of the entire list after each add +
duplicate removal run.

I'm seeing an actual problem with this when starting with the following
sys.path:
 ['/usr/bin', '/usr/lib/python25.zip', '/usr/lib/python2.5',
'/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk',
'/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages',
'/usr/lib/python2.5/site-packages/Numeric',
'/usr/lib/python2.5/site-packages/PIL',
'/usr/lib/python2.5/site-packages/TestGears-0.2-py2.5.egg-info',
'/usr/lib/python2.5/site-packages/gst-0.10',
'/usr/lib/python2.5/site-packages/gtk-2.0',
'/usr/lib/python2.5/site-packages/pyinotify',
'/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode']

/usr/lib/python2.5/site-packages/CherryPy-2.2.1-py2.5.egg is being
inserted before '/usr/lib/python2.5/site-packages'.  Then another
/usr/lib/python2.5/site-packages is entering the method and being placed
after /usr/lib/python2.5... which places it before the CherryPy egg.

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2c27X6yAic2E7kgRAraTAJ99J0Ij0OCfxei0bLzC/4l062QEFQCfUKt6
Qe3sguQ07kXv5emazhLFBMA=
=7j/M
-----END PGP SIGNATURE-----

From a.badger at gmail.com  Sat Sep  1 23:09:30 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 14:09:30 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070901180530.881D03A4092@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
Message-ID: <46D9D50A.6050801@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 09:20 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
>> Just to illustrate what I'm trying to achieve.  I've updated the Fedora
>> Packaging Guidelines[1]_ to allow two versions of a package to coexist.
>>  I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and
>> the output of the test-sql.sh script using this procedure.  The end
>> result is what we want but the build step to get there seem a tad
>> fragile and kludgey.  Since these are going to become guidelines for all
>> of our python packages, I'd like to know if either: 1) there's a better
>> way to do this or 2) the results I'm achieving are not expected and
>> could disappear with a random upgrade of setuptools.
>>
>> .. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs
> 
> Here's the thing: if you want multiple installed versions of a package,
> *and* you want there to be a default version, then you *have* to have
> easy_install (or zc.buildout, or some similar tool) generate the startup
> scripts for anything that wants to use a non-default version.
> 
You'll have to explain why that is because all my experiments show that
to be false.  Installing one package via easy_install -m and another via
- --single-version-externally-managed gets us 75% of the way to this
working.  Using easy_install -m for both and then copying/symlinking
gets us 100% of the way.

> This is true irrespective of the formats of the versions involved,
> whether they're zipfiles, directories, single-version, or whatever. 
> It's just the nature of the beast, due to the fact that there is a
> global 'working_set' that lists the projects that are currently on
> sys.path, and it is initialized when pkg_resources is imported.  (And
> currently, there is no way to *remove* a distribution from the working
> set.)
> 
Since the working_set doesn't explicitly list eggs unless they are
specified in a project's requires.txt it seems like
pkg_resources.require() can override a module which is the default
because it is installed in site-packages.  In fact, I understood this to
be a feature of setuptools: allowing someone to override the vendor
installed packages in site-packages with your own eggs.

> Thus, if you want multiple versions *and* want to be able to select a
> version after pkg_resources has been imported, you *cannot* have a
> default version.  In other words, the egg must not be on sys.path when
> pkg_resources is imported.  Then, pkg_resources can locate the desired
> version and add it.
>
Not quite.  In the single-version case, the egg is on sys.path because
the module directory is in site-packages.  Therefore pkg_resources make
a different version importable by placing a different egg's path before
site-packages in sys.path.

The goal is to have all of these things work:
1) import MODULE should import whatever the vendor decided should be the
default.
2) requires.txt in a project's egginfo should work to select a specific
version from the installed eggs.
3) In a simple script (without requires.txt), pkg_resources.requires()
should work to select a specific version from the installed eggs.

I think the basic way to enable this is:
For 1) Either install a module that is not an egg into site-packages or
install an egg as single-version-externally-managed into site-packages.
 When the user does import MODULE, python looks in site-packages, finds
the MODULE, and imports it without touching any setuptools code.

For 2) All installed modules that we want to be selectable must be eggs.
 This needs to work with both single-version and multiple-version eggs
when determining the best version match then, if necessary, modify
sys.path to place the best match in front of the other eggs.

For 3) No eggs for this module can be on sys.path before the
pkg_resources.require() call.  This does not count modules brought in
via site-packages as those are going to be overridden when we place egg
paths before site-packages.

This seems to mostly work right now with the procedure I outlined in my
previous message.  I have to fix the bug found in my other message to
see if I can get it to work all the time (and perhaps eliminate the
kludginess in my original procedures.)

You seem to think there's something wrong with this so there's obviously
something you're seeing that I don't.  Can you give an example of where
this will fail?

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2dUKX6yAic2E7kgRAiMnAJ40guBFKrh0V3Yl+kzJqRlZ6V5TJgCgrA/j
fnHPHB9Q6533xSAzELAV6Dc=
=swaR
-----END PGP SIGNATURE-----

From a.badger at gmail.com  Sat Sep  1 23:54:05 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 14:54:05 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D9CDBC.4030504@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com>
Message-ID: <46D9DF7D.705@gmail.com>

Toshio Kuratomi wrote:
> 
> This code creates an unordered list because you might have already had
> to place something after one of the duplicate locations that this code
> is removing.  We need to do a sort of the entire list after each add +
> duplicate removal run.

Here's a quick and dirty patch.  It's correct but it's inefficient.  A
better sort algorithm would help some.  Refactoring code so we call a
sort method after we finish adding entries to the path would be even better.

I'll test how this affects packaging tonight.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: setuptools-pkg_resources-order.patch
Type: text/x-patch
Size: 1929 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070901/aca10257/attachment-0001.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 251 bytes
Desc: OpenPGP digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070901/aca10257/attachment-0001.pgp 

From pje at telecommunity.com  Sun Sep  2 03:01:35 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 01 Sep 2007 21:01:35 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D9D50A.6050801@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9D50A.6050801@gmail.com>
Message-ID: <20070902005913.8D6083A4092@sparrow.telecommunity.com>

At 02:09 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>Phillip J. Eby wrote:
> > At 09:20 PM 8/31/2007 -0700, Toshio Kuratomi wrote:
> >> Just to illustrate what I'm trying to achieve.  I've updated the Fedora
> >> Packaging Guidelines[1]_ to allow two versions of a package to coexist.
> >>  I'll list here the sqlalchemy-0.4 and -0.3 build steps, filelists, and
> >> the output of the test-sql.sh script using this procedure.  The end
> >> result is what we want but the build step to get there seem a tad
> >> fragile and kludgey.  Since these are going to become guidelines for all
> >> of our python packages, I'd like to know if either: 1) there's a better
> >> way to do this or 2) the results I'm achieving are not expected and
> >> could disappear with a random upgrade of setuptools.
> >>
> >> .. _[1]: http://fedoraproject.org/wiki/PackagingDrafts/PythonEggs
> >
> > Here's the thing: if you want multiple installed versions of a package,
> > *and* you want there to be a default version, then you *have* to have
> > easy_install (or zc.buildout, or some similar tool) generate the startup
> > scripts for anything that wants to use a non-default version.
> >
>You'll have to explain why that is because all my experiments show that
>to be false.  Installing one package via easy_install -m and another via
>- --single-version-externally-managed gets us 75% of the way to this
>working.  Using easy_install -m for both and then copying/symlinking
>gets us 100% of the way.

I don't understand what you're saying.  If you have multiple versions 
installed, and one of them is active by default (i.e., listed in a 
.pth, or a single-version in site-packages), then the only simple way 
to access the non-default version is via an easy_install-generated script.


>Since the working_set doesn't explicitly list eggs unless they are
>specified in a project's requires.txt it seems like
>pkg_resources.require() can override a module which is the default
>because it is installed in site-packages.  In fact, I understood this to
>be a feature of setuptools: allowing someone to override the vendor
>installed packages in site-packages with your own eggs.

It is -- but as I said, it only works for scripts generated by 
easy_install, if you want to import the non-default version.


> > Thus, if you want multiple versions *and* want to be able to select a
> > version after pkg_resources has been imported, you *cannot* have a
> > default version.  In other words, the egg must not be on sys.path when
> > pkg_resources is imported.  Then, pkg_resources can locate the desired
> > version and add it.
> >
>Not quite.  In the single-version case, the egg is on sys.path because
>the module directory is in site-packages.  Therefore pkg_resources make
>a different version importable by placing a different egg's path before
>site-packages in sys.path.

Again, that's *only* if you don't have a conflicting egg that's 
already the default.


>The goal is to have all of these things work:
>1) import MODULE should import whatever the vendor decided should be the
>default.
>2) requires.txt in a project's egginfo should work to select a specific
>version from the installed eggs.
>3) In a simple script (without requires.txt), pkg_resources.requires()
>should work to select a specific version from the installed eggs.

#1 and #2 are easy.  #3 is possible only if you use the hack that 
easy_install-generated scripts use.


>For 3) No eggs for this module can be on sys.path before the
>pkg_resources.require() call.  This does not count modules brought in
>via site-packages as those are going to be overridden when we place egg
>paths before site-packages.

Yes it *does too* count, which is why it doesn't work as you expect.


>You seem to think there's something wrong with this so there's obviously
>something you're seeing that I don't.  Can you give an example of where
>this will fail?

Any time you expect to be able to import the non-default version of a 
package, and you either run the interpreter with no script, or run a 
script that wasn't generated by easy_install or otherwise hacked to 
work around the conflict issue.


From pje at telecommunity.com  Sun Sep  2 03:05:07 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 01 Sep 2007 21:05:07 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D9CDBC.4030504@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com>
Message-ID: <20070902010241.76F353A4092@sparrow.telecommunity.com>

At 01:38 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>I'm seeing an actual problem with this when starting with the following
>sys.path:
>  ['/usr/bin', '/usr/lib/python25.zip', '/usr/lib/python2.5',
>'/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk',
>'/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages',
>'/usr/lib/python2.5/site-packages/Numeric',
>'/usr/lib/python2.5/site-packages/PIL',
>'/usr/lib/python2.5/site-packages/TestGears-0.2-py2.5.egg-info',

Why do you have this on sys.path?  .egg-info files or directories 
should never appear on sys.path.


>'/usr/lib/python2.5/site-packages/gst-0.10',
>'/usr/lib/python2.5/site-packages/gtk-2.0',
>'/usr/lib/python2.5/site-packages/pyinotify',
>'/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode']
>
>/usr/lib/python2.5/site-packages/CherryPy-2.2.1-py2.5.egg is being
>inserted before '/usr/lib/python2.5/site-packages'.  Then another
>/usr/lib/python2.5/site-packages is entering the method and being placed
>after /usr/lib/python2.5... which places it before the CherryPy egg.

That sounds odd, since there should not be a need to add 
site-packages more than once.  In fact, that's what sounds like the 
actual bug here, since IIRC .insert_on() should never be called on a 
distribution whose .location is already on sys.path. 


From pje at telecommunity.com  Sun Sep  2 03:18:24 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 01 Sep 2007 21:18:24 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46D9DF7D.705@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
Message-ID: <20070902011557.C52373A4092@sparrow.telecommunity.com>

At 02:54 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>Toshio Kuratomi wrote:
> >
> > This code creates an unordered list because you might have already had
> > to place something after one of the duplicate locations that this code
> > is removing.  We need to do a sort of the entire list after each add +
> > duplicate removal run.
>
>Here's a quick and dirty patch.  It's correct but it's inefficient.  A
>better sort algorithm would help some.  Refactoring code so we call a
>sort method after we finish adding entries to the path would be even better.
>
>I'll test how this affects packaging tonight.

Please, please stop and back up a minute.   appreciate your eagerness 
to help, really, but right now I can't trust the correctness of the 
initial system against which you are performing your tests.  You've 
got crazy stuff on sys.path, you appear to be calling insert_on() 
(which isn't a documented API), and you're manually creating .pth 
files, among other things.

So before you do anything else, please restore your system to 
something which consists *only* of files generated by setuptools or 
easy_install without *any* added hacks, workarounds, or manually 
edited files.  That also means NO scripts that were not generated by 
easy_install.

If you *must* have other scripts, there is an undocumented internal 
feature that you can use to specify a script's requirements such that 
they override the default package versions.  What you have to do is 
add a __requires__ definition to the script, e.g.:

    __requires__= 'TurboGears>1.0', 'FibbledyDee<27.2'

This definition must be in the actual script run by Python.  When 
pkg_resources is initially imported, any __requires__ requirements 
are given higher precedence than the default versions.  You must, 
however, still import pkg_resources, and it must be imported *after* 
setting __requires__, not before.

Assuming I understand your requirements, you should be able to 
accomplish everything you want using only this one feature, plus 
single-version eggs for system default packages, and multi-version 
eggs (i.e. *no* .pth files) for everything else.

It should not be necessary for you to generate or edit any files, use 
other undocumented APIs, or anything else.


From a.badger at gmail.com  Sun Sep  2 04:07:26 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 19:07:26 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902010241.76F353A4092@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com>
	<20070902010241.76F353A4092@sparrow.telecommunity.com>
Message-ID: <46DA1ADE.7090001@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 01:38 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>> I'm seeing an actual problem with this when starting with the following
>> sys.path:
>>  ['/usr/bin', '/usr/lib/python25.zip', '/usr/lib/python2.5',
>> '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk',
>> '/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages',
>> '/usr/lib/python2.5/site-packages/Numeric',
>> '/usr/lib/python2.5/site-packages/PIL',
>> '/usr/lib/python2.5/site-packages/TestGears-0.2-py2.5.egg-info',
> 
> Why do you have this on sys.path?  .egg-info files or directories should
> never appear on sys.path.
> 
Beats me.

/me looks at the cvs log for the spec file that generated TestGears

Looks like the person that built the file initially in 2005 created a
zipped-egg and hand created a .pth to go with it.  A new maintainer took
over and when they rebuilt the package for the setuptools that changed
- --root to include --single-version-externally-managed they were confused
and changed the .pth to include the egg-info directory.

/me makes a change in cvs and removes the package from his system for now.

> 
>> '/usr/lib/python2.5/site-packages/gst-0.10',
>> '/usr/lib/python2.5/site-packages/gtk-2.0',
>> '/usr/lib/python2.5/site-packages/pyinotify',
>> '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode']
>>
>> /usr/lib/python2.5/site-packages/CherryPy-2.2.1-py2.5.egg is being
>> inserted before '/usr/lib/python2.5/site-packages'.  Then another
>> /usr/lib/python2.5/site-packages is entering the method and being placed
>> after /usr/lib/python2.5... which places it before the CherryPy egg.
> 
> That sounds odd, since there should not be a need to add site-packages
> more than once.  In fact, that's what sounds like the actual bug here,
> since IIRC .insert_on() should never be called on a distribution whose
> .location is already on sys.path.


Yeah, I thought it was odd too :-).  But I was only instrumenting the
code to figure out what was going on so I took it at face value.

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2hreX6yAic2E7kgRAmGcAKCc9vqFG0JYqdv3dLDL1jJMfeaPEgCaAlat
71bb8k2Cc+sLCaHgWH+lwEU=
=Tckz
-----END PGP SIGNATURE-----

From a.badger at gmail.com  Sun Sep  2 04:45:22 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 19:45:22 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902011557.C52373A4092@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
Message-ID: <46DA23C2.1040308@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 02:54 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>> Toshio Kuratomi wrote:
>> >
>> > This code creates an unordered list because you might have already had
>> > to place something after one of the duplicate locations that this code
>> > is removing.  We need to do a sort of the entire list after each add +
>> > duplicate removal run.
>>
>> Here's a quick and dirty patch.  It's correct but it's inefficient.  A
>> better sort algorithm would help some.  Refactoring code so we call a
>> sort method after we finish adding entries to the path would be even
>> better.
>>
>> I'll test how this affects packaging tonight.
> 
> Please, please stop and back up a minute.   appreciate your eagerness to
> help, really, but right now I can't trust the correctness of the initial
> system against which you are performing your tests.  You've got crazy
> stuff on sys.path, you appear to be calling insert_on() (which isn't a
> documented API), and you're manually creating .pth files, among other
> things.
> 
Well, *I'm* not calling insert_on().  setuptools is calling insert_on()
when it initializes via __requires__ = "TurboGears"; import
pkg_resources (I did not write that script, it's tg-admin from TurboGears.)

I just instrumented the pkg_resources() code to figure out what's going
on and found sys.path being updated incorrectly.

As for manually created .pth files, all the ones I created were removed
right after I tested that they didn't work.  None of the results I've
posted involved them.  However, there are .pth files that were generated
by other people's packaging.  I'll go through and remove or fix all of
those packages on my system.

> So before you do anything else, please restore your system to something
> which consists *only* of files generated by setuptools or easy_install
> without *any* added hacks, workarounds, or manually edited files.  That
> also means NO scripts that were not generated by easy_install.
>
Uhm... I can get rid of the two packages that I changed and take care of
the .pth files but really unless I remove everything but python and
setuptools from this system, there's going to be packages that were
built with distutils, setuptools, configure scripts, and etc here.
Everything is managed via rpm so I could do that but I'd rather know
what in particular I need to get rid of.  (Like: Remove anything that
installs a .pth file.)  If you say remove everything that installs an
egg that I haven't audited how it builds I can do that pretty easily.

> If you *must* have other scripts, there is an undocumented internal
> feature that you can use to specify a script's requirements such that
> they override the default package versions.  What you have to do is add
> a __requires__ definition to the script, e.g.:
> 
>    __requires__= 'TurboGears>1.0', 'FibbledyDee<27.2'
> 
> This definition must be in the actual script run by Python.  When
> pkg_resources is initially imported, any __requires__ requirements are
> given higher precedence than the default versions.  You must, however,
> still import pkg_resources, and it must be imported *after* setting
> __requires__, not before.
> 
> Assuming I understand your requirements, you should be able to
> accomplish everything you want using only this one feature, plus
> single-version eggs for system default packages, and multi-version eggs
> (i.e. *no* .pth files) for everything else.
>

Does this work from the interpreter shell?
>>> __requires__ = 'SQLAlchemy>=0.3,<0.4beta1'
>>> import pkg_resources
>>> import sqlalchemy

That might do the trick but it's not ideal for us to drop support for a
documented interface in favor of an undocumented one.  I'm writing
guidelines for packagers, not programmers.  By and large, we're not
writing scripts, we're packaging python modules so that people who are
writing scripts can get their work done.  It's important that the
documented way of doing things works.

pkg_resource.requires() is documented on the web page, in
pkg_resource.txt, and when you python setup.py install
- --single-version-externally-managed.  __requires__ is documented...
nowhere.  If you're willing to change documentation that says to use
pkg_resources.require() to use __requires__ instead then this will work
out perfectly.  If not, we'll need to find a way to make
pkg_resources.require() work so that we aren't constantly explaining to
people that it doesn't work because upstream tells us it doesn't work
and we're sorry that all the official upstream documentation says otherwise.

> It should not be necessary for you to generate or edit any files, use
> other undocumented APIs, or anything else.
> 
That's the unstated goal that goes along with the other three :-)

Do you IRC?  I'll fix and remove packages and then tell you what still
works and doesn't work.

- -Toshio

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2iPCX6yAic2E7kgRAkabAJoCbxLbgaiwtmjz19pyEESBTi4DZgCeO4W1
qok/7j9Jl2NW4UFQCpW12lw=
=ye7N
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Sun Sep  2 06:36:12 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 02 Sep 2007 00:36:12 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46DA23C2.1040308@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
Message-ID: <20070902043441.46DD43A40FF@sparrow.telecommunity.com>

At 07:45 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>Does this work from the interpreter shell?
> >>> __requires__ = 'SQLAlchemy>=0.3,<0.4beta1'
> >>> import pkg_resources
> >>> import sqlalchemy

Yes.


>pkg_resource.requires() is documented on the web page, in
>pkg_resource.txt, and when you python setup.py install
>- --single-version-externally-managed.  __requires__ is documented...
>nowhere.  If you're willing to change documentation that says to use
>pkg_resources.require() to use __requires__ instead then this will work
>out perfectly.  If not, we'll need to find a way to make
>pkg_resources.require() work so that we aren't constantly explaining to
>people that it doesn't work because upstream tells us it doesn't work
>and we're sorry that all the official upstream documentation says otherwise.

As I've explained repeatedly before, your choices are to either have 
a default version, or not to have one.  If you do not have one, then 
everything works as you wish, except for the fact that you must 
always explicitly require() something to use it (because there's no default).

If do you have a default version, then the only way to get something 
*other* than the default version is to use __requires__ or a 
setuptools-generated script (which automatically includes __requires__).


From a.badger at gmail.com  Sun Sep  2 08:37:25 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 01 Sep 2007 23:37:25 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902043441.46DD43A40FF@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
	<20070902043441.46DD43A40FF@sparrow.telecommunity.com>
Message-ID: <46DA5A25.8060509@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 07:45 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>> pkg_resource.requires() is documented on the web page, in
>> pkg_resource.txt, and when you python setup.py install
>> - --single-version-externally-managed.  __requires__ is documented...
>> nowhere.  If you're willing to change documentation that says to use
>> pkg_resources.require() to use __requires__ instead then this will work
>> out perfectly.  If not, we'll need to find a way to make
>> pkg_resources.require() work so that we aren't constantly explaining to
>> people that it doesn't work because upstream tells us it doesn't work
>> and we're sorry that all the official upstream documentation says
>> otherwise.
> 
> As I've explained repeatedly before, your choices are to either have a
> default version, or not to have one.  If you do not have one, then
> everything works as you wish, except for the fact that you must always
> explicitly require() something to use it (because there's no default).
> 
> If do you have a default version, then the only way to get something
> *other* than the default version is to use __requires__ or a
> setuptools-generated script (which automatically includes __requires__).
> 
> 
Yes.  And I'm repeating that the problem is the documentation doesn't
match the behaviour.  If using __requires__ works, then the
documentation needs to mention it.  Preferably, it should substitute for
all the places where pkg_resources.require() is currently highlighted as
the right way to do things.  For instance, this output from easy_install::
'''
Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

    pkg_resources.require("SQLAlchemy")  # latest installed version
    pkg_resources.require("SQLAlchemy==0.3.10")  # this exact version
    pkg_resources.require("SQLAlchemy>=0.3.10")  # this version or higher
'''

I realize that taken in the vacuum of that single easy_install run,
require() works.  But the instructions are neglecting to tell the user
that things are more complex than that.  That depending on how the other
versions of the module are installed, pkg_resources may not work at all.
 Since __require__ works for this instance and for a mixture of -m and
- -s isn't it best to give users instructions that they can use everywhere?

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2lokX6yAic2E7kgRAjtcAKCw1qg4iRrQAQxeyyX8sE9WvMwMDwCbB6ZY
ZrjwE12dAxBKrEoGra2b19s=
=qjbi
-----END PGP SIGNATURE-----

From a.badger at gmail.com  Sun Sep  2 09:49:49 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sun, 02 Sep 2007 00:49:49 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902011557.C52373A4092@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
Message-ID: <46DA6B1D.50309@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> 
> Please, please stop and back up a minute.   appreciate your eagerness to
> help, really, but right now I can't trust the correctness of the initial
> system against which you are performing your tests.  You've got crazy
> stuff on sys.path, you appear to be calling insert_on() (which isn't a
> documented API), and you're manually creating .pth files, among other
> things.
> 
> So before you do anything else, please restore your system to something
> which consists *only* of files generated by setuptools or easy_install
> without *any* added hacks, workarounds, or manually edited files.  That
> also means NO scripts that were not generated by easy_install.

Bug still exists with only one .pth:
  easy-install.pth
Only two eggs:
  setuptools-0.6c6-py2.5.egg-info:
  CherryPy-2.2.1-py2.5.egg:

cherrypy3.0 is installed as a non-egg with the cherrypy directory in
site-packages/cherrypy/

cherrypy2.2 is installed as a -m egg in:
CherryPy-2.2.1-py2.5.egg/
	cherrypy
	EGG-INFO

Here's the test:
>>> __requires__='CherryPy>=2.0,<2.100'
>>> import pkg_resources
>>> import cherrypy
>>> print cherrypy.__version__
3.0.2

Let me know what else you need.

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2msdX6yAic2E7kgRAgS+AKCVf63bWReMXxV8/hXi0GRBaT9kvwCgpDKp
iGFlzazCCgjlIcJDR1/aZCU=
=XC5y
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Sun Sep  2 21:06:33 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 02 Sep 2007 15:06:33 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46DA5A25.8060509@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
	<20070902043441.46DD43A40FF@sparrow.telecommunity.com>
	<46DA5A25.8060509@gmail.com>
Message-ID: <20070902190405.9A83D3A408D@sparrow.telecommunity.com>

At 11:37 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>I realize that taken in the vacuum of that single easy_install run,
>require() works.  But the instructions are neglecting to tell the user
>that things are more complex than that.  That depending on how the other
>versions of the module are installed, pkg_resources may not work at all.

You're taking this out of context.  When you run "easy_install -m", 
the normal result is that *there is no default package*.  So 
require() is both necessary and sufficient in that case.


>  Since __require__ works for this instance and for a mixture of -m and
>- -s isn't it best to give users instructions that they can use everywhere?

The problem is that you are adding a new situation to "everywhere", 
that previously didn't exist -- mixing single-version and 
multi-version at the system packaging level.

Either have a default version and deal with conflicts, or no default 
version and be explicit.  __requires__ is a workaround to avoid 
conflicts in specific cases.  It is intended only as a workaround, 
and really it's only for tools like easy_install and zc.buildout that 
generate scripts from a project description.

I do not intend to support it for any other usage.  If you, as a 
packager, wish to package scripts that use __requires__, I don't see 
a problem with that.  It is emphatically *not* intended for general use.

There are already tools in place to do what you want; you're just 
trying to get away with not using them.

To put it another way, if you want to support multiple versions with 
a default, then you need to support the *end-user being able to 
choose* what version is the default.  If the user does an 
easy_install of some specific version, then *that* version needs to 
become the default.  And the only way to support that, is for you to 
generate your scripts with easy_install or zc.buildout, or to make 
your own tool that generates scripts using __requires__.  If you 
don't, then your system-level Python scripts will be hosed if the 
user changes the default version of a package they use.

Now, it could be argued that you have packages that don't declare 
their dependencies currently (i.e., they don't use setuptools), and 
would also be hosed by a change in the default version.  But if 
that's true, then you're not fully supporting multiple versions.

So, my point here is that this issue is inherent to the nature of 
multiple versions.  Either you have a default, or you don't.  If you 
don't, you can't have conflicts but have to be explicit.  If you do 
have a default, then you need to *let the user change it*, and make 
everything you supply be explicit (so that it will automatically 
resolve conflicts).

I don't plan to support __requires__ for end-user scripts, but I'm 
happy to co-ordinate with system-level packagers and the authors of 
packaging tools to make sure you don't get messed up when it 
(eventually) goes away and is replaced by something better.


From a.badger at gmail.com  Sun Sep  2 22:12:52 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sun, 02 Sep 2007 13:12:52 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902190405.9A83D3A408D@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
	<20070902043441.46DD43A40FF@sparrow.telecommunity.com>
	<46DA5A25.8060509@gmail.com>
	<20070902190405.9A83D3A408D@sparrow.telecommunity.com>
Message-ID: <46DB1944.6010900@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 11:37 PM 9/1/2007 -0700, Toshio Kuratomi wrote:
>> I realize that taken in the vacuum of that single easy_install run,
>> require() works.  But the instructions are neglecting to tell the user
>> that things are more complex than that.  That depending on how the other
>> versions of the module are installed, pkg_resources may not work at all.
> 
> You're taking this out of context.  When you run "easy_install -m", the
> normal result is that *there is no default package*.  So require() is
> both necessary and sufficient in that case.
>
I'm saying that this fits into a larger context.  If we go forward with
using setuptools and eggs to manage multiple versions instead of coming
up with an ad hoc solution that changes the path ourselves, then
requires() is not sufficient.  So lacking documentation telling the end
user how to deal with this, we'll end up having to field more questions
about what to do in this situation.  This is not the situation we want
to be in.

> 
>>  Since __require__ works for this instance and for a mixture of -m and
>> - -s isn't it best to give users instructions that they can use
>> everywhere?
> 
> The problem is that you are adding a new situation to "everywhere", that
> previously didn't exist -- mixing single-version and multi-version at
> the system packaging level.
> 
Well, I'd argue that "everywhere" encompasses wierd situations that you
and I haven't even thought of yet that exist only on the planet Vogon
:-).  I'm just making use of your tool in a situation that seems logical
but hasn't been widespread until now.  So you can make a choice and say
"your situation is not the intended use of setuptools, please fork it or
come up with another way of achieving your ends" or figure out what, if
anything, is missing and help setuptools adapt to the situation.

> Either have a default version and deal with conflicts, or no default
> version and be explicit.

I'm trying to deal with conflicts.  And my understanding was that you
preferred people use __requires__ to do that.  In fact, you seemed to
say that __requires__ was the only way for people not using setuptools
to do that.

>  __requires__ is a workaround to avoid
> conflicts in specific cases.  It is intended only as a workaround, and
> really it's only for tools like easy_install and zc.buildout that
> generate scripts from a project description.
> 

1) Why a workaround?
2) What are the specific cases that it's limited to?

> I do not intend to support it for any other usage.  If you, as a
> packager, wish to package scripts that use __requires__, I don't see a
> problem with that.  It is emphatically *not* intended for general use.
> 
What is the real way to allow people to do quick and dirty scripting and
experimentation from the interpreter shell in this environment?

And once again, we are *not* creating and packaging scripts that use
__require__.  We are packaging multiple versions of modules with a
default.  We need to have instructions for end-users who want to do
quick and dirty scripting or experiment in the interpreter shell how to
select the version that they wish.

> There are already tools in place to do what you want; you're just trying
> to get away with not using them.
> 
No.  I'm trying to provide end users with an understanding of how to
work with the environment we are giving them.  We have no control over
what end users want to do so we have to give them enough information to
make an educated choice about how they can do the tasks they're used to.

> To put it another way, if you want to support multiple versions with a
> default, then you need to support the *end-user being able to choose*
> what version is the default.  

I'm trying to give end users a choice.  giving them a choice of defaults
is outside the scope of what I'm trying to accomplish but I'll be happy
if it works.

> If the user does an easy_install of some
> specific version, then *that* version needs to become the default.

This is fine with me but is really an extra level on top of what we're
trying to do.  We're working on system packaging.  The system packages
have to work together to give the user the ability to import MODULE and
the ability to get more specific than that.  If an end user
easy_installs something on top of the system packages it should "work"
with the packages installed on the system.  It doesn't matter to me, as
a system packager whether the end user decides to make it the default or
decides to make it explicitly requestable only as long as they are still
able to use their own version of the package if they so choose.

>  And
> the only way to support that, is for you to generate your scripts with
> easy_install or zc.buildout, or to make your own tool that generates
> scripts using __requires__.  If you don't, then your system-level Python
> scripts will be hosed if the user changes the default version of a
> package they use.
> 
Once again, *we are not creating any system level python scripts that
make use of a specific, non-default version*.  If we do, I'll be sure to
tell people to use easy_install to start their projects.

What we are doing is providing multiple module versions A) For end users
who need to use a different version in their scripts B) For projects
that have not been ported to a new version of the module.  We have no
control over how those upstream scripts are written -- whether they use
easy_install, hand code __requires__, or munge sys.path themselves.

> Now, it could be argued that you have packages that don't declare their
> dependencies currently (i.e., they don't use setuptools), and would also
> be hosed by a change in the default version.  But if that's true, then
> you're not fully supporting multiple versions.
> 
> So, my point here is that this issue is inherent to the nature of
> multiple versions.  Either you have a default, or you don't.  If you
> don't, you can't have conflicts but have to be explicit.  If you do have
> a default, then you need to *let the user change it*, and make
> everything you supply be explicit (so that it will automatically resolve
> conflicts).
> 
This is true.  But it's not going to change until all upstreams have
moved to using setuptools.  We provide a set of packages that work
together.  And we provide a means for the user to shoot themselves in
the foot.  But this is nothing new, they've had access to PYTHONPATH,
su, rm -rf, etc in the past.

What we can do is make as much of what they expect to work work as it
used to.  And where it differs, we offer a standard, logical, and simple
method to get the behaviour they want.

> I don't plan to support __requires__ for end-user scripts, but I'm happy
> to co-ordinate with system-level packagers and the authors of packaging
> tools to make sure you don't get messed up when it (eventually) goes
> away and is replaced by something better.
> 
I don't have too much of a problem with things changing.  But that is
also why I'm pushing so hard for this to be documented.  1) We'll want
to know when this changes and do the upgrade between releases of our
distribution with announcements of the change.  That way people will
know, in Fedora 80 we use __requires__ to request a specific version
from setuptools, in Fedora 81 we use the new and vastly superior
technique.  2) By being documented upstream, people won't accuse us of
relying on a private method that no one is supposed to use.  It'll
simply be one piece of API being replaced with another.

If you feel that __requires__ is a private method that shouldn't be used
then we'd respect that.  But between that and your teelling us that
pkg_resources.require() isn't meant to work like that, it means that
there's simply no method for making setuptools do what we need it to and
we'll have to find some other method to accomplish this.

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG2xlEX6yAic2E7kgRAvOSAKCqsrWaqyiZSqexLcLWZf6XN4ruwgCfbV5t
aan/Oqi39w8sT3yO1eI98FE=
=0Yv2
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Mon Sep  3 00:50:10 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 02 Sep 2007 18:50:10 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46DB1944.6010900@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
	<20070902043441.46DD43A40FF@sparrow.telecommunity.com>
	<46DA5A25.8060509@gmail.com>
	<20070902190405.9A83D3A408D@sparrow.telecommunity.com>
	<46DB1944.6010900@gmail.com>
Message-ID: <20070902224742.DDFF03A408D@sparrow.telecommunity.com>

At 01:12 PM 9/2/2007 -0700, Toshio Kuratomi wrote:
>What is the real way to allow people to do quick and dirty scripting and
>experimentation from the interpreter shell in this environment?

Either:

1. Select the desired default version using "easy_install package==version", or
2. Have *no* default version, and always use require() (or 
automatically-managed dependencies declared via a setup.py script).


>And once again, we are *not* creating and packaging scripts that use
>__require__.  We are packaging multiple versions of modules with a
>default.

And once again, this won't work.  Either you must not have a default 
version, or you must deal with the potential conflict in your scripts.

The supported way of doing that is to generate the scripts using 
easy_install.  If you are packaging something that doesn't use 
setuptools explicitly, you'll just have to patch it to declare its 
dependencies.  You're already patching the .spec file; heck, if it 
would make it more palatable, perhaps we could add a way to specify 
the requirements via the command line.  (Alternately, you could just 
create a requires.txt in the .egg-info directory.)


>   We need to have instructions for end-users who want to do
>quick and dirty scripting or experiment in the interpreter shell how to
>select the version that they wish.

easy_install package==version will change the default version, and 
require() will work if there is no default version.  Those are the 
supported ways of doing this.


>Once again, *we are not creating any system level python scripts that
>make use of a specific, non-default version*.

I'm saying that even your programs that use the default version, 
should explicitly specify what version they use.  If you're going to 
support multiple versions, then declare *all* the dependencies, or at 
least the ones that would otherwise have conflicts.


>B) For projects
>that have not been ported to a new version of the module.  We have no
>control over how those upstream scripts are written -- whether they use
>easy_install, hand code __requires__, or munge sys.path themselves.

The correct instructions are for them to use setuptools and declare 
their dependencies, or live with whatever the status quo is.


>This is true.  But it's not going to change until all upstreams have
>moved to using setuptools.  We provide a set of packages that work
>together.  And we provide a means for the user to shoot themselves in
>the foot.  But this is nothing new, they've had access to PYTHONPATH,
>su, rm -rf, etc in the past.
>
>What we can do is make as much of what they expect to work work as it
>used to.  And where it differs, we offer a standard, logical, and simple
>method to get the behaviour they want.

Right - and the single simplest solution for all of this is to simply 
use setuptools as it was designed: select a default version using 
easy_install, and declare dependencies for everything 
else.  Alternatively, don't install *anything* as the default 
version, and declare *all* dependencies.

If your goal is to have simple and consistent behavior, the second is 
the best choice.  If you want the greatest backward-compatibility, 
the first is the best choice.

Note that it isn't really possible for users *now* to choose a 
different package version at runtime, so requiring the use of 
easy_install to switch default versions doesn't seem like a big 
deal.  They already have to make this choice by what RPMs they 
install, and it's just as global a choice -- except that this way, 
they can do it without breaking anything.


>If you feel that __requires__ is a private method that shouldn't be used
>then we'd respect that.  But between that and your teelling us that
>pkg_resources.require() isn't meant to work like that, it means that
>there's simply no method for making setuptools do what we need it to and
>we'll have to find some other method to accomplish this.

There are *two* ways to accomplish it: have a default (and use 
easy_install to change it when desired), or don't have a default.

Either way, you are going to ultimately end up at a place where 
everything needs to declare its dependencies, preferably in a 
setup.py.  Having a default version may be useful for the transition 
period, but the best way to encourage people to actually transition 
is to have some *benefit* for declaring their dependencies -- like 
being able to use the non-default versions.

This is why I even discourage direct use of require(), as it says in the docs:

"""In general, it should not be necessary for you to call this method 
directly. It's intended more for use in quick-and-dirty scripting and 
interactive interpreter hacking than for production use. If you're 
creating an actual library or application, it's strongly recommended 
that you create a "setup.py" script using setuptools, and declare all 
your requirements there. That way, tools like EasyInstall can 
automatically detect what requirements your package has, and deal 
with them accordingly."""


From pje at telecommunity.com  Mon Sep  3 00:52:26 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 02 Sep 2007 18:52:26 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46DA6B1D.50309@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA6B1D.50309@gmail.com>
Message-ID: <20070902224958.7F1FD3A408D@sparrow.telecommunity.com>

At 12:49 AM 9/2/2007 -0700, Toshio Kuratomi wrote:
>Bug still exists with only one .pth:
>   easy-install.pth
>Only two eggs:
>   setuptools-0.6c6-py2.5.egg-info:
>   CherryPy-2.2.1-py2.5.egg:
>
>cherrypy3.0 is installed as a non-egg with the cherrypy directory in
>site-packages/cherrypy/
>
>cherrypy2.2 is installed as a -m egg in:
>CherryPy-2.2.1-py2.5.egg/
>         cherrypy
>         EGG-INFO
>
>Here's the test:
> >>> __requires__='CherryPy>=2.0,<2.100'
> >>> import pkg_resources
> >>> import cherrypy
> >>> print cherrypy.__version__
>3.0.2
>
>Let me know what else you need.

What are the current contents of easy-install.pth?


From a.badger at gmail.com  Mon Sep  3 06:34:54 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sun, 02 Sep 2007 21:34:54 -0700
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902224958.7F1FD3A408D@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA6B1D.50309@gmail.com>
	<20070902224958.7F1FD3A408D@sparrow.telecommunity.com>
Message-ID: <46DB8EEE.4070908@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 12:49 AM 9/2/2007 -0700, Toshio Kuratomi wrote:
>>
>> Let me know what else you need.
> 
> What are the current contents of easy-install.pth?
> 
> 
import sys; sys.__plen = len(sys.path)
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert =
p+len(new)

Do you want me to get rid of it and try again?

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG247uX6yAic2E7kgRAnHsAJ9KO93oSNgVH3xhuQwcB2pbd1EoaQCfftvj
VLnmUUNYGn0kV9qAYogS5FI=
=+rgQ
-----END PGP SIGNATURE-----

From stefan_ml at behnel.de  Mon Sep  3 11:19:11 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 03 Sep 2007 11:19:11 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
Message-ID: <46DBD18F.2000902@behnel.de>

Hi,

Phillip Eby asked me to forward this request to the distutils mailing list, so
here it goes.

In current setuptools, the file setuptools/extension.py checks for Pyrex being
installed and otherwise renames all .pyx file entries to .c. This does not
work when Cython is installed *instead of* Pyrex, as Cython will never be
executed in this case. It just fails saying that the .c file doesn't exist.

I'm currently shipping a fake Pyrex/Distutils/build_ext.py module with lxml to
make it work with both, but I don't even see a major advantage in the
extension.py module at all. It's plain easy to do that yourself in the
setup.py script if (and only if) you want it to build without Pyrex/Cython,
but currently, it is enforced in setuptools. This really only works if the
distributor wants it and does something to make it work, so always enforcing
this behaviour doesn't give you any real advantage.

I'd like to see this module removed from setuptools or at least have Cython
supported in the same way.

Stefan


From pje at telecommunity.com  Mon Sep  3 17:41:42 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 03 Sep 2007 11:41:42 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DBD18F.2000902@behnel.de>
References: <46DBD18F.2000902@behnel.de>
Message-ID: <20070903153913.CAAEC3A4045@sparrow.telecommunity.com>

At 11:19 AM 9/3/2007 +0200, Stefan Behnel wrote:
>In current setuptools, the file setuptools/extension.py checks for Pyrex being
>installed and otherwise renames all .pyx file entries to .c. This does not
>work when Cython is installed *instead of* Pyrex, as Cython will never be
>executed in this case. It just fails saying that the .c file doesn't exist.
>
>I'm currently shipping a fake Pyrex/Distutils/build_ext.py module with lxml to
>make it work with both, but I don't even see a major advantage in the
>extension.py module at all. It's plain easy to do that yourself in the
>setup.py script if (and only if) you want it to build without Pyrex/Cython,
>but currently, it is enforced in setuptools. This really only works if the
>distributor wants it and does something to make it work, so always enforcing
>this behaviour doesn't give you any real advantage.
>
>I'd like to see this module removed from setuptools or at least have Cython
>supported in the same way.

I guess I'm not clear on the actual problem here.  Do you want it to 
only rename .pyx to .c for .c files that actually exist?  A better 
way to specify that you can handle .pyx files?

Note that removing this feature is not an option that's on the table, 
as it was one of the original reasons I created setuptools in the 
first place, years before the idea of eggs even existed.

Now, in future versions of setuptools, the ones that eventually 
replace a lot of distutils framework with new stuff, there will 
definitely be a better way of handling this.  But I have no idea when 
that will happen.


From pje at telecommunity.com  Mon Sep  3 17:52:34 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 03 Sep 2007 11:52:34 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <46DB8EEE.4070908@gmail.com>
References: <1188364494.1528.8.camel@ignacio.lan>
	<20070829164031.24C963A4072@sparrow.telecommunity.com>
	<46D89599.4010707@gmail.com>
	<20070831233159.B10803A40AE@sparrow.telecommunity.com>
	<46D8C6B0.8060702@gmail.com> <46D8E880.5070204@gmail.com>
	<20070901180530.881D03A4092@sparrow.telecommunity.com>
	<46D9CDBC.4030504@gmail.com> <46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA6B1D.50309@gmail.com>
	<20070902224958.7F1FD3A408D@sparrow.telecommunity.com>
	<46DB8EEE.4070908@gmail.com>
Message-ID: <20070903155006.843BC3A4045@sparrow.telecommunity.com>

At 09:34 PM 9/2/2007 -0700, Toshio Kuratomi wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Phillip J. Eby wrote:
> > At 12:49 AM 9/2/2007 -0700, Toshio Kuratomi wrote:
> >>
> >> Let me know what else you need.
> >
> > What are the current contents of easy-install.pth?
> >
> >
>import sys; sys.__plen = len(sys.path)
>import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
>p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert =
>p+len(new)
>
>Do you want me to get rid of it and try again?

No.  I believe I've found the problem now; it's that .egg-info eggs 
don't need to be inserted ahead of their containing directory.  Try 
this patch; if it works for you, I'll put it out in an 0.6c7 release 
later today or tomorrow:

Index: pkg_resources.py
===================================================================
--- pkg_resources.py    (revision 55711)
+++ pkg_resources.py    (working copy)
@@ -2148,7 +2148,8 @@
          for p, item in enumerate(npath):
              if item==nloc:
                  break
-            elif item==bdir:
+            elif item==bdir and self.precedence==EGG_DIST:
+                # if it's an .egg, give it precedence over its directory
                  path.insert(p, loc)
                  npath.insert(p, nloc)
                  break


From stefan_ml at behnel.de  Mon Sep  3 18:01:47 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 03 Sep 2007 18:01:47 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
Message-ID: <46DC2FEB.2000203@behnel.de>


Phillip J. Eby wrote:
> At 11:19 AM 9/3/2007 +0200, Stefan Behnel wrote:
>> In current setuptools, the file setuptools/extension.py checks for
>> Pyrex being
>> installed and otherwise renames all .pyx file entries to .c. This does
>> not
>> work when Cython is installed *instead of* Pyrex, as Cython will never be
>> executed in this case. It just fails saying that the .c file doesn't
>> exist.
>>
>> I'm currently shipping a fake Pyrex/Distutils/build_ext.py module with
>> lxml to
>> make it work with both, but I don't even see a major advantage in the
>> extension.py module at all. It's plain easy to do that yourself in the
>> setup.py script if (and only if) you want it to build without
>> Pyrex/Cython,
>> but currently, it is enforced in setuptools. This really only works if
>> the
>> distributor wants it and does something to make it work, so always
>> enforcing
>> this behaviour doesn't give you any real advantage.
>>
>> I'd like to see this module removed from setuptools or at least have
>> Cython
>> supported in the same way.
> 
> I guess I'm not clear on the actual problem here.  Do you want it to
> only rename .pyx to .c for .c files that actually exist?

That would be an option. However, I still don't understand why it should be up
to setuptools to decide if *my* code distribution can be built without Pyrex
(or any other tool that names its files .pyx).

The extension module in setuptools could also check if build_ext is the
original function or a replacement and only do its magic if it's alone in the
field and thinks it can't hurt anyone. That would only be another heuristic,
but it would at least have prevented cases like this...


> A better way to specify that you can handle .pyx files?

You mean a hook into setuptools so that Cython can fix the extension module?
Doesn't sound like a clean solution to me...


> Note that removing this feature is not an option that's on the table, as
> it was one of the original reasons I created setuptools in the first
> place, years before the idea of eggs even existed.

So you are suggesting that people should choose: it's either eggs or Cython. I
don't think eggs are that important to everyone, but I don't think this
decision should be enforced either. Maybe you should split setuptools into a
part that serves everyone and a part that served you years before the idea of
eggs even existed.

Don't get me wrong, I really like eggs and the EasyInstall machinery. But it
shouldn't get in the way of the projects it aims to serve.


> Now, in future versions of setuptools, the ones that eventually replace
> a lot of distutils framework with new stuff, there will definitely be a
> better way of handling this.  But I have no idea when that will happen.

Well, splitting it up could make it easier to improve things gradually.

Stefan

From pje at telecommunity.com  Mon Sep  3 18:12:41 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 03 Sep 2007 12:12:41 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DC2FEB.2000203@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
Message-ID: <20070903161014.A31463A40A5@sparrow.telecommunity.com>

At 06:01 PM 9/3/2007 +0200, Stefan Behnel wrote:
>So you are suggesting that people should choose: it's either eggs or Cython.

No -- I'm suggesting that developers whose projects include C code 
generated from .pyx files should always include the resulting .c 
files in their source distributions as a courtesy to their users.

Not only is this the friendly thing for your users, it's also the 
only sane thing to do from a support standpoint.  Otherwise, you have 
to ensure that your users have exactly the same version of Pyrex (or 
Cython, or pypy, or whatever) that you do.

Please notice that these points are entirely independent of whether 
setuptools or eggs exist.  I was shipping generated Pyrex-generated 
.c files to my users for these reasons well before setuptools 
existed, let alone eggs.


From stefan_ml at behnel.de  Mon Sep  3 18:21:49 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 03 Sep 2007 18:21:49 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070903161014.A31463A40A5@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
Message-ID: <46DC349D.8050807@behnel.de>


Phillip J. Eby wrote:
> At 06:01 PM 9/3/2007 +0200, Stefan Behnel wrote:
>> So you are suggesting that people should choose: it's either eggs or
>> Cython.
> 
> No -- I'm suggesting that developers whose projects include C code
> generated from .pyx files should always include the resulting .c files
> in their source distributions as a courtesy to their users.
> 
> Not only is this the friendly thing for your users, it's also the only
> sane thing to do from a support standpoint.  Otherwise, you have to
> ensure that your users have exactly the same version of Pyrex (or
> Cython, or pypy, or whatever) that you do.
> 
> Please notice that these points are entirely independent of whether
> setuptools or eggs exist.  I was shipping generated Pyrex-generated .c
> files to my users for these reasons well before setuptools existed, let
> alone eggs.

So do I. lxml has been doing that long before we even started using
setuptools. The point is that current setuptools break developer builds. You
can't change the sources without having (a fake) Pyrex installed, because
setuptools will not let you run Cython.

Stefan

From pje at telecommunity.com  Mon Sep  3 18:40:52 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 03 Sep 2007 12:40:52 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DC349D.8050807@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
Message-ID: <20070903163824.66B8D3A40A5@sparrow.telecommunity.com>

At 06:21 PM 9/3/2007 +0200, Stefan Behnel wrote:
>The point is that current setuptools break developer builds. You
>can't change the sources without having (a fake) Pyrex installed, because
>setuptools will not let you run Cython.

If you want to use the native Extension type, do this:

    from distutils.core import Extension
    from setuptools import setup

Alternately, you can use:

    from setuptools.extension _import Extension as Extension

I will add this information to the existing setuptools documentation 
regarding Pyrex detection.

Meanwhile, if you'd like to suggest a way of detecting Cython's 
presence, I can probably add that to the setuptools.extension module.


From stefan_ml at behnel.de  Mon Sep  3 18:51:57 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 03 Sep 2007 18:51:57 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
Message-ID: <46DC3BAD.4010309@behnel.de>


Phillip J. Eby wrote:
> At 06:21 PM 9/3/2007 +0200, Stefan Behnel wrote:
>> The point is that current setuptools break developer builds. You
>> can't change the sources without having (a fake) Pyrex installed, because
>> setuptools will not let you run Cython.
> 
> If you want to use the native Extension type, do this:
> 
>    from distutils.core import Extension
>    from setuptools import setup

Have you tested this lately? I mean, with Python 2.5 distutils?

What I get is

Traceback (most recent call last):
  File "setup.py", line 89, in <module>
    **extra_options
  File "distutils/core.py", line 151, in setup
  File "distutils/dist.py", line 974, in run_commands
  File "distutils/dist.py", line 994, in run_command
  File "distutils/command/build_ext.py", line 290, in run
  File "distutils/command/build_ext.py", line 413, in build_extensions
  File "distutils/command/build_ext.py", line 315, in check_extensions_list
TypeError: iteration over non-sequence

this is the failing code:

            if isinstance(ext, Extension):
                continue                # OK! (assume type-checking done
                                        # by Extension constructor)
==>         (ext_name, build_info) = ext


> Alternately, you can use:
> 
>    from setuptools.extension _import Extension as Extension

I assume you meant

    from setuptools.extension import _Extension as Extension

That's definitely a hack, but it yields the same error as above.


> I will add this information to the existing setuptools documentation
> regarding Pyrex detection.
> 
> Meanwhile, if you'd like to suggest a way of detecting Cython's
> presence, I can probably add that to the setuptools.extension module.

    import Cython

should do the trick.

I still prefer the "only fiddle with extensions if build_ext wasn't replaced"
way, though.

Stefan

From robinbryce at gmail.com  Mon Sep  3 23:38:37 2007
From: robinbryce at gmail.com (Robin Bryce)
Date: Mon, 3 Sep 2007 22:38:37 +0100
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <20070902224742.DDFF03A408D@sparrow.telecommunity.com>
References: <1188364494.1528.8.camel@ignacio.lan> <46D9CDBC.4030504@gmail.com>
	<46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
	<20070902043441.46DD43A40FF@sparrow.telecommunity.com>
	<46DA5A25.8060509@gmail.com>
	<20070902190405.9A83D3A408D@sparrow.telecommunity.com>
	<46DB1944.6010900@gmail.com>
	<20070902224742.DDFF03A408D@sparrow.telecommunity.com>
Message-ID: <bcf87d920709031438y5bd15f15qf94ec8697405f5d0@mail.gmail.com>

On 02/09/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 01:12 PM 9/2/2007 -0700, Toshio Kuratomi wrote:
> >What is the real way to allow people to do quick and dirty scripting and
> >experimentation from the interpreter shell in this environment?
>
> Either:
>
> 1. Select the desired default version using "easy_install package==version", or
> 2. Have *no* default version, and always use require() (or
> automatically-managed dependencies declared via a setup.py script).
>

Is it really setuptools job to attempt to address "quick & dirty
scripting" or "interactive experimentation" ?

I mean, as I understand it, easy_install is specifically about giving
you a clean painless install of a piece of python software without
having to care about the details and without needing to understand
anything about python.

But the expectations and needs of the user in Toshio's scenario are
completely at odds with this. This is a person who is clearly
interested in, or needs to understand, the details of some piece of
python software but who does not want to be concerned with all the
horrors of installation / distribution management. Or perhaps it's
simply a person who is *considering* system wide installation of some
python package and wishes to "try before they buy".

I frequently want to do exactly this sort of thing - often in
circumstances where I can not count on having a consistent setuptools
environment. Or, where I want to use a mix of installed and non
installed python packages. I almost always need to deal with a mix of
distutils installed, egg installed and plain old python. Occasionally
I'm in the situation where I have a setuptools egg on disc, I'd like
to use it, but I do not want to or simply can't install it.

Almost by definition anything that needs to be "installed" is not
going to help; Bootstrapping issues abound. But I *know* roughly where
the python code is that I want to make use of. And heck & damn it all;
There are a great many python packages whose source distributions are
perfectly capable of running in place - why should I have to "install"
such software before I'm satisfied it makes a useful addition to my
system ? In something of a fit of frustration I wrote this ~1000 line
python module:

http://svn.wiretooth.com/svn/open/pyrun/trunk/pyrun.py

http://svn.wiretooth.com/svn/open/pyrun/trunk/pyrun.html

wget is all the installation it needs. The -i -d and -p modes of
operation are the ones I think might address "quick & dirty scripting"
and "experimentation" from Toshios scenario.

I've not released it on cheeseshop because I wasn't really convinced
there would be much demand. And also because I didn't like the thought
that it might be confused as "general script stub" - I really would
not like to encounter a site installed script that used pyrun.py to
shim around installation / packaging issues.

I'm not sure I fully appreciate the issues your facing Toshio but your
posts struck something of a chord with me. If you think pyrun.py can
be made to help please let me know.

Cheers,

Robin

From pje at telecommunity.com  Tue Sep  4 00:34:56 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 03 Sep 2007 18:34:56 -0400
Subject: [Distutils] Disabling --single-version-externally-managed
In-Reply-To: <bcf87d920709031438y5bd15f15qf94ec8697405f5d0@mail.gmail.co
 m>
References: <1188364494.1528.8.camel@ignacio.lan> <46D9CDBC.4030504@gmail.com>
	<46D9DF7D.705@gmail.com>
	<20070902011557.C52373A4092@sparrow.telecommunity.com>
	<46DA23C2.1040308@gmail.com>
	<20070902043441.46DD43A40FF@sparrow.telecommunity.com>
	<46DA5A25.8060509@gmail.com>
	<20070902190405.9A83D3A408D@sparrow.telecommunity.com>
	<46DB1944.6010900@gmail.com>
	<20070902224742.DDFF03A408D@sparrow.telecommunity.com>
	<bcf87d920709031438y5bd15f15qf94ec8697405f5d0@mail.gmail.com>
Message-ID: <20070903223242.BFB243A40A5@sparrow.telecommunity.com>

At 10:38 PM 9/3/2007 +0100, Robin Bryce wrote:
>There are a great many python packages whose source distributions are
>perfectly capable of running in place - why should I have to "install"
>such software before I'm satisfied it makes a useful addition to my
>system ? In something of a fit of frustration I wrote this ~1000 line
>python module:
>
>http://svn.wiretooth.com/svn/open/pyrun/trunk/pyrun.py
>
>http://svn.wiretooth.com/svn/open/pyrun/trunk/pyrun.html
>
>wget is all the installation it needs. The -i -d and -p modes of
>operation are the ones I think might address "quick & dirty scripting"
>and "experimentation" from Toshios scenario.

AFAICT, pyrun doesn't handle dependencies between packages, let alone 
manage version conflicts, so I'm not clear on how it relates to the 
present discussion.


From pje at telecommunity.com  Tue Sep  4 01:17:18 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 03 Sep 2007 19:17:18 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DC3BAD.4010309@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
Message-ID: <20070903231504.BB0A03A40A5@sparrow.telecommunity.com>

At 06:51 PM 9/3/2007 +0200, Stefan Behnel wrote:
>I still prefer the "only fiddle with extensions if build_ext wasn't replaced"
>way, though.

If build_ext wasn't replaced where and how?  Do you mean in the 
"cmdclass" argument to setup()?  In an entry point?  What?


From stefan_ml at behnel.de  Tue Sep  4 08:09:14 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 04 Sep 2007 08:09:14 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
Message-ID: <46DCF68A.1010904@behnel.de>


Phillip J. Eby wrote:
> At 06:51 PM 9/3/2007 +0200, Stefan Behnel wrote:
>> I still prefer the "only fiddle with extensions if build_ext wasn't
>> replaced"
>> way, though.
> 
> If build_ext wasn't replaced where and how?  Do you mean in the
> "cmdclass" argument to setup()?

Exactly. As I said, that would also be only a heuristic, but it would still be
better than the current 'feature' enforcement code which basically states: "if
Pyrex isn't installed, you can't possibly have meant what you were requesting,
so I'll fix your code for you". Problem being that the code isn't broken and
the 'fix' breaks it.

Stefan

From pje at telecommunity.com  Tue Sep  4 18:43:45 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Sep 2007 12:43:45 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DCF68A.1010904@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
Message-ID: <20070904164117.0A3D03A4045@sparrow.telecommunity.com>

At 08:09 AM 9/4/2007 +0200, Stefan Behnel wrote:

>Phillip J. Eby wrote:
> > At 06:51 PM 9/3/2007 +0200, Stefan Behnel wrote:
> >> I still prefer the "only fiddle with extensions if build_ext wasn't
> >> replaced"
> >> way, though.
> >
> > If build_ext wasn't replaced where and how?  Do you mean in the
> > "cmdclass" argument to setup()?
>
>Exactly.

That would mean that you'd have to write code in *every* setup.py to 
do the replacement -- *and* you'd have to get it correct so that it 
does the right thing when Pyrex and Cython are *not* installed.


>but it would still be
>better than the current 'feature' enforcement code which basically states: "if
>Pyrex isn't installed, you can't possibly have meant what you were requesting,
>so I'll fix your code for you".

...and you simply get a different error message than you were going 
to get anyway, so there's no net increase in problems.  You are going 
to have to fix the real problem (by either including the .c files or 
installing the compiler), no matter what.  So how does setuptools' 
action here make any difference?  All that changes is the error message.


>Problem being that the code isn't broken and
>the 'fix' breaks it.

After thinking this over some more, I disagree.

The key design question I ask for setuptools features is "who gets 
the pain?", since nearly all design decisions will cause *somebody* 
pain.  Ideally, I want that pain to go to somebody who's in a 
position to do something that improves things for everybody in the long term.

In this case, the only additional pain is for the developers of new 
tools that process .pyx files (such as yourself).  They have to 
provide a Pyrex-like build_ext command, but in return they get 
correct handling for all their setup.py files without having to do 
anything else.  Their users then don't have to jump through setup() 
hoops, and *their* users don't have to worry about installing the extra tools.

On the whole, then, the pain is in the right place, and guides the 
recipient to the correct action (i.e., implementing a build_ext 
command, which you've already done).  Going the other way would 
encourage .pyx developers to be sloppy about including .c files, and 
encourage developers of .pyx tools to be sloppy about making the 
whole process transparent all the way to the end user.

On balance, in other words, I'd rather hurt .pyx-tool developers 
(population 1 so far as I know) than the people downstream of them.

In the long run, my plan is to add entry points to the "build" 
command so that other commands can be added.  This would let you have 
a command whose job is to turn .pyx files to .c, but only runs if 
it's installed.  In the meantime, faking Pyrex-ness is, IMO, the best 
way to deal with this.


From pje at telecommunity.com  Tue Sep  4 18:56:19 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Sep 2007 12:56:19 -0400
Subject: [Distutils] setuptools 0.6c7 released
Message-ID: <20070904165351.5489C3A4045@sparrow.telecommunity.com>

Highlights:

* Now uses the PyPI /simple API
* ftp: URLs fixed
* nested namespace packages work in --single-version-externally-managed
* single-version packages in site-packages don't override .egg in site-packages
* workaround for distutils' "broken symlinks" bug
* fix egg_info failing on new, uncommitted SVN directories


From sklein at cpcug.org  Tue Sep  4 18:39:27 2007
From: sklein at cpcug.org (Stanley A. Klein)
Date: Tue, 4 Sep 2007 12:39:27 -0400 (EDT)
Subject: [Distutils] Why are egg-info and related pth files required for rpm
	packages?
Message-ID: <21106.207.188.248.157.1188923967.squirrel@www.cpcug.org>

I recently installed Fedora 7 and looked at
/usr/lib/python2.5/site-packages.  The directory has numerous Python
packages installed but no egg-info and few .pth files.  Of the three .pth
files in my installation, only one has a content different from the name
of the package (pointing to a subdirectory of site-packages with a
different name).

Setuptools has the advantage of simplifying preparation of rpms because it
has find_packages().  However, it produces egg-info and .pth files as part
of the rpm's.

I can understand the need for these files when packaging into eggs for
installation in places other than python/site-packages or when packaging
for non-Linux operating systems.  But why are they needed when installing
as rpms on Linux systems into site-packages?

I think this issue applies to most Linux packaging, which is usually
either rpm or deb.


Stan Klein


From lists at zopyx.com  Tue Sep  4 19:21:41 2007
From: lists at zopyx.com (Andreas Jung)
Date: Tue, 04 Sep 2007 19:21:41 +0200
Subject: [Distutils] my wishlist for easy_install
In-Reply-To: <20070829083746.20641.qmail@oak.oeko.net>
References: <20070829083746.20641.qmail@oak.oeko.net>
Message-ID: <22AB98E85BE1004211E78FC7@suxmac2.local>



--On 29. August 2007 10:37:46 +0200 Toni Mueller <support at oeko.net> wrote:

>
> Hi,
>
> I'd really like to see these two features in easy_install which I can't
> find right now:
>
>  1 - coexist with the operating system/recognize packages which are
>      installed via operating system means
>  2 - remove packages
>
> Currently, when I want to easy_install a package, it starts downloading
> all sorts of stuff from PYPI although I have already similar (the same
> - maybe other versions) packages in my system which I installed via
> the operating system's package manager. I'd really like to not have to
> manually inspect a prospective package to then manually download and
> install all dependencies via the package manager, and then to proceed
> to easy_install -N the offending package, but instead have this process
> automated.
>

Switch to isolated Python environments....pretty easy using workingenv or 
virtual_python.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070904/d74efbb2/attachment.pgp 

From stefan_ml at behnel.de  Tue Sep  4 21:06:47 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 04 Sep 2007 21:06:47 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070904164117.0A3D03A4045@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
Message-ID: <46DDACC7.9080008@behnel.de>

Hi,

I noticed you just released a new version of setuptools without mentioning a
fix for the problem in the changelog, so I assume it's not fixed?


Phillip J. Eby wrote:
> At 08:09 AM 9/4/2007 +0200, Stefan Behnel wrote:
> 
>> Phillip J. Eby wrote:
>> > At 06:51 PM 9/3/2007 +0200, Stefan Behnel wrote:
>> >> I still prefer the "only fiddle with extensions if build_ext wasn't
>> >> replaced"
>> >> way, though.
>> >
>> > If build_ext wasn't replaced where and how?  Do you mean in the
>> > "cmdclass" argument to setup()?
>>
>> Exactly.
> 
> That would mean that you'd have to write code in *every* setup.py to do
> the replacement -- *and* you'd have to get it correct so that it does
> the right thing when Pyrex and Cython are *not* installed.

You already have to write code that imports and replaces build_ext, so having
to add a bit of code that replaces .pyx by .c is not asking to much IMHO.


>> but it would still be
>> better than the current 'feature' enforcement code which basically
>> states: "if
>> Pyrex isn't installed, you can't possibly have meant what you were
>> requesting,
>> so I'll fix your code for you".
> 
> ...and you simply get a different error message than you were going to
> get anyway, so there's no net increase in problems.  You are going to
> have to fix the real problem (by either including the .c files or
> installing the compiler), no matter what.  So how does setuptools'
> action here make any difference?  All that changes is the error message.

Because there is no easy way around it. That's the problem. Having setuptools
do its monkey patching in the background means you have to write code that
works around the patching, and, yes, you have to get it right, which is far
from obvious, involving a couple of test imports, catching ImportErrors and
the like. How is that better than writing a straight forward 1-liner for
replacing .pyx by .c?


>> Problem being that the code isn't broken and
>> the 'fix' breaks it.
> 
> After thinking this over some more, I disagree.
> 
> The key design question I ask for setuptools features is "who gets the
> pain?", since nearly all design decisions will cause *somebody* pain. 
> Ideally, I want that pain to go to somebody who's in a position to do
> something that improves things for everybody in the long term.
> 
> In this case, the only additional pain is for the developers of new
> tools that process .pyx files (such as yourself).

Wrong. I cannot fix this as a developer of Cython - at least not without
providing a fake build_ext.py in a fake Pyrex package, which would make
installing Pyrex and Cython on the same machine near impossible and which
would break every tool that requires Pyrex instead on Cython.

I can only fix this problem as a developer of lxml which *uses* Cython, by
shipping a fake Pyrex with my distribution and adding it to the SVN repository.

So you are currently shifting the burden on every developer who wants to
support Cython (maybe in addition to Pyrex). Great idea.


> [stripped incorrect comments based on a wrong assumption]
>
> In the long run, my plan is to add entry points to the "build" command
> so that other commands can be added.  This would let you have a command
> whose job is to turn .pyx files to .c, but only runs if it's installed. 
> In the meantime, faking Pyrex-ness is, IMO, the best way to deal with this.

No, it's not. I would rather like to have a way to deal with this from Cython
than from Cython-implemented distributions. But the best option IMHO would be
to fix setuptools.

Stefan

From stefan_ml at behnel.de  Tue Sep  4 21:24:41 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 04 Sep 2007 21:24:41 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DDACC7.9080008@behnel.de>
References: <46DBD18F.2000902@behnel.de>	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>	<46DC2FEB.2000203@behnel.de>	<20070903161014.A31463A40A5@sparrow.telecommunity.com>	<46DC349D.8050807@behnel.de>	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>	<46DC3BAD.4010309@behnel.de>	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>	<46DCF68A.1010904@behnel.de>	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de>
Message-ID: <46DDB0F9.5090900@behnel.de>


Stefan Behnel wrote:
> Phillip J. Eby wrote:
>> The key design question I ask for setuptools features is "who gets the
>> pain?", since nearly all design decisions will cause *somebody* pain. 
>> Ideally, I want that pain to go to somebody who's in a position to do
>> something that improves things for everybody in the long term.
>>
>> In this case, the only additional pain is for the developers of new
>> tools that process .pyx files (such as yourself).
> 
> Wrong. I cannot fix this as a developer of Cython - at least not without
> providing a fake build_ext.py in a fake Pyrex package, which would make
> installing Pyrex and Cython on the same machine near impossible and which
> would break every tool that requires Pyrex instead on Cython.

Ok, admittedly, I could change sys.path to point it to an internal directory
of the installed Cython distribution, so that setuptools would be tricked into
succeeding with a fake Pyrex import - but how ugly is that?

The right way of fixing this is definitely in setuptools, and the quick fix is
to extend the test import of Pyrex.Distutils.build_ext.build_ext with an
additional test for Cython.Distutils.build_ext.build_ext in the failure case -
if you don't feel like taking the cleaner step of checking for a "build_ext"
replacement...

Stefan

From pje at telecommunity.com  Tue Sep  4 21:52:49 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Sep 2007 15:52:49 -0400
Subject: [Distutils] my wishlist for easy_install
In-Reply-To: <20070829083746.20641.qmail@oak.oeko.net>
References: <20070829083746.20641.qmail@oak.oeko.net>
Message-ID: <20070904195021.8A9473A4045@sparrow.telecommunity.com>

At 10:37 AM 8/29/2007 +0200, Toni Mueller wrote:
>I'd really like to see these two features in easy_install which I can't
>find right now:
>
>  1 - coexist with the operating system/recognize packages which are
>      installed via operating system means
>  2 - remove packages
>
>Currently, when I want to easy_install a package, it starts downloading
>all sorts of stuff from PYPI although I have already similar (the same
>- maybe other versions) packages in my system which I installed via
>the operating system's package manager.

As of Python 2.5, packages installed using the distutils will have 
.egg-info files that tell setuptools they are installed...  assuming 
that the OS distributor doesn't remove them.

If you have a version of Python <2.5, you will need to either 
manually create .egg-info files or else live with easy_install 
installing duplicates.


>I'd also be able to say something like
>
>easy_install -r package
>
>to remove such a package, but only if I installed it via easy_install
>in the first place, leaving operating system installed packages alone.

This is a feature that may appear in setuptools 0.7 some time in the 
future, but it probably won't be an easy_install option (as 
easy_install already has way too many options).


From pje at telecommunity.com  Tue Sep  4 21:57:15 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Sep 2007 15:57:15 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DDB0F9.5090900@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
Message-ID: <20070904195447.55A3D3A40F0@sparrow.telecommunity.com>

At 09:24 PM 9/4/2007 +0200, Stefan Behnel wrote:

>Stefan Behnel wrote:
> > Phillip J. Eby wrote:
> >> The key design question I ask for setuptools features is "who gets the
> >> pain?", since nearly all design decisions will cause *somebody* pain.
> >> Ideally, I want that pain to go to somebody who's in a position to do
> >> something that improves things for everybody in the long term.
> >>
> >> In this case, the only additional pain is for the developers of new
> >> tools that process .pyx files (such as yourself).
> >
> > Wrong. I cannot fix this as a developer of Cython - at least not without
> > providing a fake build_ext.py in a fake Pyrex package, which would make
> > installing Pyrex and Cython on the same machine near impossible and which
> > would break every tool that requires Pyrex instead on Cython.
>
>Ok, admittedly, I could change sys.path to point it to an internal directory
>of the installed Cython distribution, so that setuptools would be tricked into
>succeeding with a fake Pyrex import - but how ugly is that?

Isn't Cython a *replacement* for Pyrex?  If not, then why does it use 
.pyx files?  If it is, then why is installing both a problem?  (It's 
not like you can't switch back and forth between them.)


>The right way of fixing this is definitely in setuptools, and the quick fix is
>to extend the test import of Pyrex.Distutils.build_ext.build_ext with an
>additional test for Cython.Distutils.build_ext.build_ext in the failure case -
>if you don't feel like taking the cleaner step of checking for a "build_ext"
>replacement...

Perhaps you'd care to produce a patch to implement that "cleaner 
step"?  It's not at all obvious to me how to do that without 
introducing instability that would be unsuitable for an 0.6cN release.


From pje at telecommunity.com  Tue Sep  4 22:04:36 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Sep 2007 16:04:36 -0400
Subject: [Distutils] Why are egg-info and related pth files required for
 rpm packages?
In-Reply-To: <21106.207.188.248.157.1188923967.squirrel@www.cpcug.org>
References: <21106.207.188.248.157.1188923967.squirrel@www.cpcug.org>
Message-ID: <20070904200208.4A2963A40F0@sparrow.telecommunity.com>

At 12:39 PM 9/4/2007 -0400, Stanley A. Klein wrote:
>I recently installed Fedora 7 and looked at
>/usr/lib/python2.5/site-packages.  The directory has numerous Python
>packages installed but no egg-info and few .pth files.  Of the three .pth
>files in my installation, only one has a content different from the name
>of the package (pointing to a subdirectory of site-packages with a
>different name).
>
>Setuptools has the advantage of simplifying preparation of rpms because it
>has find_packages().  However, it produces egg-info and .pth files as part
>of the rpm's.
>
>I can understand the need for these files when packaging into eggs for
>installation in places other than python/site-packages or when packaging
>for non-Linux operating systems.  But why are they needed when installing
>as rpms on Linux systems into site-packages?
>
>I think this issue applies to most Linux packaging, which is usually
>either rpm or deb.

The .egg-info files or directories are required in order to contain 
project-level metadata.  Without them, the fact that the packages are 
installed can't be detected in a cross-platform way, and other 
projects *simply won't function without them*.  For example, Chandler 
i18n support is provided via the contents of .egg-info, as is plugin 
registration.  A wide range of plugin-supporting libraries and tools 
absolutely require .egg-info, including TurboGears, Buffet, Trac, 
Chandler and setuptools itself.  Setuptools can't even build itself 
unless its own .egg-info is present!

So .egg-info is absolutely indispensable, regardless of installation method.

As for .pth files, the only .pth files that should be generated by 
setuptools are the ones needed to support namespace packages.  When 
you have multiple projects spanning a namespace package, each of 
those projects would contain "somepackage/__init__.py" in its naive 
layout.  But this would cause conflicts between the RPMs, so 
setuptools uses uniquely-named .pth files to work around the absence 
of an __init__.py.  So, these "Project-version-nspkg.pth" files are 
also indispensable, as the packages involved won't be importable without them.

However, the .pth files you described don't sound like ones generated 
by setuptools.

Note, by the way, that as of Python 2.5, *all* distutils-generated 
packages include .egg-info; they just use a single file instead of a 
directory.  This makes it easy to detect what Python packages are 
installed on a system, as long as the platform maintainers don't 
remove this file.


From a.badger at gmail.com  Tue Sep  4 23:04:46 2007
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Tue, 04 Sep 2007 14:04:46 -0700
Subject: [Distutils] Why are egg-info and related pth files required for
 rpm packages?
In-Reply-To: <20070904200208.4A2963A40F0@sparrow.telecommunity.com>
References: <21106.207.188.248.157.1188923967.squirrel@www.cpcug.org>
	<20070904200208.4A2963A40F0@sparrow.telecommunity.com>
Message-ID: <46DDC86E.1030908@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 12:39 PM 9/4/2007 -0400, Stanley A. Klein wrote:
>> I recently installed Fedora 7 and looked at
>> /usr/lib/python2.5/site-packages.  The directory has numerous Python
>> packages installed but no egg-info and few .pth files.  Of the three .pth
>> files in my installation, only one has a content different from the name
>> of the package (pointing to a subdirectory of site-packages with a
>> different name).
>>
I have some egg-info files in my site-packages.  Those are from
setuptools using packages.  I noticed the  seemingly useless .pth files
the other day and am puzzled by their presence.  More info on those
later in this post.

[...]

> The .egg-info files or directories are required in order to contain 
> project-level metadata.
[...]
> So .egg-info is absolutely indispensable, regardless of installation method.
> 
Absolutely agreed.

> As for .pth files, the only .pth files that should be generated by 
> setuptools are the ones needed to support namespace packages.  When 
> you have multiple projects spanning a namespace package, each of 
> those projects would contain "somepackage/__init__.py" in its naive 
> layout.  But this would cause conflicts between the RPMs, so 
> setuptools uses uniquely-named .pth files to work around the absence 
> of an __init__.py.  So, these "Project-version-nspkg.pth" files are 
> also indispensable, as the packages involved won't be importable without them.
> 
> However, the .pth files you described don't sound like ones generated 
> by setuptools.
>
I looked into this briefly when attempting to get rid of .pth's and eggs
to diagnose the earlier bug (thanks again for the quick patch and
release!)  The packages I've looked at so far are all being generated by
distutils and have a C component.  I haven't had a chance to delve deeper.

> Note, by the way, that as of Python 2.5, *all* distutils-generated 
> packages include .egg-info; they just use a single file instead of a 
> directory.  This makes it easy to detect what Python packages are 
> installed on a system, as long as the platform maintainers don't 
> remove this file.
>
I'm sorry to say that this is not true on Fedora 7's python2.5.  There's
a patch that disables generating egg-info files for distutils.  I've
started talking with the python maintainer in Fedora to find out why the
patch exists and if it can be removed but he needs some time to find out
why the patch was added in the first place.

(A note in the spec files implies that the patch was added so as not to
generate egg-info for python core libraries and it might not have been
meant to affect distutils as a whole.  I have to figure out if even that
level of meddling is going to prove bothersome and make a
recommendation.  If you can think of some cases where that would be bad,
please reply so that I can include them in our discussion.)

- -Toshio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFG3chuX6yAic2E7kgRAs5nAJwPYy3Qh5udkOujX6Hz5VoemUyoOACcDwxe
vJLPiVHaB38bbhRrvw0j/+c=
=sXCC
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Tue Sep  4 23:25:14 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Sep 2007 17:25:14 -0400
Subject: [Distutils] Why are egg-info and related pth files required for
 rpm packages?
In-Reply-To: <46DDC86E.1030908@gmail.com>
References: <21106.207.188.248.157.1188923967.squirrel@www.cpcug.org>
	<20070904200208.4A2963A40F0@sparrow.telecommunity.com>
	<46DDC86E.1030908@gmail.com>
Message-ID: <20070904212246.C6B0A3A4045@sparrow.telecommunity.com>

At 02:04 PM 9/4/2007 -0700, Toshio Kuratomi wrote:
> > Note, by the way, that as of Python 2.5, *all* distutils-generated
> > packages include .egg-info; they just use a single file instead of a
> > directory.  This makes it easy to detect what Python packages are
> > installed on a system, as long as the platform maintainers don't
> > remove this file.
> >
>I'm sorry to say that this is not true on Fedora 7's python2.5.

Well, as I said, "as long as the platform maintainers don't remove" 
it.  :)  I didn't know of anybody specifically removing it, but given 
the long and colorful history of distributions hacking up the way 
Python is installed, I expected that *somebody* would be unable to 
resist tinkering.  :)


>   There's
>a patch that disables generating egg-info files for distutils.  I've
>started talking with the python maintainer in Fedora to find out why the
>patch exists and if it can be removed but he needs some time to find out
>why the patch was added in the first place.
>
>(A note in the spec files implies that the patch was added so as not to
>generate egg-info for python core libraries and it might not have been
>meant to affect distutils as a whole.  I have to figure out if even that
>level of meddling is going to prove bothersome and make a
>recommendation.  If you can think of some cases where that would be bad,
>please reply so that I can include them in our discussion.)

If you mean the Python-2.5.egg-info and wsgiref.egg-info, they are 
both definitely supposed to be there.

The latter is there so that packages that depend on a specific 
version of wsgiref can do so without needing to check what Python 
version they're on.

I really would have liked to include .egg-info for *all* the 2.5 
stdlib add-ons including ctypes and ElementTree, but there wasn't 
time to co-ordinate that before the release.


From stefan_ml at behnel.de  Wed Sep  5 08:30:43 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 05 Sep 2007 08:30:43 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
Message-ID: <46DE4D13.2040909@behnel.de>


Phillip J. Eby wrote:
> Isn't Cython a *replacement* for Pyrex?

Sure, but there's some Pyrex code that Cython deliberately does not compile.
However, the biggest problem is that most setup.py scripts simply do not use
Cython and require Pyrex instead. So users *will* install both.


> If it is, then why is installing both a problem?  (It's not
> like you can't switch back and forth between them.)

I might have said it before: this is about supporting setuptools, not about
supporting Cython. Supporting Cython is easy: uninstall setuptools, done once
and for all. Much less work than installing the right compiler *each time* you
want to install a new version of X or want to work on Y.


>> The right way of fixing this is definitely in setuptools, and the
>> quick fix is
>> to extend the test import of Pyrex.Distutils.build_ext.build_ext with an
>> additional test for Cython.Distutils.build_ext.build_ext in the
>> failure case -
>> if you don't feel like taking the cleaner step of checking for a
>> "build_ext"
>> replacement...
> 
> Perhaps you'd care to produce a patch to implement that "cleaner step"? 
> It's not at all obvious to me how to do that without introducing
> instability that would be unsuitable for an 0.6cN release.

Ok, testing for Cython is as easy as testing for Pyrex, so here's the obvious
patch against extension.py that should have gone into 0.6c7.

One way of implementing the above change would be to move the replacement code
into build_ext rather than Extension. Something like the (untested)
build_ext-patcher.py I attached. Note the type check that tests for build_ext
being subclassed.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: extension.py-check-for-cython.patch
Type: text/x-diff
Size: 485 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070905/a28a9a4e/attachment.patch 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_ext-patcher.py
Type: text/x-python
Size: 691 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070905/a28a9a4e/attachment.py 

From miles at jamkit.com  Wed Sep  5 18:14:24 2007
From: miles at jamkit.com (Miles)
Date: Wed, 05 Sep 2007 17:14:24 +0100
Subject: [Distutils] zc.recipe.cmmi
Message-ID: <fbml4s$r8l$1@sea.gmane.org>

Hi,

I've created a branch for a change to this recipe to support the ability 
to download patches via http rather than just using what's on the 
filesystem.

http://svn.zope.org/zc.recipe.cmmi/branches/miwa-patch-url/

We use this to keep common configuration tasks and patches as a "base" 
config (in one place) that specific projects can extend.

Any thoughts at all?  I plan to merge this in at the end of the week.

Thanks,

M


From chrism at plope.com  Wed Sep  5 19:03:51 2007
From: chrism at plope.com (Chris McDonough)
Date: Wed, 5 Sep 2007 13:03:51 -0400
Subject: [Distutils] profiling setuptools console scripts
Message-ID: <ACA94C52-B012-42D5-9676-2BE941A57776@plope.com>

This is very minor, but I recently ran across a case where I wanted  
to profile some code invoked that is invoked by a setuptools  
console_script.  The generated script is in the form:

sys.exit(
    load_entry_point('supervisor==3.0a2', 'console_scripts',  
'supervisord')()
)

Due to the explicit sys.exit() call, I could not (or at least could  
not figure out how to) place the profiling hair in the invoked code.   
I wound up having to change the script to look like this:

cmd = "load_entry_point 
('supervisor==3.0a2','console_scripts','supervisord')()"

if os.environ.has_key('SUPERVISOR_PROFILE'):
     import profile
     import pstats
     profile.runctx(cmd, globals(), locals(), '/tmp/superprofile')
     stats = pstats.Stats('/tmp/superprofile')
     stats.strip_dirs()
     stats.sort_stats('cumulative', 'calls', 'time')
     stats.print_stats(.3)
else:
     sys.exit(eval(cmd))

I'm wondering if the default generated console_script should refrain  
from explicitly calling sys.exit.

What a wonderful thing setuptools is, seriously.  I'm really enjoying  
using it.

- C


From pje at telecommunity.com  Wed Sep  5 19:49:52 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Sep 2007 13:49:52 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DE4D13.2040909@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
Message-ID: <20070905174724.6A1843A4045@sparrow.telecommunity.com>

At 08:30 AM 9/5/2007 +0200, Stefan Behnel wrote:
> > Perhaps you'd care to produce a patch to implement that "cleaner step"?
> > It's not at all obvious to me how to do that without introducing
> > instability that would be unsuitable for an 0.6cN release.
>
>One way of implementing the above change would be to move the replacement code
>into build_ext rather than Extension. Something like the (untested)
>build_ext-patcher.py I attached. Note the type check that tests for build_ext
>being subclassed.

You're illustrating my point.  It's easy to hand-wave about how it 
should be done, but not so easy to actually *do*.  Did you look at 
where all the .sources attribute gets used?  How the build_ext 
command can get called by other commands?

You're also ignoring my larger point: the current mechanism allows 
you to write setup scripts that *don't* need to subclass 
build_ext.  A setuptools-based setup script just refers to '.pyx' 
files, and everything else happens automatically.

And if you need to be able to distinguish between Cython-specific and 
Pyrex-specific files, why are you using the same file extension?


From pje at telecommunity.com  Wed Sep  5 19:52:34 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Sep 2007 13:52:34 -0400
Subject: [Distutils] profiling setuptools console scripts
In-Reply-To: <ACA94C52-B012-42D5-9676-2BE941A57776@plope.com>
References: <ACA94C52-B012-42D5-9676-2BE941A57776@plope.com>
Message-ID: <20070905175010.3A1A13A4045@sparrow.telecommunity.com>

At 01:03 PM 9/5/2007 -0400, Chris McDonough wrote:
>This is very minor, but I recently ran across a case where I wanted
>to profile some code invoked that is invoked by a setuptools
>console_script.  The generated script is in the form:
>
>sys.exit(
>     load_entry_point('supervisor==3.0a2', 'console_scripts',
>'supervisord')()
>)
>
>Due to the explicit sys.exit() call, I could not (or at least could
>not figure out how to) place the profiling hair in the invoked code.

Why not just change the entry point to refer to your profiling function? 


From stefan_ml at behnel.de  Wed Sep  5 20:00:35 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 05 Sep 2007 20:00:35 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070905174724.6A1843A4045@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
Message-ID: <46DEEEC3.9090208@behnel.de>


Phillip J. Eby wrote:
> At 08:30 AM 9/5/2007 +0200, Stefan Behnel wrote:
>> > Perhaps you'd care to produce a patch to implement that "cleaner step"?
>> > It's not at all obvious to me how to do that without introducing
>> > instability that would be unsuitable for an 0.6cN release.
>>
>> One way of implementing the above change would be to move the
>> replacement code
>> into build_ext rather than Extension. Something like the (untested)
>> build_ext-patcher.py I attached. Note the type check that tests for
>> build_ext
>> being subclassed.
> 
> You're illustrating my point.  It's easy to hand-wave about how it
> should be done, but not so easy to actually *do*.  Did you look at where
> all the .sources attribute gets used?  How the build_ext command can get
> called by other commands?

Sort of. Do you doubt it should work that way?


> You're also ignoring my larger point: the current mechanism allows you
> to write setup scripts that *don't* need to subclass build_ext.  A
> setuptools-based setup script just refers to '.pyx' files, and
> everything else happens automatically.

The script I posted doesn't change that. Read it.


> And if you need to be able to distinguish between Cython-specific and
> Pyrex-specific files, why are you using the same file extension?

We don't distinguish, it's the same language. But users will have to install
both of them to satisfy the requirements of the software they install, so
there are limits to what the Cython developers can do on their side.

You're actually forgetting the point you stressed most. If we prevent users
from installing Cython and Pyrex at the same time, they will have to start
modifying setup.py scripts. So it's pushing the burden on them.

Stefan

From pje at telecommunity.com  Wed Sep  5 20:46:47 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Sep 2007 14:46:47 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DEEEC3.9090208@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
	<46DEEEC3.9090208@behnel.de>
Message-ID: <20070905184419.914C63A4045@sparrow.telecommunity.com>

At 08:00 PM 9/5/2007 +0200, Stefan Behnel wrote:

>Phillip J. Eby wrote:
> > At 08:30 AM 9/5/2007 +0200, Stefan Behnel wrote:
> >> > Perhaps you'd care to produce a patch to implement that "cleaner step"?
> >> > It's not at all obvious to me how to do that without introducing
> >> > instability that would be unsuitable for an 0.6cN release.
> >>
> >> One way of implementing the above change would be to move the
> >> replacement code
> >> into build_ext rather than Extension. Something like the (untested)
> >> build_ext-patcher.py I attached. Note the type check that tests for
> >> build_ext
> >> being subclassed.
> >
> > You're illustrating my point.  It's easy to hand-wave about how it
> > should be done, but not so easy to actually *do*.  Did you look at where
> > all the .sources attribute gets used?  How the build_ext command can get
> > called by other commands?
>
>Sort of. Do you doubt it should work that way?

I know that other commands use the .sources of build_ext indirectly, 
at a time when run() has not yet been called.  That means that your 
approach may change those other commands' behavior in non-obvious 
ways.  That's why I'm not going to do something like that in an 0.6cN release.


> > You're also ignoring my larger point: the current mechanism allows you
> > to write setup scripts that *don't* need to subclass build_ext.  A
> > setuptools-based setup script just refers to '.pyx' files, and
> > everything else happens automatically.
>
>The script I posted doesn't change that. Read it.

Actually, now that I've read it in detail, I see that it's also 
broken in other ways, such as confusing modules and classes.

But it's also broken in the way I said: it *requires* subclassing of 
build_ext in setup.py, which the current mechanism does not.



> > And if you need to be able to distinguish between Cython-specific and
> > Pyrex-specific files, why are you using the same file extension?
>
>We don't distinguish, it's the same language. But users will have to install
>both of them

If it's the "same language" then, *why* do they need this?  If Pyrex 
is still required, then clearly it's *not* the "same language".


>You're actually forgetting the point you stressed most. If we prevent users
>from installing Cython and Pyrex at the same time, they will have to start
>modifying setup.py scripts. So it's pushing the burden on them.

Actually, I think you need to decide whether you are really providing 
an extended Pyrex compiler.  If so, then use .pyx as the extension 
and include a Pyrex-compatible build_ext module, and then there is no 
need to have both installed on the same system, and everything works 
splendidly for everyone.

If on the other hand you are *not* providing a Pyrex replacement and 
must co-exist, then use a different file extension (.cy, perhaps?) 
and you get exactly the behavior you wanted -- i.e., setuptools 
doesn't touch your files, and explicit action is required to change 
the file extensions.

And there is still a third solution: provide your own Extension class 
for users to import, since they will have to import from Cython in 
order to use your extended build_ext class anyway, nothing stops them 
from importing the Extension type too.  (Just have your 
Extension.__init__ save the sources and restore them after calling 
the superclass; it'll then work whether the patched version is present or not.)



From stefan_ml at behnel.de  Wed Sep  5 21:49:36 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 05 Sep 2007 21:49:36 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070905184419.914C63A4045@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
	<46DEEEC3.9090208@behnel.de>
	<20070905184419.914C63A4045@sparrow.telecommunity.com>
Message-ID: <46DF0850.1040506@behnel.de>


Phillip J. Eby wrote:
> At 08:00 PM 9/5/2007 +0200, Stefan Behnel wrote:
> 
>> Phillip J. Eby wrote:
>> > At 08:30 AM 9/5/2007 +0200, Stefan Behnel wrote:
>> >> > Perhaps you'd care to produce a patch to implement that "cleaner
>> step"?
>> >> > It's not at all obvious to me how to do that without introducing
>> >> > instability that would be unsuitable for an 0.6cN release.
>> >>
>> >> One way of implementing the above change would be to move the
>> >> replacement code
>> >> into build_ext rather than Extension. Something like the (untested)
>> >> build_ext-patcher.py I attached. Note the type check that tests for
>> >> build_ext
>> >> being subclassed.
>> >
>> > You're illustrating my point.  It's easy to hand-wave about how it
>> > should be done, but not so easy to actually *do*.  Did you look at
>> where
>> > all the .sources attribute gets used?  How the build_ext command can
>> get
>> > called by other commands?
>>
>> Sort of. Do you doubt it should work that way?
> 
> I know that other commands use the .sources of build_ext indirectly, at
> a time when run() has not yet been called.  That means that your
> approach may change those other commands' behavior in non-obvious ways. 

I bet there's a method that you can safely override to do what I meant. Might
take me a few minutes longer to find than you, but I actually doubt I
currently want to invest that. You shouldn't forget that it's your code that's
broken.


>> > You're also ignoring my larger point: the current mechanism allows you
>> > to write setup scripts that *don't* need to subclass build_ext.  A
>> > setuptools-based setup script just refers to '.pyx' files, and
>> > everything else happens automatically.
>>
>> The script I posted doesn't change that. Read it.
> 
> Actually, now that I've read it in detail, I see that it's also broken
> in other ways, such as confusing modules and classes.

I guess you meant the sys.modules bit. That should be easy to fix. It's
untested, as I said. It was just meant to make clear what I was proposing.


> But it's also broken in the way I said: it *requires* subclassing of
> build_ext in setup.py, which the current mechanism does not.

No, it just prevents .pyx file mangling *if* you have subclassed build_ext.
"If" meaning: "in the case that", so it's a thing that may be, but doesn't
have to be. Not very close to the definition of "requires".

*If* build_ext was subclassed, doing nothing is a safe bet, as a subclass can
do whatever it likes and should not suffer from setuptools patching stuff it
might not even know about. If you do not subclass, it behaves as before
(probably minus some problems, as you suggested above).


>> You're actually forgetting the point you stressed most. If we prevent
>> users
>> from installing Cython and Pyrex at the same time, they will have to
>> start
>> modifying setup.py scripts. So it's pushing the burden on them.
> 
> Actually, I think you need to decide whether you are really providing an
> extended Pyrex compiler.  If so, then use .pyx as the extension and
> include a Pyrex-compatible build_ext module, and then there is no need
> to have both installed on the same system, and everything works
> splendidly for everyone.
> 
> If on the other hand you are *not* providing a Pyrex replacement and
> must co-exist, then use a different file extension (.cy, perhaps?) and
> you get exactly the behavior you wanted -- i.e., setuptools doesn't
> touch your files, and explicit action is required to change the file
> extensions.
> 
> And there is still a third solution: provide your own Extension class
> for users to import, since they will have to import from Cython in order
> to use your extended build_ext class anyway, nothing stops them from
> importing the Extension type too.  (Just have your Extension.__init__
> save the sources and restore them after calling the superclass; it'll
> then work whether the patched version is present or not.)

Patch wars, eh? :)

You're still forgetting that that pushes the burden on each writer of a
setup.py script. They have to take caution to support Pyrex *and* Cython (*if*
they want to, which is currently up to them), and they have to care about the
case where none of them is importable to get a build_ext replacement, which is
required by both Pyrex and Cython. I still can't see the case where your
extension.py patching becomes truly beneficial to anyone. It would be if
people really had to do nothing, but that's not the case.

But maybe we should just start shipping Cython with a fake Pyrex that people
can import and still get Cython instead. Although that doesn't really fix the
bug in setuptools. But then, that's your code.

Stefan

From pje at telecommunity.com  Wed Sep  5 22:10:12 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Sep 2007 16:10:12 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DF0850.1040506@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
	<46DEEEC3.9090208@behnel.de>
	<20070905184419.914C63A4045@sparrow.telecommunity.com>
	<46DF0850.1040506@behnel.de>
Message-ID: <20070905200743.031A63A4045@sparrow.telecommunity.com>

At 09:49 PM 9/5/2007 +0200, Stefan Behnel wrote:
>get a build_ext replacement, which is
>required by both Pyrex and Cython.

No, it isn't.  A setuptools-based project does not need to import 
*anything* from Pyrex; it just declares a setuptools Extension() with 
.pyx sources.

Further, if Cython includes a Pyrex-replacing build_ext in the same 
module location, it's not necessary for Cython either; setuptools 
will simply call the Cython one.

That's why I'm saying that if Cython is actually a Pyrex 
*replacement*, it should work just fine, with no need for anybody to 
change anything.

Conversely, if Cython is *not* a replacement for Pyrex, then it 
should use some other file extension and thus make the choice of tool explicit.

My initial understanding was that Cython was a Pyrex 
replacement...  which led me to wonder what all the fuss was 
about.  But this assumption about replacing build_ext being necessary 
is clearly a big part of what's leading us to different conclusions.


From stefan_ml at behnel.de  Wed Sep  5 22:21:29 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 05 Sep 2007 22:21:29 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070905200743.031A63A4045@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
	<46DEEEC3.9090208@behnel.de>
	<20070905184419.914C63A4045@sparrow.telecommunity.com>
	<46DF0850.1040506@behnel.de>
	<20070905200743.031A63A4045@sparrow.telecommunity.com>
Message-ID: <46DF0FC9.90500@behnel.de>


Phillip J. Eby wrote:
> At 09:49 PM 9/5/2007 +0200, Stefan Behnel wrote:
>> get a build_ext replacement, which is
>> required by both Pyrex and Cython.
> 
> No, it isn't.  A setuptools-based project does not need to import
> *anything* from Pyrex; it just declares a setuptools Extension() with
> .pyx sources.
[...]
> But this assumption
> about replacing build_ext being necessary is clearly a big part of
> what's leading us to different conclusions.

Ah, I missed that part. So setuptools special cases Pyrex in other places,
too, thanks for pointing that out.

Maybe I should really consider adding a Pyrex replacement to Cython... but
that needs to be discussed on the cython-dev list.

I still wouldn't mind having a few additional if-Pyrex-import-fails-try-Cython
lines in setuptools for now...

Thanks,
Stefan

From pje at telecommunity.com  Wed Sep  5 22:43:16 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Sep 2007 16:43:16 -0400
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <46DF0FC9.90500@behnel.de>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
	<46DEEEC3.9090208@behnel.de>
	<20070905184419.914C63A4045@sparrow.telecommunity.com>
	<46DF0850.1040506@behnel.de>
	<20070905200743.031A63A4045@sparrow.telecommunity.com>
	<46DF0FC9.90500@behnel.de>
Message-ID: <20070905204049.68C423A4045@sparrow.telecommunity.com>

At 10:21 PM 9/5/2007 +0200, Stefan Behnel wrote:

>Phillip J. Eby wrote:
> > At 09:49 PM 9/5/2007 +0200, Stefan Behnel wrote:
> >> get a build_ext replacement, which is
> >> required by both Pyrex and Cython.
> >
> > No, it isn't.  A setuptools-based project does not need to import
> > *anything* from Pyrex; it just declares a setuptools Extension() with
> > .pyx sources.
>[...]
> > But this assumption
> > about replacing build_ext being necessary is clearly a big part of
> > what's leading us to different conclusions.
>
>Ah, I missed that part. So setuptools special cases Pyrex in other places,
>too, thanks for pointing that out.

 From the docs:

Distributing Extensions compiled with Pyrex
-------------------------------------------

``setuptools`` includes transparent support for building Pyrex extensions, as
long as you define your extensions using ``setuptools.Extension``, *not*
``distutils.Extension``.  You must also not import anything from Pyrex in
your setup script.

If you follow these rules, you can safely list ``.pyx`` files as the source
of your ``Extension`` objects in the setup script.  ``setuptools`` will detect
at build time whether Pyrex is installed or not.  If it is, then ``setuptools``
will use it.  If not, then ``setuptools`` will silently change the
``Extension`` objects to refer to the ``.c`` counterparts of the ``.pyx``
files, so that the normal distutils C compilation process will occur.

Of course, for this to work, your source distributions must include the C
code generated by Pyrex, as well as your original ``.pyx`` files.  This means
that you will probably want to include current ``.c`` files in your revision
control system, rebuilding them whenever you check changes in for the ``.pyx``
source files.  This will ensure that people tracking your project in CVS or
Subversion will be able to build it even if they don't have Pyrex installed,
and that your source releases will be similarly usable with or without Pyrex.


>Maybe I should really consider adding a Pyrex replacement to Cython... but
>that needs to be discussed on the cython-dev list.
>
>I still wouldn't mind having a few additional if-Pyrex-import-fails-try-Cython
>lines in setuptools for now...

Well, does that need to be done for the actual build_ext command, 
too?  (See setuptools.command.build_ext)


From chrism at plope.com  Wed Sep  5 22:50:23 2007
From: chrism at plope.com (Chris McDonough)
Date: Wed, 5 Sep 2007 16:50:23 -0400
Subject: [Distutils] profiling setuptools console scripts
In-Reply-To: <20070905175010.3A1A13A4045@sparrow.telecommunity.com>
References: <ACA94C52-B012-42D5-9676-2BE941A57776@plope.com>
	<20070905175010.3A1A13A4045@sparrow.telecommunity.com>
Message-ID: <EE0B6A44-1E31-4784-AD1D-441001C5F463@plope.com>

On Sep 5, 2007, at 1:52 PM, Phillip J. Eby wrote:

> At 01:03 PM 9/5/2007 -0400, Chris McDonough wrote:
>> This is very minor, but I recently ran across a case where I wanted
>> to profile some code invoked that is invoked by a setuptools
>> console_script.  The generated script is in the form:
>>
>> sys.exit(
>>     load_entry_point('supervisor==3.0a2', 'console_scripts',
>> 'supervisord')()
>> )
>>
>> Due to the explicit sys.exit() call, I could not (or at least could
>> not figure out how to) place the profiling hair in the invoked code.
>
> Why not just change the entry point to refer to your profiling  
> function?

That works great.

- C



From stefan_ml at behnel.de  Wed Sep  5 23:14:13 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 05 Sep 2007 23:14:13 +0200
Subject: [Distutils] setuptools special case Pyrex and break Cython
In-Reply-To: <20070905204049.68C423A4045@sparrow.telecommunity.com>
References: <46DBD18F.2000902@behnel.de>
	<20070903153913.CAAEC3A4045@sparrow.telecommunity.com>
	<46DC2FEB.2000203@behnel.de>
	<20070903161014.A31463A40A5@sparrow.telecommunity.com>
	<46DC349D.8050807@behnel.de>
	<20070903163824.66B8D3A40A5@sparrow.telecommunity.com>
	<46DC3BAD.4010309@behnel.de>
	<20070903231504.BB0A03A40A5@sparrow.telecommunity.com>
	<46DCF68A.1010904@behnel.de>
	<20070904164117.0A3D03A4045@sparrow.telecommunity.com>
	<46DDACC7.9080008@behnel.de> <46DDB0F9.5090900@behnel.de>
	<20070904195447.55A3D3A40F0@sparrow.telecommunity.com>
	<46DE4D13.2040909@behnel.de>
	<20070905174724.6A1843A4045@sparrow.telecommunity.com>
	<46DEEEC3.9090208@behnel.de>
	<20070905184419.914C63A4045@sparrow.telecommunity.com>
	<46DF0850.1040506@behnel.de>
	<20070905200743.031A63A4045@sparrow.telecommunity.com>
	<46DF0FC9.90500@behnel.de>
	<20070905204049.68C423A4045@sparrow.telecommunity.com>
Message-ID: <46DF1C25.70403@behnel.de>


Phillip J. Eby wrote:
> ``setuptools`` includes transparent support for building Pyrex
> extensions, as
> long as you define your extensions using ``setuptools.Extension``, *not*
> ``distutils.Extension``.  You must also not import anything from Pyrex in
> your setup script.

I did that with lxml a while ago and then switched back to special casing
various situations to avoid any hard dependency on setuptools. Forcing users
to install and use setuptools is nothing you really want as a distributor, so
you automatically end up doing these things by hand in your own script. Maybe
that's why I missed that part of the docs.


>> Maybe I should really consider adding a Pyrex replacement to Cython...
>> but that needs to be discussed on the cython-dev list.
>>
>> I still wouldn't mind having a few additional
>> if-Pyrex-import-fails-try-Cython
>> lines in setuptools for now...
> 
> Well, does that need to be done for the actual build_ext command, too? 
> (See setuptools.command.build_ext)

Cython currently uses the same setup as Pyrex, so the same procedure is
required here.

Stefan

From b3i4old02 at sneakemail.com  Wed Sep  5 19:31:35 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Wed, 05 Sep 2007 18:31:35 +0100
Subject: [Distutils] HTTP access to svn://svn.eby-sarna.com/svnroot/ez_setup
Message-ID: <fbmp67$bhm$1@sea.gmane.org>

Is there a way to get HTTP access to the directory available at 
<svn://svn.eby-sarna.com/svnroot/ez_setup>? I'd like to set 
svn:externals access to this directory, but one of the hosts I use is 
behind a firewall and can't use port 3690. It can access HTTP through a 
proxy.

I tried this:

$ svn co http://svn.eby-sarna.com/svnroot/ez_setup
svn: PROPFIND request failed on '/'
svn: PROPFIND of '/': 200 OK (http://svn.eby-sarna.com)

$ svn co http://svn.eby-sarna.com/ez_setup
svn: PROPFIND request failed on '/ez_setup'
svn: PROPFIND of '/ez_setup': 302 Moved Temporarily 
(http://svn.eby-sarna.com)
-- 
Michael Hoffman


From pje at telecommunity.com  Thu Sep  6 03:37:59 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Sep 2007 21:37:59 -0400
Subject: [Distutils] HTTP access to
 svn://svn.eby-sarna.com/svnroot/ez_setup
In-Reply-To: <fbmp67$bhm$1@sea.gmane.org>
References: <fbmp67$bhm$1@sea.gmane.org>
Message-ID: <20070906013531.4A7FF3A4045@sparrow.telecommunity.com>

At 06:31 PM 9/5/2007 +0100, Michael Hoffman wrote:
>Is there a way to get HTTP access to the directory available at
><svn://svn.eby-sarna.com/svnroot/ez_setup>?

No.


From tseaver at palladion.com  Thu Sep  6 17:33:36 2007
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 06 Sep 2007 11:33:36 -0400
Subject: [Distutils] HTTP access to
	svn://svn.eby-sarna.com/svnroot/ez_setup
In-Reply-To: <fbmp67$bhm$1@sea.gmane.org>
References: <fbmp67$bhm$1@sea.gmane.org>
Message-ID: <46E01DD0.5010003@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Hoffman wrote:
> Is there a way to get HTTP access to the directory available at 
> <svn://svn.eby-sarna.com/svnroot/ez_setup>? I'd like to set 
> svn:externals access to this directory, but one of the hosts I use is 
> behind a firewall and can't use port 3690. It can access HTTP through a 
> proxy.
> 
> I tried this:
> 
> $ svn co http://svn.eby-sarna.com/svnroot/ez_setup
> svn: PROPFIND request failed on '/'
> svn: PROPFIND of '/': 200 OK (http://svn.eby-sarna.com)
> 
> $ svn co http://svn.eby-sarna.com/ez_setup
> svn: PROPFIND request failed on '/ez_setup'
> svn: PROPFIND of '/ez_setup': 302 Moved Temporarily 
> (http://svn.eby-sarna.com)

Perhaps you (Michael) could mirror the repository using SVK?

  http://www.howtoforge.com/read_only_svn_mirror_with_svk


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4B3Q+gerLs4ltQ4RAiQzAKCK/a5YKnc9Tj+9BJOWcsQaYlytpgCgzEMr
LSs88STPYeG5lnGMOMmra4o=
=YWZZ
-----END PGP SIGNATURE-----


From tseaver at palladion.com  Thu Sep  6 17:33:36 2007
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 06 Sep 2007 11:33:36 -0400
Subject: [Distutils] HTTP access to
	svn://svn.eby-sarna.com/svnroot/ez_setup
In-Reply-To: <fbmp67$bhm$1@sea.gmane.org>
References: <fbmp67$bhm$1@sea.gmane.org>
Message-ID: <46E01DD0.5010003@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Hoffman wrote:
> Is there a way to get HTTP access to the directory available at 
> <svn://svn.eby-sarna.com/svnroot/ez_setup>? I'd like to set 
> svn:externals access to this directory, but one of the hosts I use is 
> behind a firewall and can't use port 3690. It can access HTTP through a 
> proxy.
> 
> I tried this:
> 
> $ svn co http://svn.eby-sarna.com/svnroot/ez_setup
> svn: PROPFIND request failed on '/'
> svn: PROPFIND of '/': 200 OK (http://svn.eby-sarna.com)
> 
> $ svn co http://svn.eby-sarna.com/ez_setup
> svn: PROPFIND request failed on '/ez_setup'
> svn: PROPFIND of '/ez_setup': 302 Moved Temporarily 
> (http://svn.eby-sarna.com)

Perhaps you (Michael) could mirror the repository using SVK?

  http://www.howtoforge.com/read_only_svn_mirror_with_svk


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4B3Q+gerLs4ltQ4RAiQzAKCK/a5YKnc9Tj+9BJOWcsQaYlytpgCgzEMr
LSs88STPYeG5lnGMOMmra4o=
=YWZZ
-----END PGP SIGNATURE-----

From ckkart at hoc.net  Fri Sep  7 03:15:54 2007
From: ckkart at hoc.net (Christian K.)
Date: Fri, 7 Sep 2007 01:15:54 +0000 (UTC)
Subject: [Distutils] errors compiling c++ extensions with swig on windows
Message-ID: <loom.20070907T031044-695@post.gmane.org>


Hi,

using the following setup file to build a extension  from c++ source on
windows/mingw with swig 1.3.31,

import distutils
from distutils.core import setup, Extension

ext = Extension("_libpyorigin",
                ["libpyorigin.i","OPJFile.cpp"],
                extra_compile_args = ['-w'])

setup(name = "libpyorigin",
      version = "0.1",
      options={'build_ext':{'swig_opts':'-c++'}},
      ext_modules = [ext])

I get the following error:

$ python setup.py build_ext --compiler=mingw32
running build_ext
building '_libpyorigin' extension
swigging libpyorigin.i to libpyorigin_wrap.cpp
C:\msys\1.0\local\bin\swig.exe -python -c++ -o libpyorigin_wrap.cpp libpyorigin.i
c:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -Ic:\Python25\include
-Ic:\Python25\PC -c libpyorigin_wrap.cpp -o
build\temp.win32-2.5\Release\libpyorigin_wrap.o -w
writing build\temp.win32-2.5\Release\_libpyorigin.def
c:\mingw\bin\g++.exe -mno-cygwin -mdll -static --entry _DllMain at 12 --output-lib
build\temp.win32-2.5\Release\lib_libpyorigin.a --def
build\temp.win32-2.5\Release\_libpyorigin.def -s
build\temp.win32-2.5\Release\libpyorigin_wrap.o
build\temp.win32-2.5\Release\opjfile.o -Lc:\Python25\libs -Lc:\Python25\PCBuild
-lpython25 -lmsvcr71 -o build\lib.win32-2.5\_libpyorigin.pyd
g++: build\temp.win32-2.5\Release\lib_libpyorigin.a: No such file or directory
error: command 'g++' failed with exit status 1

What is distutils trying to do with that lib_libpyorigin.a? I could not find the
meaning of --output-lib but I suspect g++ is trying to link with that lib but it
 was meant to build it, right? Is this a bug in distutils or did I do seomething
wrong?

The same works perfectly on linux, btw.

Thanks, Christian



From miles at jamkit.com  Sat Sep  8 19:04:03 2007
From: miles at jamkit.com (Miles)
Date: Sat, 08 Sep 2007 18:04:03 +0100
Subject: [Distutils] zc.recipe.cmmi
In-Reply-To: <fbml4s$r8l$1@sea.gmane.org>
References: <fbml4s$r8l$1@sea.gmane.org>
Message-ID: <fbul60$rai$1@sea.gmane.org>

Okay, this is now done.  If there could be an egg release sometime not 
too distant, that would be great!

Thanks,

Miles wrote:
> Hi,
> 
> I've created a branch for a change to this recipe to support the ability 
> to download patches via http rather than just using what's on the 
> filesystem.
> 
> http://svn.zope.org/zc.recipe.cmmi/branches/miwa-patch-url/
> 
> We use this to keep common configuration tasks and patches as a "base" 
> config (in one place) that specific projects can extend.
> 
> Any thoughts at all?  I plan to merge this in at the end of the week.


From ignas.mikalajunas at gmail.com  Tue Sep 11 19:33:47 2007
From: ignas.mikalajunas at gmail.com (Ignas Mikalajunas)
Date: Tue, 11 Sep 2007 20:33:47 +0300
Subject: [Distutils] What is the legal status of virtual-python.py
Message-ID: <cc0c36330709111033p30b3b1b6vaa1de06ad76da49d@mail.gmail.com>

 Hello, I would like to reuse pieces of code from
  http://peak.telecommunity.com/dist/virtual-python.py
in my "personal" edition of zc.buildout's bootstrap.py  (i actually
did that already). So running "python bootstrap.py" creates a virtual
python in the same directory, and uses that virtual (clean) python to
bootstrap zc.buildout. So i could use system python, and at the same
time avoid bugs related with linux distribution installed packages
conflicting with zc.buildout.

All is nice, but now i have no idea what the legal status of this
newly created bootstrap.py is. I mean it contains code that is under
ZPL (original bootstrap.py), and code that has no license at all and
was worked on by at least 2 authors - Phillip Eby and Ian Bicking.

Can I even use/distribute the newly created bootstrap.py?

Ignas Mikalaj?as

From granted14 at yahoo.com  Tue Sep 11 23:09:29 2007
From: granted14 at yahoo.com (Etienne Robillard)
Date: Tue, 11 Sep 2007 17:09:29 -0400 (EDT)
Subject: [Distutils] Problem adding a new setup command..
Message-ID: <819163.3074.qm@web55604.mail.re4.yahoo.com>


Hi there,

I'm a new setuptools user. I think setuptools is very
good, and I'm currently using it for a personal
project. [1]

However, I'm experimenting a small issue when trying
to use a new setup command like so:
 
 $ python setup.py install_media 
 Distribution.parse_config_files():
  reading setup.cfg
options (after parsing config files):
option dict for 'egg_info' command:
  {'tag_build': ('setup.cfg', 'p1'),
'tag_svn_revision': ('setup.cfg', '0')}
options (after parsing command line):
option dict for 'aliases' command:
  {}
option dict for 'egg_info' command:
  {'tag_build': ('setup.cfg', 'p1'),
'tag_svn_revision': ('setup.cfg', '0')}
option dict for 'install_media' command:
  {'args': ('command line', [])}
running install_media
Distribution.get_command_obj(): creating
'install_media' command object
  setting options for 'install_media' command:
    args = [] (from command line)
Traceback (most recent call last):
  File "setup.py", line 143, in <module>
    zip_safe=(sys.version >= 2.5)
  File
"/opt/python25/lib/python2.5/distutils/core.py", line
151, in setup
    dist.run_commands()
  File
"/opt/python25/lib/python2.5/distutils/dist.py", line
974, in run_commands
    self.run_command(cmd)
  File
"/opt/python25/lib/python2.5/distutils/dist.py", line
992, in run_command
    cmd_obj = self.get_command_obj(command)
  File
"/opt/python25/lib/python2.5/distutils/dist.py", line
879, in get_command_obj
    self._set_command_options(cmd_obj, options)
  File
"/opt/python25/lib/python2.5/distutils/dist.py", line
919, in _set_command_options
    % (source, command_name, option))
distutils.errors.DistutilsOptionError: error in
command line: command 'install_media' has no such
option 'a     rgs'

Here's the setuptools.Command subclass:

from setuptools import Command
class install_media(Command):
    description = "Install app-specific media files"
    command_consumes_arguments = True
    # List of options for this command
    user_options = [
        # Perhaps the better option is to use the
'--prefix' arg.
        # ('with-docroot=', None, 'Install media files
to this directory'),
        ('with-apps=', None,
         "Search theses subdirs for media files
(comma-separated list)")
    ]
    def initialize_options(self):
        pass
    def finalize_options(self):
        pass
    def run(self):
        return self._find_media_files()
    def _find_media_files(self):
        pass


Any insightful hints how to debug this problem? 
The Big Idea was to add a new setup command (named
install_media) which could presumely allow things 
like:

 $ python setup.py install_media --prefix=/var/www
--with-apps=foo,bar


Thanks in advance,

Etienne

1. http://tools.assembla.com/notmm/


----
Etienne Robillard
7680 de jouvence, La Plaine
J7M-2K9, Qu?bec
Telephone: 450-478-5026
Yahoo Messenger ID: granted14
Skype ID: incidah


      Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com

From b3i4old02 at sneakemail.com  Wed Sep 12 20:07:00 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Wed, 12 Sep 2007 19:07:00 +0100
Subject: [Distutils] HTTP access to
	svn://svn.eby-sarna.com/svnroot/ez_setup
In-Reply-To: <46E01DD0.5010003@palladion.com>
References: <fbmp67$bhm$1@sea.gmane.org> <46E01DD0.5010003@palladion.com>
Message-ID: <fc99sj$a3j$1@sea.gmane.org>

Tres Seaver wrote:

>> Is there a way to get HTTP access to the directory available at 
>> <svn://svn.eby-sarna.com/svnroot/ez_setup>? I'd like to set 
>> svn:externals access to this directory, but one of the hosts I use is 
>> behind a firewall and can't use port 3690. It can access HTTP through a 
>> proxy.
 >
> Perhaps you (Michael) could mirror the repository using SVK?
> 
>   http://www.howtoforge.com/read_only_svn_mirror_with_svk

Thanks for the tip.


From zooko at zooko.com  Thu Sep 13 04:31:11 2007
From: zooko at zooko.com (zooko)
Date: Wed, 12 Sep 2007 19:31:11 -0700
Subject: [Distutils] How to easy_install from source package with mingw32 on
	Windows?
Message-ID: <A0A94EC5-9EE8-4E5D-811F-08B2E37A8240@zooko.com>

Folks:

Thank you for the various Python packaging tools!

I just published a new version of a Free Software package:

http://cheeseshop.python.org/pypi/zfec

And I tested it with "easy_install zfec" on Windows XP, and I was  
disappointed to see this error message:

"""
error: Setup script exited with error: Python was built with version  
7.1 of Visual Studio, and extensions need to be built with the same  
version of the compiler, but it isn't installed.
"""

If I download the zfec source tarball and execute

"""
./setup.py install
"""

Then I get the same error message, but if I execute

"""
./setup.py build -c mingw32 install
"""

Then it successfully builds and installs.

(This is on Python 2.4 -- I believe that in Python 2.5 that error  
message has been updated to mention that mingw is an option.)

So my first question is: how can I configure the zfec source package  
so that someone who has mingw installed and who runs "easy_install  
zfec" successfully builds it?

A deeper question is: wouldn't it be nice if "./setup.py install"  
automatically used the "build -c mingw32" feature when necessary?   
Would anyone be interested in extending distutils to do that?

Regards,

Zooko



From robert.kern at gmail.com  Thu Sep 13 04:54:57 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 12 Sep 2007 21:54:57 -0500
Subject: [Distutils] How to easy_install from source package with
	mingw32 on Windows?
In-Reply-To: <A0A94EC5-9EE8-4E5D-811F-08B2E37A8240@zooko.com>
References: <A0A94EC5-9EE8-4E5D-811F-08B2E37A8240@zooko.com>
Message-ID: <fca8qc$2ic$1@sea.gmane.org>

zooko wrote:

> A deeper question is: wouldn't it be nice if "./setup.py install"  
> automatically used the "build -c mingw32" feature when necessary?   
> Would anyone be interested in extending distutils to do that?

You can configure it to do so already. Making distutils automatically figure out
that this is desired and safe is probably more magic than it's worth.

  http://docs.python.org/inst/config-syntax.html

Use a section in your pydistutils.cfg file like this:

[build_ext]
compiler=mingw32

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From b3i4old02 at sneakemail.com  Thu Sep 13 11:30:45 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Thu, 13 Sep 2007 10:30:45 +0100
Subject: [Distutils] NumPy extensions and setup_requires
Message-ID: <fcb00n$uf1$1@sea.gmane.org>

I'm creating a package that uses NumPy to build an extension module. In 
order to do this, one must call numpy.get_include() like this:

ext_modules = [Extension("_fwd_bwd",
                          ["src/_fwd_bwd.pyx", "src/fwd_bwd.c"],
                          include_dirs=[numpy.get_include()])]

The problem is that if NumPy is not already installed, this will fail. I 
can't use setup_requires in a simple manner because the function call 
needs to happen before the call to setup().

The solution of creating another Distribution object to satisfy 
setup_requires before running setup() suggested in 
<http://mail.python.org/pipermail/distutils-sig/2006-October/006744.html> 
is unsatisfactory for the reasons also suggested there--if I do this 
--help will not work and it also seems a bit clunky.

I was thinking of instead making a subclass of Extension that will call 
populate include_dirs from a function rather than a list of strings. Is 
this the best solution?
-- 
Michael


From b3i4old02 at sneakemail.com  Thu Sep 13 12:17:48 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Thu, 13 Sep 2007 11:17:48 +0100
Subject: [Distutils] NumPy extensions and setup_requires
In-Reply-To: <fcb00n$uf1$1@sea.gmane.org>
References: <fcb00n$uf1$1@sea.gmane.org>
Message-ID: <fcb2ot$7qf$1@sea.gmane.org>

Michael Hoffman wrote:
> I'm creating a package that uses NumPy to build an extension module. In 
> order to do this, one must call numpy.get_include() like this:
> 
> ext_modules = [Extension("_fwd_bwd",
>                           ["src/_fwd_bwd.pyx", "src/fwd_bwd.c"],
>                           include_dirs=[numpy.get_include()])]
> 
> The problem is that if NumPy is not already installed, this will fail. I 
> can't use setup_requires in a simple manner because the function call 
> needs to happen before the call to setup().
> 
> The solution of creating another Distribution object to satisfy 
> setup_requires before running setup() suggested in 
> <http://mail.python.org/pipermail/distutils-sig/2006-October/006744.html> 
> is unsatisfactory for the reasons also suggested there--if I do this 
> --help will not work and it also seems a bit clunky.
> 
> I was thinking of instead making a subclass of Extension that will call 
> populate include_dirs from a function rather than a list of strings. Is 
> this the best solution?

The below seems to work partially. The problem is that you don't 
immediately get help when running setup.py --help. It tries to install 
numpy first, which isn't very friendly.

setuptools would also be more friendly if 
setuptools.dist.fetch_build_egg printed a message that it was about to 
go off and fetch software much like easy_install does. The first warning 
I get is "Running from numpy source directory." only after a long delay 
to download the source. PJE, would you consider adding such a message?

My NumpyExtension subclass:

"""
from setuptools import Extension, setup

class NumpyExtension(Extension):
     def __init__(self, *args, **kwargs):
         Extension.__init__(self, *args, **kwargs)

         self._include_dirs = self.include_dirs
         del self.include_dirs # restore overwritten property

     # warning: Extension is a classic class so it's not really read-only
     @property
     def include_dirs(self):
         from numpy import get_include

         return self._include_dirs + [get_include()]

ext_modules = [NumpyExtension("_fwd_bwd",
                               ["src/_fwd_bwd.pyx", "src/fwd_bwd.c"])]
"""

$ python -c "import pkg_resources; print 
pkg_resources.require('setuptools')"
[setuptools 0.6c7 
(/usr/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg)]


From zooko at zooko.com  Thu Sep 13 23:58:03 2007
From: zooko at zooko.com (zooko)
Date: Thu, 13 Sep 2007 14:58:03 -0700
Subject: [Distutils] How to easy_install from source package with
	mingw32 on Windows?
In-Reply-To: <fca8qc$2ic$1@sea.gmane.org>
References: <A0A94EC5-9EE8-4E5D-811F-08B2E37A8240@zooko.com>
	<fca8qc$2ic$1@sea.gmane.org>
Message-ID: <7F9BFCBC-F2BC-48DC-A1ED-1D83FE790E15@zooko.com>

(quoting out-of-order)

On Sep 12, 2007, at 7:54 PM, Robert Kern wrote:
> Use a section in your pydistutils.cfg file like this:
>
> [build_ext]
> compiler=mingw32

Thank you very much!


> You can configure it to do so already. Making distutils  
> automatically figure out
> that this is desired and safe is probably more magic than it's worth.

So why isn't the Free Software compiler the default when no compiler  
is specified?  It works fine.  If I understand, there is currently no  
gratis Microsoft compiler available that can be used to produce  
Python modules.

By the way, glancing at the python-dev archives, it appears that  
every time a new version of Python is ready to be released, people  
discover that Microsoft has stopped shipping the version of their  
compiler that was used for the previous version of Python, requiring  
everyone to agree about whether to upgrade to the newest Microsoft  
compiler, or to somehow acquire copies of the old, no-longer-shipped  
Microsoft compiler.  Occasionally, in these long, flamey threads on  
python-dev, one of the participants eventually says "Oh!  I've just  
learned that I can use the Free Software compiler to compile my  
Python modules, so I didn't need to spend all this time attempting to  
acquire an obsolete Microsoft compiler!".

If distutils used the Free Software compiler without requiring  
special configuration, it would be that much easier for folks to  
compile Python modules.

Regards,

Zooko

From robert.kern at gmail.com  Fri Sep 14 01:44:17 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 13 Sep 2007 18:44:17 -0500
Subject: [Distutils] How to easy_install from source package with
	mingw32 on Windows?
In-Reply-To: <7F9BFCBC-F2BC-48DC-A1ED-1D83FE790E15@zooko.com>
References: <A0A94EC5-9EE8-4E5D-811F-08B2E37A8240@zooko.com>	<fca8qc$2ic$1@sea.gmane.org>
	<7F9BFCBC-F2BC-48DC-A1ED-1D83FE790E15@zooko.com>
Message-ID: <fcci0t$blv$1@sea.gmane.org>

zooko wrote:
> (quoting out-of-order)
> 
> On Sep 12, 2007, at 7:54 PM, Robert Kern wrote:
>> Use a section in your pydistutils.cfg file like this:
>>
>> [build_ext]
>> compiler=mingw32
> 
> Thank you very much!
> 
>> You can configure it to do so already. Making distutils  
>> automatically figure out
>> that this is desired and safe is probably more magic than it's worth.
> 
> So why isn't the Free Software compiler the default when no compiler  
> is specified?  It works fine. 

Not always. mingw32 is still based on msvcrt6. You can make it link against
msvcr7 to match the runtime of Python, but many of the implementations of the
standard C headers are still intended for msvcrt6. C++ extensions using
std::iostreams are vulnerable to segfaults if you reconfigure mingw32's specs
file to only use msvcr7. Other extensions that pass FILE* pointers between them
and Python will segfault if you don't reconfigure mingw32. There is no
configuration of mingw32 that works for all extensions.

The default is the default because it's the only one that's known to be safe.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From stefan_ml at behnel.de  Sat Sep 15 08:30:04 2007
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 15 Sep 2007 08:30:04 +0200
Subject: [Distutils] easy_install should consider the development status of
	packages
Message-ID: <46EB7BEC.2040105@behnel.de>

Hi,

currently, lxml has both a stable release series (1.3.x) and an unstable alpha
release series (2.0alphaX) on PyPI. When you "easy_install lxml", you get the
2.0alpha version. I don't think that's what users expect and it's definitely
not what most users want, but at the same time I would like to keep 2.0
visible and easy_install-able to make people aware of it and to let them
decide by themselves.

I would like to have easy_install changed to either consider the Trove
development status of a package and prefer the highest one (or at least those
from 5 up), and/or to check the version string for the typical alpha/beta
substrings and ignore them by default. It would then be easy to add a command
line option like "--unstable" for those who know better.

Any comments on that?

Stefan

From zooko at zooko.com  Sat Sep 15 08:33:12 2007
From: zooko at zooko.com (zooko)
Date: Fri, 14 Sep 2007 23:33:12 -0700
Subject: [Distutils] patch: ez_setup.py: separate default version and
	minimum version of setuptools
Message-ID: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>

Folks:

Thank you for setuptools!  We are using it in the Allmydata-Tahoe  
project [1].  We've patched ez_setup.py so that installation can  
proceed if an older version of setuptools is already present, as long  
as that version is not too old.  This allows ez_setup.py to work  
seamlessly in more situations.  Please apply.

The patch is in-lined and attached.

Regards,

Zooko

[1] http://allmydata.org

Thu Sep 13 20:20:02 PDT 2007  zooko at zooko.com
   * change the 'ez_setup.py' script to have distinct desired &  
minimum required versions of setuptools

   Note that this preserves backwards compatibility -- if no minimum  
version is
   specified then the default version is used as the minimum version,  
which is
   identical to the behavior before this patch.

   This patch is originally due to Rob Kinninmont.

diff -rN -u old-setuptools/setuptools-0.6c7/ez_setup.py new- 
setuptools/setuptools-0.6c7/ez_setup.py
--- old-setuptools/setuptools-0.6c7/ez_setup.py 2007-09-13  
20:20:58.000000000 -0700
+++ new-setuptools/setuptools-0.6c7/ez_setup.py 2007-09-13  
20:20:58.000000000 -0700
@@ -61,7 +61,7 @@
def use_setuptools(
      version=DEFAULT_VERSION, download_base=DEFAULT_URL,  
to_dir=os.curdir,
-    download_delay=15
+    download_delay=15, min_version=None
):
      """Automatically find/download setuptools and make it available  
on sys.path
@@ -89,7 +89,9 @@
      import pkg_resources
      try:
-        pkg_resources.require("setuptools>="+version)
+        if not min_version:
+            min_version = version
+        pkg_resources.require("setuptools>="+min_version)
      except pkg_resources.VersionConflict, e:
          # XXX could we install in a subprocess here?
@@ -97,7 +99,7 @@
              "The required version of setuptools (>=%s) is not  
available, and\n"
              "can't be installed while this script is running.  
Please install\n"
              " a more recent version first.\n\n(Currently using %r)"
-        ) % (version, e.args[0])
+        ) % (min_version, e.args[0])
          sys.exit(2)
def download_setuptools(

-------------- next part --------------
A non-text attachment was scrubbed...
Name: x
Type: application/octet-stream
Size: 2138 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070914/40eca90a/attachment.obj 

From lxander.m at gmail.com  Sat Sep 15 13:18:52 2007
From: lxander.m at gmail.com (Alexander Michael)
Date: Sat, 15 Sep 2007 07:18:52 -0400
Subject: [Distutils] easy_install should consider the development status
	of packages
In-Reply-To: <46EB7BEC.2040105@behnel.de>
References: <46EB7BEC.2040105@behnel.de>
Message-ID: <525f23e80709150418j4316762au811371c046b96193@mail.gmail.com>

On 9/15/07, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> currently, lxml has both a stable release series (1.3.x) and an unstable alpha
> release series (2.0alphaX) on PyPI. When you "easy_install lxml", you get the
> 2.0alpha version. I don't think that's what users expect and it's definitely
> not what most users want, but at the same time I would like to keep 2.0
> visible and easy_install-able to make people aware of it and to let them
> decide by themselves.
>
> I would like to have easy_install changed to either consider the Trove
> development status of a package and prefer the highest one (or at least those
> from 5 up), and/or to check the version string for the typical alpha/beta
> substrings and ignore them by default. It would then be easy to add a command
> line option like "--unstable" for those who know better.
>
> Any comments on that?

A related desire of mine is to have both the latest stable release of
an egg and the latest development release installed with *the latest
stable release being the default* (i.e. listed, with all its stable
dependencies, in easy-install.pth). Something along the lines of an -M
option, which like -m, performs a multi-version install but doesn't
touch easy-install.pth. I'm not sure if this is feasible (or maybe it
is even already possible!), but it sounds nice to me. :)

Regards,
Alex

From ianb at colorstudy.com  Sat Sep 15 18:09:44 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Sat, 15 Sep 2007 11:09:44 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
Message-ID: <46EC03C8.9000604@colorstudy.com>

Hi all.  I'm kind of giving up on workingenv, and have started working 
from virtual-python as a basis instead 
(http://pypi.python.org/pypi/virtualenv/).

So the basic technique here is to copy the executable into 
/ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard 
Python installed on a Mac doesn't seem to do this -- the prefix remains 
'/opt/local/Library/Frameworks/Python.framework/Versions/2.4' 
regardless.  (Custom built Python's on Mac work like normal.)

Does anyone know a way to get the normal effect on this Mac Python?

-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From philipp at weitershausen.de  Sat Sep 15 18:30:51 2007
From: philipp at weitershausen.de (Philipp von Weitershausen)
Date: Sat, 15 Sep 2007 18:30:51 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46EC03C8.9000604@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
Message-ID: <170DC51A-E5E1-405A-AAC6-B10FA576E1D2@weitershausen.de>

On 15 Sep 2007, at 18:09 , Ian Bicking wrote:
> Hi all.  I'm kind of giving up on workingenv, and have started  
> working from virtual-python as a basis instead (http:// 
> pypi.python.org/pypi/virtualenv/).
>
> So the basic technique here is to copy the executable into /ENV/bin/ 
> python, and then sys.prefix will be '/ENV'.  The standard Python  
> installed on a Mac doesn't seem to do this -- the prefix remains '/ 
> opt/local/Library/Frameworks/Python.framework/Versions/2.4'  
> regardless.  (Custom built Python's on Mac work like normal.)

Since I'm the one who's experiencing this, here's some more info:

* Both the Python 2.4 from MacPorts as well as the binary MacPython  
package fail to create a virtualenv because sys.prefix of the copied  
executable still points to the old location.

* Self-compiled Python 2.4 and 2.5 doesn't have this problem, nor  
does the Python 2.5 from MacPorts or the MacPython 2.5 binary (!!!)

So this seems to be constrained to Python 2.4 from MacPorts and  
MacPython.


From chrism at plope.com  Sat Sep 15 19:30:34 2007
From: chrism at plope.com (Chris McDonough)
Date: Sat, 15 Sep 2007 13:30:34 -0400
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <170DC51A-E5E1-405A-AAC6-B10FA576E1D2@weitershausen.de>
References: <46EC03C8.9000604@colorstudy.com>
	<170DC51A-E5E1-405A-AAC6-B10FA576E1D2@weitershausen.de>
Message-ID: <F862F184-E2F3-4D08-B9A1-5EE4399A8A17@plope.com>

FWIW, the same thing seems to happen for me on Tiger using the  
original virtual-python.py against the Apple-compiled Python that  
ships with it:

[chrism at kingfish macsandbox]$ /usr/bin/python ~/bin/virtual-python.py  
--prefix=.Creating ./lib/python2.3
Creating ./lib/python2.3/site-packages
Creating ./include/python2.3
Creating ./bin
Copying /usr/bin/python to ./bin
You're now ready to download ez_setup.py, and run
./bin/python ez_setup.py
[chrism at kingfish macsandbox]$ ls
bin  include  lib
[chrism at kingfish macsandbox]$ bin/python
Python 2.3.5 (#1, Jan 13 2006, 20:13:11)
[GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import sys
 >>> sys.prefix
'/System/Library/Frameworks/Python.framework/Versions/2.3'
 >>>


- C

On Sep 15, 2007, at 12:30 PM, Philipp von Weitershausen wrote:

> On 15 Sep 2007, at 18:09 , Ian Bicking wrote:
>> Hi all.  I'm kind of giving up on workingenv, and have started
>> working from virtual-python as a basis instead (http://
>> pypi.python.org/pypi/virtualenv/).
>>
>> So the basic technique here is to copy the executable into /ENV/bin/
>> python, and then sys.prefix will be '/ENV'.  The standard Python
>> installed on a Mac doesn't seem to do this -- the prefix remains '/
>> opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>> regardless.  (Custom built Python's on Mac work like normal.)
>
> Since I'm the one who's experiencing this, here's some more info:
>
> * Both the Python 2.4 from MacPorts as well as the binary MacPython
> package fail to create a virtualenv because sys.prefix of the copied
> executable still points to the old location.
>
> * Self-compiled Python 2.4 and 2.5 doesn't have this problem, nor
> does the Python 2.5 from MacPorts or the MacPython 2.5 binary (!!!)
>
> So this seems to be constrained to Python 2.4 from MacPorts and
> MacPython.
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>


From b3i4old02 at sneakemail.com  Sun Sep 16 00:23:02 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Sat, 15 Sep 2007 23:23:02 +0100
Subject: [Distutils] Package in both setup_requires and install_requires
	will not be installed
Message-ID: <fchm0n$2da$1@sea.gmane.org>

Joshua Boverhof previously reported this in 
<http://mail.python.org/pipermail/distutils-sig/2007-March/007436.html>.

If you specify a package in setup_requires it will be built in the 
current directory. But even if it is in install_requires as well, it 
won't be installed because the requirement is already satisfied at setup 
time by the package in the build directory.

Is there a workaround for this?


From lists at zopyx.com  Sun Sep 16 13:54:41 2007
From: lists at zopyx.com (Andreas Jung)
Date: Sun, 16 Sep 2007 13:54:41 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46EC03C8.9000604@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
Message-ID: <FD75BE3798EB19560FA638F4@suxmac2-local.local>



--On 15. September 2007 11:09:44 -0500 Ian Bicking <ianb at colorstudy.com> 
wrote:

> Hi all.  I'm kind of giving up on workingenv, and have started working
> from virtual-python as a basis instead
> (http://pypi.python.org/pypi/virtualenv/).
>

I tried virtualenv - however the python in the new environment contains
all eggs in sys.path that are installed for the original Python interpreter.

Bug or feature?

Andreas

--------------

jung at galactica:/tmp> /opt/python-2.4.3/bin/virtualenv envtest
Creating envtest/lib/python2.4
Creating envtest/lib/python2.4/site-packages
Creating envtest/bin
Copying /opt/python-2.4.3/bin/python to envtest/bin
Installing setuptools
Processing setuptools-0.6c7-py2.4.egg
creating /tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg
Extracting setuptools-0.6c7-py2.4.egg to 
/tmp/envtest/lib/python2.4/site-packages
Adding setuptools 0.6c7 to easy-install.pth file
Installing easy_install script to /tmp/envtest/bin
Installing easy_install-2.4 script to /tmp/envtest/bin

Installed 
/tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg
Processing dependencies for setuptools==0.6c7
Finished processing dependencies for setuptools==0.6c7
Processing setuptools-0.6c7-py2.4.egg
removing 
'/tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg' (and 
everything under it)
creating /tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg
Extracting setuptools-0.6c7-py2.4.egg to 
/tmp/envtest/lib/python2.4/site-packages
setuptools 0.6c7 is already the active version in easy-install.pth
Installing easy_install script to /tmp/envtest/bin
Installing easy_install-2.4 script to /tmp/envtest/bin

Installed 
/tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg
Processing dependencies for setuptools==0.6c7
Finished processing dependencies for setuptools==0.6c7

ajung at galactica:/tmp> cd envtest/

ajung at galactica:/tmp/envtest> bin/python -c "import sys; print sys.path"

['', '/tmp/envtest/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/psycopg2-2.0.5.1-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Slimp-0.3-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/chardet-1.0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/zc.catalog-1.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/textindexng-1.0_trunk-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/epydoc-3.0beta1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/pytz-2007c-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/vobject-0.4.8-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/AOPython-1.0.3-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/MyghtyUtils-0.52-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/docutils-0.4-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Elixir-0.3.0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Cheesecake-0.6.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/migrate-0.2.2-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/enfold.xapian-0.9.14-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/lemur.xapian-0.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/simplejson-1.7.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/setuptools-0.6c6-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/zc.buildout-1.0.0b25-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/MySQL_python-1.2.2-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/igraph-0.4.2-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/pyflakes-0.2.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/pylint-0.13.2-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/pudge-0.1.3-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/python_dateutil-1.2-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Mako-0.1.8-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Paste-1.4-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/PasteDeploy-1.3.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/PasteScript-1.3.5-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/i18ndude-3.0b2-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/workingenv.py-0.6.5-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Beaker-0.7.4-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/grok-0.9dev_r77767-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/martian-0.8-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/ZConfig-2.3.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/bebop.recipe.svncheckout-0.0.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/TextIndexNGExtensions-4.0.0-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/python_Levenshtein-0.10-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/grokproject-0.5.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/yolk-0.3.0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/SQLAlchemy-0.3.10-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/pyparsing-1.4.7-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/kid-0.9.6-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/pysqlite-2.3.5-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/zopeproject-0.3.2-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/ctypes-1.0.2-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/buildbot-0.7.5-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/BeautifulSoup-3.0.4-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/z3c.sqlalchemy-1.0.11-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/feedparser-4.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/zopyx.textindexng3-4.0.0-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/zopyx.slimp-0.2.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/haufe.testrunner-0.2.1-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/coverage-2.77-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/genesis.wsl-0.2dev_r48613-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/antlr_python_runtime-3.0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/ZopeSkel-0.8-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/Cheetah-1.0-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/haufe.python_repos_handler-0.1.0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/PythonEggTools-1.0a0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/haufe.selenium-0.1.0-py2.4.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/python_ldap-2.3.1-py2.4-linux-x86_64.egg', 
'/opt/python-2.4.3/lib/python2.4/site-packages/virtualenv-0.8.1-py2.4.egg', 
'/tmp/envtest/lib/python24.zip', '/opt/python-2.4.3/lib/python2.4', 
'/tmp/envtest/lib/python2.4', '/tmp/envtest/lib/python2.4/plat-linux2', 
'/tmp/envtest/lib/python2.4/lib-tk', 
'/opt/python-2.4.3/lib/python2.4/lib-dynload', 
'/tmp/envtest/lib/python2.4/site-packages', 
'/opt/python-2.4.3/lib/python2.4/site-packages', 
'/opt/python-2.4.3/lib/python2.4/site-packages/PIL']
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070916/6caeea2a/attachment.pgp 

From ianb at colorstudy.com  Sun Sep 16 18:29:05 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Sun, 16 Sep 2007 11:29:05 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <FD75BE3798EB19560FA638F4@suxmac2-local.local>
References: <46EC03C8.9000604@colorstudy.com>
	<FD75BE3798EB19560FA638F4@suxmac2-local.local>
Message-ID: <46ED59D1.5070508@colorstudy.com>

Andreas Jung wrote:
> 
> 
> --On 15. September 2007 11:09:44 -0500 Ian Bicking <ianb at colorstudy.com> 
> wrote:
> 
>> Hi all.  I'm kind of giving up on workingenv, and have started working
>> from virtual-python as a basis instead
>> (http://pypi.python.org/pypi/virtualenv/).
>>
> 
> I tried virtualenv - however the python in the new environment contains
> all eggs in sys.path that are installed for the original Python 
> interpreter.
> 
> Bug or feature?

You have to use --no-site-packages to avoid that.

-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From ianb at colorstudy.com  Sun Sep 16 21:44:52 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Sun, 16 Sep 2007 14:44:52 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
Message-ID: <46ED87B4.7050006@colorstudy.com>

Ronald Oussoren wrote:
> 
> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
> 
>> Hi all.  I'm kind of giving up on workingenv, and have started working
>> from virtual-python as a basis instead
>> (http://pypi.python.org/pypi/virtualenv/).
>>
>> So the basic technique here is to copy the executable into
>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
>> Python installed on a Mac doesn't seem to do this -- the prefix remains
>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>> regardless.  (Custom built Python's on Mac work like normal.)
> 
> All framework builds behave as you describe, Modules/getpath.c 
> special-cases calculation of sys.prefix for framework builds of Python 
> (the prefix is inside the framework regardless of where the executable is).

Is there any way to effect that calculation?  I.e., in a normal build 
that calculation is based on the location of the executable, so 
virtualenv moves the executable to effect that.

>> Does anyone know a way to get the normal effect on this Mac Python?
> 
> You could copy the framework into the virtual python environment and 
> rewrite linker commands in the python command-line executable to use 
> that version of the framework.

That sounds like the similar situation to the standard library on other 
posix systems, which I've avoided copying.  To do that, I put a text 
file in that points to the original stdlib location, and in site.py I 
add that in (I only copy the portions of the stdlib that are needed when 
site.py is loaded -- os, re, etc).  So the same thing should be possible 
with the framework, and that's not too much of a problem.

I don't have any idea how to figure out the linker commands to change 
the Python command-line executable...?  I don't even have the vocabulary 
to begin figuring that out, I'm afraid.  Can you give an example?


-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From ronaldoussoren at mac.com  Sun Sep 16 21:49:23 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 16 Sep 2007 21:49:23 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46ED87B4.7050006@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
Message-ID: <803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>


On 16 Sep, 2007, at 21:44, Ian Bicking wrote:

> Ronald Oussoren wrote:
>> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
>>> Hi all.  I'm kind of giving up on workingenv, and have started  
>>> working
>>> from virtual-python as a basis instead
>>> (http://pypi.python.org/pypi/virtualenv/).
>>>
>>> So the basic technique here is to copy the executable into
>>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
>>> Python installed on a Mac doesn't seem to do this -- the prefix  
>>> remains
>>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>>> regardless.  (Custom built Python's on Mac work like normal.)
>> All framework builds behave as you describe, Modules/getpath.c  
>> special-cases calculation of sys.prefix for framework builds of  
>> Python (the prefix is inside the framework regardless of where the  
>> executable is).
>
> Is there any way to effect that calculation?  I.e., in a normal  
> build that calculation is based on the location of the executable,  
> so virtualenv moves the executable to effect that.

Move the framework.

>
>
>>> Does anyone know a way to get the normal effect on this Mac Python?
>> You could copy the framework into the virtual python environment  
>> and rewrite linker commands in the python command-line executable  
>> to use that version of the framework.
>
> That sounds like the similar situation to the standard library on  
> other posix systems, which I've avoided copying.  To do that, I put  
> a text file in that points to the original stdlib location, and in  
> site.py I add that in (I only copy the portions of the stdlib that  
> are needed when site.py is loaded -- os, re, etc).  So the same  
> thing should be possible with the framework, and that's not too much  
> of a problem.
>
>
> I don't have any idea how to figure out the linker commands to  
> change the Python command-line executable...?  I don't even have the  
> vocabulary to begin figuring that out, I'm afraid.  Can you give an  
> example?

macholib (which is on PyPI) can do that. I'm not aware of a command- 
line tool that can do the same thing.

>
>
>
> -- 
> Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
>            : Write code, do good : http://topp.openplans.org/careers


From ianb at colorstudy.com  Sun Sep 16 21:55:19 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Sun, 16 Sep 2007 14:55:19 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
Message-ID: <46ED8A27.7050303@colorstudy.com>

Ronald Oussoren wrote:
> 
> On 16 Sep, 2007, at 21:44, Ian Bicking wrote:
> 
>> Ronald Oussoren wrote:
>>> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
>>>> Hi all.  I'm kind of giving up on workingenv, and have started working
>>>> from virtual-python as a basis instead
>>>> (http://pypi.python.org/pypi/virtualenv/).
>>>>
>>>> So the basic technique here is to copy the executable into
>>>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
>>>> Python installed on a Mac doesn't seem to do this -- the prefix remains
>>>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>>>> regardless.  (Custom built Python's on Mac work like normal.)
>>> All framework builds behave as you describe, Modules/getpath.c 
>>> special-cases calculation of sys.prefix for framework builds of 
>>> Python (the prefix is inside the framework regardless of where the 
>>> executable is).
>>
>> Is there any way to effect that calculation?  I.e., in a normal build 
>> that calculation is based on the location of the executable, so 
>> virtualenv moves the executable to effect that.
> 
> Move the framework.

I don't really know what that means...?  What exactly is the framework?


-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From ronaldoussoren at mac.com  Sun Sep 16 21:38:16 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Sun, 16 Sep 2007 21:38:16 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46EC03C8.9000604@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
Message-ID: <A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>


On 15 Sep, 2007, at 18:09, Ian Bicking wrote:

> Hi all.  I'm kind of giving up on workingenv, and have started working
> from virtual-python as a basis instead
> (http://pypi.python.org/pypi/virtualenv/).
>
> So the basic technique here is to copy the executable into
> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
> Python installed on a Mac doesn't seem to do this -- the prefix  
> remains
> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
> regardless.  (Custom built Python's on Mac work like normal.)

All framework builds behave as you describe, Modules/getpath.c special- 
cases calculation of sys.prefix for framework builds of Python (the  
prefix is inside the framework regardless of where the executable is).

>
>
> Does anyone know a way to get the normal effect on this Mac Python?

You could copy the framework into the virtual python environment and  
rewrite linker commands in the python command-line executable to use  
that version of the framework.

Ronald

>
>
> -- 
> Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
>             : Write code, do good : http://topp.openplans.org/careers
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


From ronaldoussoren at mac.com  Mon Sep 17 19:15:31 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Mon, 17 Sep 2007 19:15:31 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46ED8A27.7050303@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
Message-ID: <912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>


On 16 Sep, 2007, at 21:55, Ian Bicking wrote:

> Ronald Oussoren wrote:
>> On 16 Sep, 2007, at 21:44, Ian Bicking wrote:
>>> Ronald Oussoren wrote:
>>>> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
>>>>> Hi all.  I'm kind of giving up on workingenv, and have started  
>>>>> working
>>>>> from virtual-python as a basis instead
>>>>> (http://pypi.python.org/pypi/virtualenv/).
>>>>>
>>>>> So the basic technique here is to copy the executable into
>>>>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
>>>>> Python installed on a Mac doesn't seem to do this -- the prefix  
>>>>> remains
>>>>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>>>>> regardless.  (Custom built Python's on Mac work like normal.)
>>>> All framework builds behave as you describe, Modules/getpath.c  
>>>> special-cases calculation of sys.prefix for framework builds of  
>>>> Python (the prefix is inside the framework regardless of where  
>>>> the executable is).
>>>
>>> Is there any way to effect that calculation?  I.e., in a normal  
>>> build that calculation is based on the location of the executable,  
>>> so virtualenv moves the executable to effect that.
>> Move the framework.
>
> I don't really know what that means...?  What exactly is the  
> framework?

The python framework, that is /Library/Frameworks/Python.framework  
(or /System/... if you use Apple's build of Python).  getpath.c uses  
some API calls to determine the absolute path of the python framework  
linked into the current executable and sets sys.prefix to a directory  
inside that framework.

As background info for anyone that's not used to the mac way: OSX  
supports the usual unix organisation of shared libraries but also has  
a different method: frameworks. A framework is basicly a directory  
containing the shared library, header files and resources (the last  
two are optional) and also supports versioning, although Apple's  
development tools don't offer full support for that.

I should be possible to coax a framework install into support virtual- 
python by creating a directory structure for a python.framework inside  
the virtual-python environment and using a simular mechanism as you  
have now for adding the real stdlib to sys.path. You will have to  
modify the copied python executable to load this mini-framework  
because the OSX linker adds absolute paths to shared libraries and  
frameworks to the executable.

The macholib library can be used to do this last task, it is used by  
py2app for rewriting the linker commands in shared libraries that are  
used in application bundles. I don't have an example for that handy,  
but it should be easy enough to extract code from macho_standalone.

Ronald
>
>
>
> -- 
> Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
>            : Write code, do good : http://topp.openplans.org/careers


From karl.kobata at syncira.com  Mon Sep 17 20:15:39 2007
From: karl.kobata at syncira.com (Karl Kobata)
Date: Mon, 17 Sep 2007 11:15:39 -0700
Subject: [Distutils] Export make file from distutils
Message-ID: <5p5kll$1k3n2f@rrcs-agw-02.hrndva.rr.com>

Please help.

I am having a linking error porting wxPython to the laptop cygwin
environment.

 

Is there a way to get a make file exported from the setup.py file included
with wxPython?

 

Will I then be able to execute the make file instead of setup.py for my
particular case.

 

Please help.

 

Thanks

karl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070917/74891987/attachment.htm 

From matt at matt-good.net  Tue Sep 18 02:16:01 2007
From: matt at matt-good.net (Matt Good)
Date: Mon, 17 Sep 2007 17:16:01 -0700
Subject: [Distutils] Export make file from distutils
In-Reply-To: <5p5kll$1k3n2f@rrcs-agw-02.hrndva.rr.com>
References: <5p5kll$1k3n2f@rrcs-agw-02.hrndva.rr.com>
Message-ID: <90CFDCC7-0555-4635-A977-D9E2512AB1DB@matt-good.net>

On Sep 17, 2007, at 11:15 AM, Karl Kobata wrote:
> I am having a linking error porting wxPython to the laptop cygwin  
> environment.
>
>
>
> Is there a way to get a make file exported from the setup.py file  
> included with wxPython?
>
>
>
> Will I then be able to execute the make file instead of setup.py  
> for my particular case.
>

Distutils does not use make files.  The setup.py file serves the same  
purpose as a make file.  If you post the linking error someone may be  
able to help you with that.

-- Matt

From b3i4old02 at sneakemail.com  Tue Sep 18 01:42:06 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Tue, 18 Sep 2007 00:42:06 +0100
Subject: [Distutils] Package in both setup_requires and install_requires
 will not be installed
In-Reply-To: <fchm0n$2da$1@sea.gmane.org>
References: <fchm0n$2da$1@sea.gmane.org>
Message-ID: <fcn3d0$p1f$1@sea.gmane.org>

Michael Hoffman wrote:

> If you specify a package in setup_requires it will be built in the 
> current directory. But even if it is in install_requires as well, it 
> won't be installed because the requirement is already satisfied at setup 
> time by the package in the build directory.

I came up with this workaround:

from setuptools.command.install import install

class SubprocessEasyInstall(install):
     def run(self):
         install.run(self)

         args = [sys.executable, "-m", "easy_install"] + setup_requires
         check_call(args)

And in the setup() call, ensure you include:

cmdclass=dict(install=SubprocessEasyInstall)

This will easy_install the setup_requires in a separate process which is 
unpolluted by the distributions already downloaded during setup.


From karl.kobata at syncira.com  Tue Sep 18 06:58:38 2007
From: karl.kobata at syncira.com (Karl Kobata)
Date: Mon, 17 Sep 2007 21:58:38 -0700
Subject: [Distutils] wxPython setup.py linking error
Message-ID: <5p5kll$1kaiar@rrcs-agw-02.hrndva.rr.com>

 

Please help.

I am compiling wxPython in the cygwin environment on a laptop.  I have
posted a similar questions with wxPython user group, but since it is a
compiler/linker related problem, the help is limited. 

 

I am apparently missing libraries from GTK2/ATK/etc. required to resolve the
references.

What libraries are they?

How can I verify that I have them?

What changes or options are required to resolve the errors?

What changes will I need to make to distutils to compile and link properly.

 

Please help.

karl 

 

 

1) the configure files used is as follows:

../configure  \

   --prefix=/opt/wx/2.8  \

   --enable-optimise  \

   --enable-debug_flag  \

   --with-gtk=gtk+-2.0  \

   --disable-stl  \

   --disable-shared  \

   --disable-sound  \

   --enable-rpath=/opt/wx/2.8/lib

2) once installed I have edited the config.py file with

  - UNICODE = 0

  - WX_CONFIG = '/opt/wx/2.8/bin/wx-config'

3) set environment variable:

setenv LD_LIBRARY_PATH /opt/wx/2.8/lib

setenv PATH /opt/wx/2.8/bin:$PATH

4) execute python setup.py build_ext --inplace --debug

5) when the linking starts I get a slew of unresolved variables.  See below.

 

----------------------------------

> python setup.py build_ext --inplace --debug

Preparing CORE...

Preparing GLCANVAS...

Preparing STC...

Preparing GIZMOS...

running build_ext

building '_core_' extension

creating build-gtk2

creating build-gtk2/temp.cygwin-1.5.24-i686-2.5

creating build-gtk2/temp.cygwin-1.5.24-i686-2.5/src

creating build-gtk2/temp.cygwin-1.5.24-i686-2.5/src/gtk

gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -g
-DSWIG_TYPE_TABLE=_wxPython_table -DSWIG_PYTHON_OUTPUT_TUPLE

-DWXP_USE_THREAD=1 -UNDEBUG -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES
-D__WXDEBUG__ -D__WXGTK__ -DXTHREADS -DXUSE_MTSAFE_API -Iinclude -Isrc

-I/opt/wx/2.8/lib/wx/include/gtk2-ansi-debug-static-2.8

-I/opt/wx/2.8/include/wx-2.8 -I/usr/include/gtk-2.0
-I/usr/lib/gtk-2.0/include -I/usr/X11R6/include -I/usr/include/atk-1.0
-I/usr/include/pango-1.0 -I/usr/include/freetype2 -I/usr/include/glib-2.0
-I/usr/lib/glib-2.0/include -I/usr/include/python2.5 -c src/helpers.cpp -o
build-gtk2/temp.cygwin-1.5.24-i686-2.5/src/helpers.o -g -O0

cc1plus: warning: command line option "-Wstrict-prototypes" is valid for
Ada/C/ObjC but not for C++ gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -g -DSWIG_TYPE_TABLE=_wxPython_table
-DSWIG_PYTHON_OUTPUT_TUPLE

-DWXP_USE_THREAD=1 -UNDEBUG -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES
-D__WXDEBUG__ -D__WXGTK__ -DXTHREADS -DXUSE_MTSAFE_API -Iinclude -Isrc

-I/opt/wx/2.8/lib/wx/include/gtk2-ansi-debug-static-2.8

-I/opt/wx/2.8/include/wx-2.8 -I/usr/include/gtk-2.0
-I/usr/lib/gtk-2.0/include -I/usr/X11R6/include -I/usr/include/atk-1.0
-I/usr/include/pango-1.0 -I/usr/include/freetype2 -I/usr/include/glib-2.0
-I/usr/lib/glib-2.0/include -I/usr/include/python2.5 -c
src/gtk/_core_wrap.cpp -o
build-gtk2/temp.cygwin-1.5.24-i686-2.5/src/gtk/_core_wrap.o -g -O0

cc1plus: warning: command line option "-Wstrict-prototypes" is valid for
Ada/C/ObjC but not for C++

src/gtk/_core_wrap.cpp: In function `PyObject*
_wrap_GetAccelFromString(PyObject*, PyObject*, PyObject*)':

src/gtk/_core_wrap.cpp:32501: warning: `wxGetAccelFromString' is deprecated
(declared at /opt/wx/2.8/include/wx-2.8/wx/utils.h:571)

src/gtk/_core_wrap.cpp:32501: warning: `wxGetAccelFromString' is deprecated
(declared at /opt/wx/2.8/include/wx-2.8/wx/utils.h:571)

g++ -shared -Wl,--enable-auto-image-base -g

build-gtk2/temp.cygwin-1.5.24-i686-2.5/src/helpers.o

build-gtk2/temp.cygwin-1.5.24-i686-2.5/src/gtk/_core_wrap.o -L/usr/X11R6/lib
-L/opt/wx/2.8/lib -L/usr/X11R6/lib -L/usr/X11R6/lib
-L/usr/lib/python2.5/config -lgtk-x11-2.0 -lgthread-2.0 -lgdk-x11-2.0
-latk-1.0 -lgdk_pixbuf-2.0 -lpangoxft-1.0 -lXft -lfreetype -lXrender -lXext
-lfontconfig -lpangox-1.0 -lX11 -lpango-1.0 -lm -lgobject-2.0 -lgmodule-2.0
-lglib-2.0 -lintl -lSM -lexpat -lwxtiffd-2.8 -lwxjpegd-2.8 -lwxpngd-2.8 -lz
-lpthread -liconv -lpython2.5 -o wx/_core_.dll
/opt/wx/2.8/lib/libwx_gtk2d_aui-2.8.a /opt/wx/2.8/lib/libwx_gtk2d_xrc-2.8.a

/opt/wx/2.8/lib/libwx_gtk2d_qa-2.8.a /opt/wx/2.8/lib/libwx_gtk2d_html-2.8.a

/opt/wx/2.8/lib/libwx_gtk2d_adv-2.8.a /opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a

/opt/wx/2.8/lib/libwx_based_xml-2.8.a /opt/wx/2.8/lib/libwx_based_net-2.8.a

/opt/wx/2.8/lib/libwx_based-2.8.a

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_colour.o):colour.cpp:(.text+0

x7b): undefined reference to `_gdk_colormap_free_colors'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_colour.o):colour.cpp:(.text+0

xf1): undefined reference to `_gdk_colormap_alloc_color'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_colour.o):colour.cpp:(.text+0

x7c7): undefined reference to `_gdk_color_parse'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x9c):

undefined reference to `_g_source_remove'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xb2):

undefined reference to `_gtk_widget_get_type'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xbd):

undefined reference to `_g_type_class_peek'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xd4):

undefined reference to `_g_signal_lookup'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xf7):

undefined reference to `_g_signal_add_emission_hook'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x1f1):

undefined reference to `_gtk_main_iteration'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x1f6):

undefined reference to `_gtk_events_pending'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x2e3):

undefined reference to `_gtk_widget_get_type'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x2f8):

undefined reference to `_g_type_class_peek'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x316):

undefined reference to `_gdk_threads_enter'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x344):

undefined reference to `_gtk_events_pending'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x354):

undefined reference to `_gdk_threads_leave'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x3f8):

undefined reference to `_gtk_widget_get_type'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x40d):

undefined reference to `_g_type_class_peek'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xd4):

undefined reference to `_g_signal_lookup'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xf7):

undefined reference to `_g_signal_add_emission_hook'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x1f1):

undefined reference to `_gtk_main_iteration'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x1f6):

undefined reference to `_gtk_events_pending'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x2e3):

undefined reference to `_gtk_widget_get_type'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x2f8):

undefined reference to `_g_type_class_peek'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x316):

undefined reference to `_gdk_threads_enter'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x344):

undefined reference to `_gtk_events_pending'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x354):

undefined reference to `_gdk_threads_leave'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x3f8):

undefined reference to `_gtk_widget_get_type'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x40d):

undefined reference to `_g_type_class_peek'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x4c9):

undefined reference to `_g_signal_lookup'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x4ec):

undefined reference to `_g_signal_add_emission_hook'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x507):

undefined reference to `_g_signal_lookup'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x52a):

undefined reference to `_g_signal_add_emission_hook'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x53e):

undefined reference to `_g_source_remove'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x578):

undefined reference to `_gdk_threads_enter'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x5b1):

undefined reference to `_gdk_threads_leave'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x6cb):

undefined reference to `_g_idle_add_full'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x7a9):

undefined reference to `_gtk_window_new'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0x7b6):

undefined reference to `_gtk_widget_realize'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xa99):

undefined reference to `_g_source_remove'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xb59):

undefined reference to `_g_source_remove'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xc19):

undefined reference to `_g_source_remove'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xcbc):

undefined reference to `_gdk_visual_get_best_with_both'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xcda):

undefined reference to `_gdk_colormap_new'

/opt/wx/2.8/lib/libwx_gtk2d_core-2.8.a(corelib_app.o):app.cpp:(.text+0xce2):

undefined reference to `_gtk_widget_set_default_colorma ...........

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x40

5): undefined reference to `_inflateEnd'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x4e

5): undefined reference to `_inflateEnd'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x6a

b): undefined reference to `_inflate'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0xa3

9): undefined reference to `_zlibVersion'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0xa5

1): undefined reference to `_zlibVersion'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0xbc

5): undefined reference to `_inflateInit2_'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x12

ea): undefined reference to `_deflateInit2_'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x17

33): undefined reference to `_deflateEnd'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x18

0a): undefined reference to `_deflate'

/opt/wx/2.8/lib/libwx_based-2.8.a(baselib_zstream.o):zstream.cpp:(.text+0x1a

15): undefined reference to `_deflate'

collect2: ld returned 1 exit status

error: command 'g++' failed with exit status 1

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070917/954a0399/attachment-0001.htm 

From ianb at colorstudy.com  Tue Sep 18 23:17:46 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 18 Sep 2007 16:17:46 -0500
Subject: [Distutils] Windows Python sys.prefix
Message-ID: <46F0407A.5000309@colorstudy.com>

Related to the issue I noted with sys.prefix on a Mac, moving the Python 
executable on Windows also doesn't seem to effect sys.prefix.

How does Python on Windows determine sys.prefix?  Is there a way to 
effect that?

-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From zooko at zooko.com  Tue Sep 18 23:29:25 2007
From: zooko at zooko.com (zooko)
Date: Tue, 18 Sep 2007 15:29:25 -0600
Subject: [Distutils] patches: ez_setup.py: don't import setuptools into your
	current address space in order to learn its version number
Message-ID: <3E6E0CC5-FF5D-4127-A4EA-730786508910@zooko.com>

Folks:

We're gradually converting the allmydata.org tahoe project [1] and  
its spin-off packages to use setuptools.  One problem that comes up  
is that ez_setup.py might require a newer version of setuptools than  
the version of setuptools already installed.

For example, although Tahoe currently uses setuptools to manage its  
dependencies on zfec, foolscap, and nevow [2], but if we execute  
simplejson's ez_setup.py then it requires setuptools v0.6c7 and  
refuses to proceed if an earlier version is installed.

One solution for this problem could be for the packager of simplejson  
(Bob Ippolito) to use the "min_version" patch that we contributed to  
ez_setup.py [3].  This is assuming that simplejson doesn't actually  
*require* the latest version of setuptools, and we could successfully  
install with a slightly older version, but what if a package actually  
does require a newer version of setuptools than the one that is  
already installed?

The following patch was created by Tahoe contributor Arno Washck and  
then modified by me in order to get around such a problem.  The idea  
is simple enough -- reload setuptools.  There may be some subtleties  
that we still need to work out.  This patch hasn't been tested in its  
current form.

--- ez_setup.py~	2007-09-10 12:36:30.000000000 -0600
+++ ez_setup.py	2007-09-18 15:15:23.000000000 -0600
@@ -95,13 +95,9 @@
          pkg_resources.require("setuptools>="+version)
      except pkg_resources.VersionConflict, e:
-        # XXX could we install in a subprocess here?
-        print >>sys.stderr, (
-            "The required version of setuptools (>=%s) is not  
available, and\n"
-            "can't be installed while this script is running. Please  
install\n"
-            " a more recent version first.\n\n(Currently using %r)"
-        ) % (version, e.args[0])
-        sys.exit(2)
+        egg = download_setuptools(version, download_base, to_dir,  
download_delay)
+        sys.path.insert(0, egg)
+        reload(setuptools); setuptools.bootstrap_install_from = egg
def download_setuptools(
      version=DEFAULT_VERSION, download_base=DEFAULT_URL,  
to_dir=os.curdir,

-------

An alternative idea figure out the version of the current install of  
setuptools without importing the package.  The following patch might  
accomplish that.  It is also not tested.

--- ez_setup.py~	2007-09-10 12:36:30.000000000 -0600
+++ ez_setup.py	2007-09-18 15:26:50.000000000 -0600
@@ -77,31 +77,13 @@
      this routine will print a message to ``sys.stderr`` and raise  
SystemExit in
      an attempt to abort the calling script.
      """
-    try:
-        import setuptools
-        if setuptools.__version__ == '0.0.1':
-            print >>sys.stderr, (
-            "You have an obsolete version of setuptools installed.   
Please\n"
-            "remove it from your system entirely before rerunning  
this script."
-            )
-            sys.exit(2)
-    except ImportError:
-        egg = download_setuptools(version, download_base, to_dir,  
download_delay)
-        sys.path.insert(0, egg)
-        import setuptools; setuptools.bootstrap_install_from = egg
-
+    verstr = os.system("python -c \"import setuptools;print  
setuptools.__version__\"")
      import pkg_resources
-    try:
-        pkg_resources.require("setuptools>="+version)
-    except pkg_resources.VersionConflict, e:
-        # XXX could we install in a subprocess here?
-        print >>sys.stderr, (
-            "The required version of setuptools (>=%s) is not  
available, and\n"
-            "can't be installed while this script is running. Please  
install\n"
-            " a more recent version first.\n\n(Currently using %r)"
-        ) % (version, e.args[0])
-        sys.exit(2)
+    if pkg_resources.parse_version(verstr) <  
pkg_resources.parse_version(version):
+        egg = download_setuptools(version, download_base, to_dir)
+        sys.path.insert(0, egg)
+        import setuptools; setuptools.bootstrap_install_from = egg
def download_setuptools(
      version=DEFAULT_VERSION, download_base=DEFAULT_URL,  
to_dir=os.curdir,

-------

Regards,

Zooko

[1] http://allmydata.org
[2] http://allmydata.org/trac/tahoe/browser/calcdeps.py
[3] http://mail.python.org/pipermail/distutils-sig/2007-September/ 
008257.html

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ez_setup-hack.patch
Type: application/octet-stream
Size: 886 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070918/d6f64925/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ez_setup-hack2.patch
Type: application/octet-stream
Size: 1743 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070918/d6f64925/attachment-0001.obj 

From ianb at colorstudy.com  Wed Sep 19 00:51:57 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 18 Sep 2007 17:51:57 -0500
Subject: [Distutils] Windows Python sys.prefix
In-Reply-To: <06a901c7fa45$3f4da8d0$bde8fa70$@com.au>
References: <46F0407A.5000309@colorstudy.com>
	<06a901c7fa45$3f4da8d0$bde8fa70$@com.au>
Message-ID: <46F0568D.8000609@colorstudy.com>

Mark Hammond wrote:
>> How does Python on Windows determine sys.prefix?
> 
> By consulting the registry.  The main use-case for that is when Python is
> embedded in another executable - often, but not always, via COM.  When
> Python is hosted inside excel.exe, for example, its impossible to calculate
> sys.path or sys.prefix based on that executable.  In that same example,
> Python's DLL will have been loaded from a directory on the global PATH,
> which generally means the \Windows\System32 directory - so that too is no
> help in calculating the path.
> 
> [IIRC, Python actually tries to use the executable to locate its 'landmark'
> and only falls back to the registry when that fails - but that sounds
> exactly like what is happening in your example.  The gory details are in
> PC/getpathp.c]

I can certainly create a landmark if at all reasonable.  I'm looking at 
getpathp.c, and it looks like it just looks for lib/os.py ... and I 
think it walks up from the current directory until it finds it?  So 
right now I have:

   bin/python.exe
   lib/python2.5/os.py

But perhaps if I just change it to:

   bin/python.exe
   lib/os.py

it will work?  On windows, does it just leave out the "/python2.5/" 
portion of the lib path?


>> Is there a way to effect that?
> 
> Setting PYTHONHOME in the environment is the only way I'm aware of.

That's kind of what workingenv did, which I'm trying to avoid -- you 
have to worry about, for instance, calling another Python subprocess if 
you don't want that process to be in the same environment.


-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From mhammond at skippinet.com.au  Wed Sep 19 00:42:45 2007
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 19 Sep 2007 08:42:45 +1000
Subject: [Distutils] Windows Python sys.prefix
In-Reply-To: <46F0407A.5000309@colorstudy.com>
References: <46F0407A.5000309@colorstudy.com>
Message-ID: <06a901c7fa45$3f4da8d0$bde8fa70$@com.au>

> How does Python on Windows determine sys.prefix?

By consulting the registry.  The main use-case for that is when Python is
embedded in another executable - often, but not always, via COM.  When
Python is hosted inside excel.exe, for example, its impossible to calculate
sys.path or sys.prefix based on that executable.  In that same example,
Python's DLL will have been loaded from a directory on the global PATH,
which generally means the \Windows\System32 directory - so that too is no
help in calculating the path.

[IIRC, Python actually tries to use the executable to locate its 'landmark'
and only falls back to the registry when that fails - but that sounds
exactly like what is happening in your example.  The gory details are in
PC/getpathp.c]


> Is there a way to effect that?

Setting PYTHONHOME in the environment is the only way I'm aware of.

Cheers,

Mark


From mhammond at skippinet.com.au  Wed Sep 19 02:12:21 2007
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 19 Sep 2007 10:12:21 +1000
Subject: [Distutils] Windows Python sys.prefix
In-Reply-To: <46F0568D.8000609@colorstudy.com>
References: <46F0407A.5000309@colorstudy.com>
	<06a901c7fa45$3f4da8d0$bde8fa70$@com.au>
	<46F0568D.8000609@colorstudy.com>
Message-ID: <06b301c7fa51$e4781b40$ad6851c0$@com.au>

> I can certainly create a landmark if at all reasonable.  I'm looking at
> getpathp.c, and it looks like it just looks for lib/os.py ... and I
> think it walks up from the current directory until it finds it?  So
> right now I have:
> 
>    bin/python.exe
>    lib/python2.5/os.py
> 
> But perhaps if I just change it to:
> 
>    bin/python.exe
>    lib/os.py
> 
> it will work?  

Yep, and a quick test here works fine.

> On windows, does it just leave out the "/python2.5/"
> portion of the lib path?

well - it doesn't "leave it out" as such - it just never adds anything like
that :)

Mark


From ianb at colorstudy.com  Wed Sep 19 04:50:36 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 18 Sep 2007 21:50:36 -0500
Subject: [Distutils] Windows Python sys.prefix
In-Reply-To: <06b301c7fa51$e4781b40$ad6851c0$@com.au>
References: <46F0407A.5000309@colorstudy.com>
	<06a901c7fa45$3f4da8d0$bde8fa70$@com.au>
	<46F0568D.8000609@colorstudy.com>
	<06b301c7fa51$e4781b40$ad6851c0$@com.au>
Message-ID: <46F08E7C.4030001@colorstudy.com>

After that and a bunch of other little updates, virtualenv (0.8.3) now 
works on Windows!

Now it's just the blasted Mac Framework stuff that stops this from being 
a complete workingenv replacement.

Mark Hammond wrote:
>> I can certainly create a landmark if at all reasonable.  I'm looking at
>> getpathp.c, and it looks like it just looks for lib/os.py ... and I
>> think it walks up from the current directory until it finds it?  So
>> right now I have:
>>
>>    bin/python.exe
>>    lib/python2.5/os.py
>>
>> But perhaps if I just change it to:
>>
>>    bin/python.exe
>>    lib/os.py
>>
>> it will work?  
> 
> Yep, and a quick test here works fine.
> 
>> On windows, does it just leave out the "/python2.5/"
>> portion of the lib path?
> 
> well - it doesn't "leave it out" as such - it just never adds anything like
> that :)
> 
> Mark
> 


-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From nad at acm.org  Wed Sep 19 06:20:53 2007
From: nad at acm.org (Ned Deily)
Date: Tue, 18 Sep 2007 21:20:53 -0700
Subject: [Distutils] Windows Python sys.prefix
References: <46F0407A.5000309@colorstudy.com>
	<06a901c7fa45$3f4da8d0$bde8fa70$@com.au>
	<46F0568D.8000609@colorstudy.com>
	<06b301c7fa51$e4781b40$ad6851c0$@com.au>
	<46F08E7C.4030001@colorstudy.com>
Message-ID: <nad-2C7863.21205318092007@sea.gmane.org>

In article <46F08E7C.4030001 at colorstudy.com>,
 Ian Bicking <ianb at colorstudy.com> wrote:
> Now it's just the blasted Mac Framework stuff that stops this from being 
> a complete workingenv replacement.

It *might* help to move that discussion to the mac python list (a.k.a 
gmane.comp.python.apple).

-- 
 Ned Deily,
 nad at acm.org


From ianb at colorstudy.com  Wed Sep 19 06:25:06 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 18 Sep 2007 23:25:06 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
	<912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
Message-ID: <46F0A4A2.1090900@colorstudy.com>

Ronald Oussoren wrote:
> On 16 Sep, 2007, at 21:55, Ian Bicking wrote:
>> Ronald Oussoren wrote:
>>> On 16 Sep, 2007, at 21:44, Ian Bicking wrote:
>>>> Ronald Oussoren wrote:
>>>>> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
>>>>>> Hi all.  I'm kind of giving up on workingenv, and have started 
>>>>>> working
>>>>>> from virtual-python as a basis instead
>>>>>> (http://pypi.python.org/pypi/virtualenv/).
>>>>>>
>>>>>> So the basic technique here is to copy the executable into
>>>>>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
>>>>>> Python installed on a Mac doesn't seem to do this -- the prefix 
>>>>>> remains
>>>>>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>>>>>> regardless.  (Custom built Python's on Mac work like normal.)
>>>>> All framework builds behave as you describe, Modules/getpath.c 
>>>>> special-cases calculation of sys.prefix for framework builds of 
>>>>> Python (the prefix is inside the framework regardless of where the 
>>>>> executable is).
>>>>
>>>> Is there any way to effect that calculation?  I.e., in a normal 
>>>> build that calculation is based on the location of the executable, 
>>>> so virtualenv moves the executable to effect that.
>>> Move the framework.
>>
>> I don't really know what that means...?  What exactly is the framework?
> 
> The python framework, that is /Library/Frameworks/Python.framework (or 
> /System/... if you use Apple's build of Python).  getpath.c uses some 
> API calls to determine the absolute path of the python framework linked 
> into the current executable and sets sys.prefix to a directory inside 
> that framework.

Do you have a reference to the getpath.c that it uses?  Windows seems to 
have something kind of hardcoded, but also a detection scheme, and maybe 
similarly on Mac there's something I can do to avoid the hardcoded portion.

> As background info for anyone that's not used to the mac way: OSX 
> supports the usual unix organisation of shared libraries but also has a 
> different method: frameworks. A framework is basicly a directory 
> containing the shared library, header files and resources (the last two 
> are optional) and also supports versioning, although Apple's development 
> tools don't offer full support for that.
> 
> I should be possible to coax a framework install into support 
> virtual-python by creating a directory structure for a python.framework 
> inside the virtual-python environment and using a simular mechanism as 
> you have now for adding the real stdlib to sys.path. You will have to 
> modify the copied python executable to load this mini-framework because 
> the OSX linker adds absolute paths to shared libraries and frameworks to 
> the executable.
> 
> The macholib library can be used to do this last task, it is used by 
> py2app for rewriting the linker commands in shared libraries that are 
> used in application bundles. I don't have an example for that handy, but 
> it should be easy enough to extract code from macho_standalone.

I noticed in p2app it has a file main.c in it, which I think is the 
Python interpreter code... maybe it recompiles the interpreter?


-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From ronaldoussoren at mac.com  Wed Sep 19 07:06:50 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 19 Sep 2007 07:06:50 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46F0A4A2.1090900@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
	<912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
	<46F0A4A2.1090900@colorstudy.com>
Message-ID: <487BB084-90D9-45AB-A5F3-C4EF469FBB7F@mac.com>


On 19 Sep, 2007, at 6:25, Ian Bicking wrote:

> Ronald Oussoren wrote:
>> On 16 Sep, 2007, at 21:55, Ian Bicking wrote:
>>> Ronald Oussoren wrote:
>>>> On 16 Sep, 2007, at 21:44, Ian Bicking wrote:
>>>>> Ronald Oussoren wrote:
>>>>>> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
>>>>>>> Hi all.  I'm kind of giving up on workingenv, and have started  
>>>>>>> working
>>>>>>> from virtual-python as a basis instead
>>>>>>> (http://pypi.python.org/pypi/virtualenv/).
>>>>>>>
>>>>>>> So the basic technique here is to copy the executable into
>>>>>>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The  
>>>>>>> standard
>>>>>>> Python installed on a Mac doesn't seem to do this -- the  
>>>>>>> prefix remains
>>>>>>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>>>>>>> regardless.  (Custom built Python's on Mac work like normal.)
>>>>>> All framework builds behave as you describe, Modules/getpath.c  
>>>>>> special-cases calculation of sys.prefix for framework builds of  
>>>>>> Python (the prefix is inside the framework regardless of where  
>>>>>> the executable is).
>>>>>
>>>>> Is there any way to effect that calculation?  I.e., in a normal  
>>>>> build that calculation is based on the location of the  
>>>>> executable, so virtualenv moves the executable to effect that.
>>>> Move the framework.
>>>
>>> I don't really know what that means...?  What exactly is the  
>>> framework?
>> The python framework, that is /Library/Frameworks/Python.framework  
>> (or /System/... if you use Apple's build of Python).  getpath.c  
>> uses some API calls to determine the absolute path of the python  
>> framework linked into the current executable and sets sys.prefix to  
>> a directory inside that framework.
>
> Do you have a reference to the getpath.c that it uses?  Windows  
> seems to have something kind of hardcoded, but also a detection  
> scheme, and maybe similarly on Mac there's something I can do to  
> avoid the hardcoded portion.

It is in the python.org source tree: Modules/getpath.c

>
>
>> As background info for anyone that's not used to the mac way: OSX  
>> supports the usual unix organisation of shared libraries but also  
>> has a different method: frameworks. A framework is basicly a  
>> directory containing the shared library, header files and resources  
>> (the last two are optional) and also supports versioning, although  
>> Apple's development tools don't offer full support for that.
>> I should be possible to coax a framework install into support  
>> virtual-python by creating a directory structure for a  
>> python.framework inside the virtual-python environment and using a  
>> simular mechanism as you have now for adding the real stdlib to  
>> sys.path. You will have to modify the copied python executable to  
>> load this mini-framework because the OSX linker adds absolute paths  
>> to shared libraries and frameworks to the executable.
>> The macholib library can be used to do this last task, it is used  
>> by py2app for rewriting the linker commands in shared libraries  
>> that are used in application bundles. I don't have an example for  
>> that handy, but it should be easy enough to extract code from  
>> macho_standalone.
>
> I noticed in p2app it has a file main.c in it, which I think is the  
> Python interpreter code... maybe it recompiles the interpreter?

No, py2app's main.c is something completely different. It is the main  
executable for an application bundle and is very much tuned for that.

Ronald


From ianb at colorstudy.com  Wed Sep 19 07:26:49 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 19 Sep 2007 00:26:49 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <487BB084-90D9-45AB-A5F3-C4EF469FBB7F@mac.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
	<912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
	<46F0A4A2.1090900@colorstudy.com>
	<487BB084-90D9-45AB-A5F3-C4EF469FBB7F@mac.com>
Message-ID: <46F0B319.9010009@colorstudy.com>

Ronald Oussoren wrote:
> 
> On 19 Sep, 2007, at 6:25, Ian Bicking wrote:
> 
>> Ronald Oussoren wrote:
>>> On 16 Sep, 2007, at 21:55, Ian Bicking wrote:
>>>> Ronald Oussoren wrote:
>>>>> On 16 Sep, 2007, at 21:44, Ian Bicking wrote:
>>>>>> Ronald Oussoren wrote:
>>>>>>> On 15 Sep, 2007, at 18:09, Ian Bicking wrote:
>>>>>>>> Hi all.  I'm kind of giving up on workingenv, and have started 
>>>>>>>> working
>>>>>>>> from virtual-python as a basis instead
>>>>>>>> (http://pypi.python.org/pypi/virtualenv/).
>>>>>>>>
>>>>>>>> So the basic technique here is to copy the executable into
>>>>>>>> /ENV/bin/python, and then sys.prefix will be '/ENV'.  The standard
>>>>>>>> Python installed on a Mac doesn't seem to do this -- the prefix 
>>>>>>>> remains
>>>>>>>> '/opt/local/Library/Frameworks/Python.framework/Versions/2.4'
>>>>>>>> regardless.  (Custom built Python's on Mac work like normal.)
>>>>>>> All framework builds behave as you describe, Modules/getpath.c 
>>>>>>> special-cases calculation of sys.prefix for framework builds of 
>>>>>>> Python (the prefix is inside the framework regardless of where 
>>>>>>> the executable is).
>>>>>>
>>>>>> Is there any way to effect that calculation?  I.e., in a normal 
>>>>>> build that calculation is based on the location of the executable, 
>>>>>> so virtualenv moves the executable to effect that.
>>>>> Move the framework.
>>>>
>>>> I don't really know what that means...?  What exactly is the framework?
>>> The python framework, that is /Library/Frameworks/Python.framework 
>>> (or /System/... if you use Apple's build of Python).  getpath.c uses 
>>> some API calls to determine the absolute path of the python framework 
>>> linked into the current executable and sets sys.prefix to a directory 
>>> inside that framework.
>>
>> Do you have a reference to the getpath.c that it uses?  Windows seems 
>> to have something kind of hardcoded, but also a detection scheme, and 
>> maybe similarly on Mac there's something I can do to avoid the 
>> hardcoded portion.
> 
> It is in the python.org source tree: Modules/getpath.c

That's the file used in the Framework build of Python?  I only see some 
small references to __APPLE__, none of which seem related to Frameworks. 
  Also, if you build Python from source it works fine -- it's only the 
python that Apple ships that has the problem.  They must use some 
patched version of this file that they use...?


-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From ronaldoussoren at mac.com  Wed Sep 19 08:29:18 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Tue, 18 Sep 2007 23:29:18 -0700
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46F0B319.9010009@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
	<912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
	<46F0A4A2.1090900@colorstudy.com>
	<487BB084-90D9-45AB-A5F3-C4EF469FBB7F@mac.com>
	<46F0B319.9010009@colorstudy.com>
Message-ID: <E4467712-0115-1000-B075-FFB9CDD66EA4-Webmail-10014@mac.com>

 
On Wednesday, September 19, 2007, at 07:26AM, "Ian Bicking" <ianb at colorstudy.com> wrote:

>> 
>> It is in the python.org source tree: Modules/getpath.c
>
>That's the file used in the Framework build of Python?  I only see some 
>small references to __APPLE__, none of which seem related to Frameworks. 
>  Also, if you build Python from source it works fine -- it's only the 
>python that Apple ships that has the problem.  They must use some 
>patched version of this file that they use...?

Start reading at line 694 in Modules/getpath.c, that's in the head of the release25-maint branch.

I'm pretty sure that you get the same problem when you build from source and give the --enable-framework option to configure. Python.framework is not an Apple invention.

Ronald


From ianb at colorstudy.com  Wed Sep 19 19:23:23 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 19 Sep 2007 12:23:23 -0500
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <E4467712-0115-1000-B075-FFB9CDD66EA4-Webmail-10014@mac.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
	<912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
	<46F0A4A2.1090900@colorstudy.com>
	<487BB084-90D9-45AB-A5F3-C4EF469FBB7F@mac.com>
	<46F0B319.9010009@colorstudy.com>
	<E4467712-0115-1000-B075-FFB9CDD66EA4-Webmail-10014@mac.com>
Message-ID: <46F15B0B.6070403@colorstudy.com>

Ronald Oussoren wrote:
>  
> On Wednesday, September 19, 2007, at 07:26AM, "Ian Bicking" <ianb at colorstudy.com> wrote:
> 
>>> It is in the python.org source tree: Modules/getpath.c
>> That's the file used in the Framework build of Python?  I only see some 
>> small references to __APPLE__, none of which seem related to Frameworks. 
>>  Also, if you build Python from source it works fine -- it's only the 
>> python that Apple ships that has the problem.  They must use some 
>> patched version of this file that they use...?
> 
> Start reading at line 694 in Modules/getpath.c, that's in the head of the release25-maint branch.

It's only 695 lines long...?

At line 458 it seems like it is looking for lib/python2.5/os.py, in some 
Framework-specific code.  It will look relative to the binary, but only 
after looking in the Framework first.  Simply breaking the Framework 
lookup somehow would do it.

Here's the code to find the Framework locations:

     pythonModule = 
NSModuleForSymbol(NSLookupAndBindSymbol("_Py_Initialize"));
     /* Use dylib functions to find out where the framework was loaded 
from */
     buf = (char *)NSLibraryNameForModule(pythonModule);

I'm not sure how to do anything to that though.

-- 
Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
             : Write code, do good : http://topp.openplans.org/careers

From ronaldoussoren at mac.com  Wed Sep 19 20:40:54 2007
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 19 Sep 2007 20:40:54 +0200
Subject: [Distutils] virtual-python.py, sys.prefix, and Mac
In-Reply-To: <46F15B0B.6070403@colorstudy.com>
References: <46EC03C8.9000604@colorstudy.com>
	<A5EF39FD-4026-4413-92F4-FAF2427EA858@mac.com>
	<46ED87B4.7050006@colorstudy.com>
	<803386A4-61C1-4BA5-AF71-21A1C7B0AF68@mac.com>
	<46ED8A27.7050303@colorstudy.com>
	<912E0333-62A0-434C-8F2A-FB0BE2A0F941@mac.com>
	<46F0A4A2.1090900@colorstudy.com>
	<487BB084-90D9-45AB-A5F3-C4EF469FBB7F@mac.com>
	<46F0B319.9010009@colorstudy.com>
	<E4467712-0115-1000-B075-FFB9CDD66EA4-Webmail-10014@mac.com>
	<46F15B0B.6070403@colorstudy.com>
Message-ID: <7C18CD2B-6105-4462-AFDE-3F5CF0C9C399@mac.com>


On 19 Sep, 2007, at 19:23, Ian Bicking wrote:

> Ronald Oussoren wrote:
>> On Wednesday, September 19, 2007, at 07:26AM, "Ian Bicking" <ianb at colorstudy.com 
>> > wrote:
>>>> It is in the python.org source tree: Modules/getpath.c
>>> That's the file used in the Framework build of Python?  I only see  
>>> some small references to __APPLE__, none of which seem related to  
>>> Frameworks.  Also, if you build Python from source it works fine  
>>> -- it's only the python that Apple ships that has the problem.   
>>> They must use some patched version of this file that they use...?
>> Start reading at line 694 in Modules/getpath.c, that's in the head  
>> of the release25-maint branch.
>
> It's only 695 lines long...?

Oops, I've only been using vi for a decade or so... Luckily you found  
the right bit anyway ;-)
>
>
> At line 458 it seems like it is looking for lib/python2.5/os.py, in  
> some Framework-specific code.  It will look relative to the binary,  
> but only after looking in the Framework first.  Simply breaking the  
> Framework lookup somehow would do it.
>
> Here's the code to find the Framework locations:
>
>    pythonModule =  
> NSModuleForSymbol(NSLookupAndBindSymbol("_Py_Initialize"));
>    /* Use dylib functions to find out where the framework was loaded  
> from */
>    buf = (char *)NSLibraryNameForModule(pythonModule);
>
> I'm not sure how to do anything to that though.


I don't think you can. What you can do create the directory structure  
for a framework inside the virtual python environment, including a  
copy of the actual shared library, then copy a minimal subset of the  
stdlibrary into that (anything you need for your custom site.py) and  
have that site.py build the correct sys.path.

You must then copy sys.executable into the virtual python environment  
as well and use macholib to rewrite the link command for the python  
framework from /Library/Frameworks/Python.framework/... to  
@executable_path/../Frameworks/Python.framework/.... This should  
ensure that the right shared library is loaded and your environment  
works as you want.

You could also ship with the sources to the command-line interpreter  
and build that with the right LDFLAGS during installation of the  
virtual-python package.

Ronald
>
>
> -- 
> Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org
>            : Write code, do good : http://topp.openplans.org/careers


From karl.kobata at syncira.com  Wed Sep 19 22:09:40 2007
From: karl.kobata at syncira.com (Karl Kobata)
Date: Wed, 19 Sep 2007 13:09:40 -0700
Subject: [Distutils]  errors compiling c++ extensions with swig on windows
Message-ID: <5p5kll$1kuvpu@rrcs-agw-02.hrndva.rr.com>

Christian,

 

Did you get a reply for your questions?

If so, please forward or post.

 

Thanks

karl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070919/bfc9124c/attachment.htm 

From tseaver at palladion.com  Fri Sep 21 00:11:27 2007
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 20 Sep 2007 18:11:27 -0400
Subject: [Distutils] virtualenv patch
Message-ID: <46F2F00F.1060408@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been working with virtualenv-0.8.4 on Linux, and found two bugs
when using it to built environments outside the tree containing the
source Python's 'lib' directory.

My setup:

 - I have a custom-built Python, which I use for all Zope-related work,
   in '~/projects/Zope/bin/python'.

 - I also have a "virtual" Python, built from /usr/bin/python using the
   old 'virtual_python.py' script, in '~/bin/python'.

 - I was trying to use virtualenv.py to build out new environments
   under '~/tmp', using the "Zope" python as the "source" Python.

The bugs:

 - This first one causes the 'exec_prefix' search to blow up if another
   lib directory *is* "above" the environment;  the 'lib-dyload' from
   the "foreign" Python ends up on sys.path.

 - The next one is that distutils' attempts to parse 'config/Makefile'
   barfs.

The attached patch fixes both, by adding 'lib-dynload' and 'config' to
the set of things which are copied / linked into the target environment.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG8vAP+gerLs4ltQ4RArabAJ9bJvdqIMIRShDKJL2B2931KE8pEQCgoXCa
eGflzC6K2yGyV8yHHgIkB+k=
=DlKN
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: virtualenv-0.8.4-out_of_tree.patch
Type: text/x-patch
Size: 592 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070920/ecd2e10c/attachment.bin 

From tseaver at palladion.com  Tue Sep 25 14:32:09 2007
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 25 Sep 2007 08:32:09 -0400
Subject: [Distutils] Known working sets II [was: Eggification redux]
In-Reply-To: <46F8F9A9.7010903@weitershausen.de>
References: <fd9jc6$kgl$1@sea.gmane.org>	<31ADBDA8-E367-44FB-9045-764F99E1FA79@begeistert.org>	<fdae7f$gtu$1@sea.gmane.org>	<701EA296-36FE-4741-B5FA-270C5B4D6905@zope.com>
	<46F8F5AF.4010505@palladion.com>
	<46F8F9A9.7010903@weitershausen.de>
Message-ID: <46F8FFC9.7080606@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(CC'ing the Python distutils list, which is where setuptools development
is discussed;  maybe I should be including the catalog-sig too, but I
don't read that list).

Philipp von Weitershausen wrote:
> (Also CCing zope3-dev where the first known working set discussion 
> started a while ago)
> 
> Tres Seaver wrote:
>> This is the "known good" problem.  I'm pretty convinced that adding some
>> kind of "PyPI subset", where gardeners for a given "package set" keep
>> the list of packages / versions known to work well together, is the only
>> way out of this issue.  E.g., a URL like:
>>
>>   http://pypi.python.org/pypi/release/zope3.4
>>
>> would be usable as an 'index-url' for setuptools:  when used this way,
>> setuptools would only find / install eggs from the "gardened" set,
>> rather than whatever anyone happened to have uploaded that day.
>>
>> If PyPI can't be tweaked to provide such a feature, we may need to come
>> up with some kind of mirroring scheme which does allow it.
> 
> This is certainly an interesting approach. I'd be curious how you would 
> garden this known working set. Martijn makes a pretty good case for 
> maintaining such working sets close to the package in question (e.g. the 
> grok egg, the Plone egg, etc.).

I would argue that this problem is too big for "developer convenience"
to drive it:  we need concerted effort from the different "communities
of interest" to manage the problem, in much the same way that Debian /
Fedora etc. manage their various distribution releases.

I see a PyPI subset implemented as follows:

 - Subset creator / owner defines the list of PyPI project names which
   are included in the subset.

 - For each project, the available releases known to PyPI fall into one
   of the following states:

   o 'unknown' is the default:  newly-uploaded distributions start here.

   o 'known_good' is a terminal state:  the creator / owner of the
     subset has blessed this version.

   o 'known_bad' is also a terminal state:  this version won't *ever*
     be compatible with the others in the subset.

 - Other users should be able to report "works for me" / "broken"
   against a given distribution *for that subset.*  Perhaps we can
   set up an RSS feed for each subset showning the "unknown" packages,
   so that people know they need testing.

 - Note that it would be possible (assuming setuptools requirements
   specs are sufficiently flexible) to generate a meta-egg from the
   "known_good" distribution list:  such an egg's 'install_requires'
   would need to list the "known_good" values, rather than attempting
   to do range arithmetic.

 - The subset homepage would be usable as 'index_url' for setuptools,
   so that folks wanting to 'easy_install' a package (or drive buildout)
   could restrict the available packages to the "known good" versions.

> I see two more solutions:
> 
> * A versions.cfg that's loaded via HTTP. zc.buildout actually supports 
> this now which makes it quite appealing. Also, far as I know, all major 
> deployers of Zope3 that use zc.buildout for deployment use this form of 
> pinning egg versions right now, which means it's actually being used out 
> there.

Locking down the file doesn't solve the problem of knowing that there
are new candidate / "unknown" distributions which need testing, nor of
colleting the "works for me" / "broken" information from users.  A
subset could certainly generate such a view, however, which would make
zc.buildout integration work on a par with the
'easy_install --index-url" usecase.

> * Adding version conflict resolution to zc.buildout and/or setuptools. 
> That way you could create meta eggs (e.g. the 'Zope' egg) with '==' 
> version specificers (for Grok, the 'grok' egg would function as the meta 
> egg as well). If this would cause version conflicts (and they often 
> occur in buildout due to the lack of a full dependency tree before 
> download), it should be possible to say which egg's versions are 
> authoritative.

As with an apt / yum repository, a subset could harvest and export the
full dependency graph information for all included distributions,
assuming that setuptools was willing to use such information rather than
incrementally discovering dependencies after download.  I'm not sure
there is much point in trying to compute such a "pickle" across the
whole of PyPI, however.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG+P/I+gerLs4ltQ4RAqnvAJ43njIxLFcs3CeBiPg59EsvIP+RhACg3HZT
/+6YTbnB1V50SkLtYzuNeRc=
=yVfp
-----END PGP SIGNATURE-----


From chrism at plope.com  Wed Sep 26 10:27:32 2007
From: chrism at plope.com (Chris McDonough)
Date: Wed, 26 Sep 2007 04:27:32 -0400
Subject: [Distutils] licenses for ez_setup and virtual_python
Message-ID: <D8177FCD-1E3C-4EC2-B60E-CBEC5588FF19@plope.com>

We'd like to distribute ez_setup.py and virtual_python.py with a good  
number of our packages.  I suspect ez_setup.py is licensed under the  
same dual ZPL/PSF license as is setuptools.  I'm not sure how  
virtual_python.py is licensed (I got it from From http:// 
peak.telecommunity.com/dist/virtual-python.py).

I'd like to do the right thing for licensing and attribution.  What  
would the authors of these packages like to see as far as attribution  
and licensing artifacts in packages that include them?

- C


From faassen at startifact.com  Wed Sep 26 15:11:08 2007
From: faassen at startifact.com (Martijn Faassen)
Date: Wed, 26 Sep 2007 15:11:08 +0200
Subject: [Distutils] install_recommends proposal
Message-ID: <fddlqi$pdt$1@sea.gmane.org>

Hi there,

Currently setup.py allows install_requires to specify the dependencies, 
possibly including exact version numbers of a package.

It is however beneficial to only specify the *minimum* of expected 
requirements in there, to retain maximum flexibility of use. This means 
you'd typically say:

   install_requires = [
      'foo',
      'bar >= 1.3',
   ]

if you know you need some version of foo, and at least version 1.3 of 
your package. This allows developers who use your package to choose 
themselves which version of foo and bar they want to use, without 
getting version conflicts as long as you stay within the stated constraints.

If you write a framework, or a library that uses other libraries, you 
sometimes want to do something more. You'd like to ensure that if 
someone uses version X of your framework, they use a list of *exact* 
dependencies, that is, exactly foo 1.2 and bar 1.4.3. This makes all 
installations of framework X bug for bug compatible. New releases of 
packages you depend on don't break your framework install, as the 
framework install will ensure those versions.

I'd therefore like to propose a new setup.py section where these can be 
specified:

    install_recommends = [
      'foo == 1.2',
      'bar == 1.4.3',
    ]

The person or tool installing the package is entirely free to ignore 
this information; it's a recommendation, not a requirement. The only 
specifications that can be made are *exact*; there is no support for 
anything but '=='. Distutils at most needs to be taught that 
install_recommends cannot directly conflict with install_requires. This 
would be a conflict:

   install_requires = [
      'bar > 1.3'
   ],
   install_recommends = [
      'bar == 1.2',
   ]

By default, distutils and setuptools would otherwise ignore this 
information. I do presume setuptools would include this information in 
egg-info so that external tools can make use of it.

This information can then be (optionally) used by a tool like buildout 
to ensure exact dependencies are installed (once zc.buildout is taught 
to use this information, of course). This means I can release a 
framework that allows flexibility to knowledgable developers, but also, 
in its default install, doesn't have the chance of breaking the moment 
someone releases a new version of some dependency.

For an extensive article where I describe what led me to look for this 
read this:

http://faassen.n--tree.net/blog/view/weblog/2007/09/26/0

It goes into much detail on the concerns and reasoning behind this in 
the context of the development of Zope 3, and Grok, a framework based on 
many Zope 3 packages. This isn't a theoretical problem but is currently 
biting us regularly. I'll also note that it looks like zc.buildout 
doesn't appear to need much work to start using this information (but 
Jim will correct me if I'm wrong).

Regards,

Martijn


From philipp at weitershausen.de  Wed Sep 26 15:39:21 2007
From: philipp at weitershausen.de (Philipp von Weitershausen)
Date: Wed, 26 Sep 2007 15:39:21 +0200
Subject: [Distutils] setuptools fails to install package data from ZIP
	distribution
Message-ID: <fddnjl$vto$1@sea.gmane.org>

Packages with Zope software typically contain lots of package data 
(templates, resources, translations, configuration files). Last night we 
had somebody with a Windows machine make distributions of some of our 
eggs (using python setup.py sdist) for the first time. It turns out, 
'sdist' on Windows creates a ZIP file while on other platforms it 
creates a gzipped tarball. I find that difference in behaviour a bit 
disturbing.

That behaviour isn't the actual root of the problem we experienced, 
however. These ZIP distributions were then uploaded to PyPI and other 
people happily started installing them. The only problem was that on 
Linux and MacOSX, the installed eggs no longer contained any of the 
package data. The distributions (the ZIP files) do contain it, but 
somehow setuptools fails to install it properly.

Is this a known bug?

Is there a reason for the different default behaviour of 'sdist' on the 
different platforms?


-- 
http://worldcookery.com -- Professional Zope documentation and training


From chrism at plope.com  Wed Sep 26 16:07:54 2007
From: chrism at plope.com (Chris McDonough)
Date: Wed, 26 Sep 2007 10:07:54 -0400
Subject: [Distutils] setuptools fails to install package data from ZIP
	distribution
In-Reply-To: <fddnjl$vto$1@sea.gmane.org>
References: <fddnjl$vto$1@sea.gmane.org>
Message-ID: <1C7AADD3-DD96-4551-B2EA-696473B391F5@plope.com>

In my experience, that's usually a problem that happens when you do  
"python setup.py sdist" using:

- a setup.py expects that "find_package_data()" will find package  
data based on searching for things
   checked into version control.

- a source tree that isn't checked out from version control and does  
not have a MANIFEST.in file.

- C



On Sep 26, 2007, at 9:39 AM, Philipp von Weitershausen wrote:

> Packages with Zope software typically contain lots of package data
> (templates, resources, translations, configuration files). Last  
> night we
> had somebody with a Windows machine make distributions of some of our
> eggs (using python setup.py sdist) for the first time. It turns out,
> 'sdist' on Windows creates a ZIP file while on other platforms it
> creates a gzipped tarball. I find that difference in behaviour a bit
> disturbing.
>
> That behaviour isn't the actual root of the problem we experienced,
> however. These ZIP distributions were then uploaded to PyPI and other
> people happily started installing them. The only problem was that on
> Linux and MacOSX, the installed eggs no longer contained any of the
> package data. The distributions (the ZIP files) do contain it, but
> somehow setuptools fails to install it properly.
>
> Is this a known bug?
>
> Is there a reason for the different default behaviour of 'sdist' on  
> the
> different platforms?
>
>
> -- 
> http://worldcookery.com -- Professional Zope documentation and  
> training
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>


From zooko at zooko.com  Wed Sep 26 16:47:43 2007
From: zooko at zooko.com (zooko)
Date: Wed, 26 Sep 2007 08:47:43 -0600
Subject: [Distutils] install_recommends proposal
In-Reply-To: <fddlqi$pdt$1@sea.gmane.org>
References: <fddlqi$pdt$1@sea.gmane.org>
Message-ID: <3394F11D-EAFF-422E-BC6F-DF09E5614271@zooko.com>

The allmydata.org "Tahoe" project does something similar in a non- 
machine-readable way.  We specify a minimum version if one is known,  
and we also mention specific versions that have been tested and are  
known to work:

http://allmydata.org/trac/tahoe/browser/README

The fact that this kind of practice is common suggests that a machine- 
readable implementation of it might be useful.

Regards,

Zooko


From philipp at weitershausen.de  Wed Sep 26 17:39:24 2007
From: philipp at weitershausen.de (Philipp von Weitershausen)
Date: Wed, 26 Sep 2007 17:39:24 +0200
Subject: [Distutils] setuptools fails to install package data from ZIP
	distribution
In-Reply-To: <1C7AADD3-DD96-4551-B2EA-696473B391F5@plope.com>
References: <fddnjl$vto$1@sea.gmane.org>
	<1C7AADD3-DD96-4551-B2EA-696473B391F5@plope.com>
Message-ID: <4A51CE45-BA62-4770-A438-681D534AD646@weitershausen.de>

On 26 Sep 2007, at 16:07 , Chris McDonough wrote:
> In my experience, that's usually a problem that happens when you do  
> "python setup.py sdist" using:
>
> - a setup.py expects that "find_package_data()" will find package  
> data based on searching for things
>   checked into version control.
>
> - a source tree that isn't checked out from version control and  
> does not have a MANIFEST.in file.

I'm fully aware of the problem you're talking about. However, it  
isn't what I'm describing.

In any of the circumstances you mention, the *distribution* won't  
include the package data. In my case, it's the installed *egg* that  
doesn't include package data. The distribution (created with sdist)  
is actually fine, it does include the package data. setuptools just  
doesn't install it into the egg.

For example, download and unzip http://pypi.python.org/packages/ 
source/z/zope.app.securitypolicy/zope.app.securitypolicy-3.4.0.zip.  
You will see it includes all of the extra data files. But when you do

   easy_install zope.app.securitypolicy-3.4.0.zip

(best done in a virtual-python or virtualenv), you'll find that the  
installed egg does not have any of the ZCML files.


From pje at telecommunity.com  Wed Sep 26 19:13:39 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Sep 2007 13:13:39 -0400
Subject: [Distutils] setuptools fails to install package data from ZIP
 distribution
In-Reply-To: <fddnjl$vto$1@sea.gmane.org>
References: <fddnjl$vto$1@sea.gmane.org>
Message-ID: <20070926171103.0A6113A4045@sparrow.telecommunity.com>

At 03:39 PM 9/26/2007 +0200, Philipp von Weitershausen wrote:
>Packages with Zope software typically contain lots of package data
>(templates, resources, translations, configuration files). Last night we
>had somebody with a Windows machine make distributions of some of our
>eggs (using python setup.py sdist) for the first time. It turns out,
>'sdist' on Windows creates a ZIP file while on other platforms it
>creates a gzipped tarball. I find that difference in behaviour a bit
>disturbing.
>
>That behaviour isn't the actual root of the problem we experienced,
>however. These ZIP distributions were then uploaded to PyPI and other
>people happily started installing them. The only problem was that on
>Linux and MacOSX, the installed eggs no longer contained any of the
>package data. The distributions (the ZIP files) do contain it, but
>somehow setuptools fails to install it properly.
>
>Is this a known bug?

It is now.  :)


>Is there a reason for the different default behaviour of 'sdist' on the
>different platforms?

The Windows-created SOURCES.txt contains \r characters, which when 
read on non-Windows platforms causes setuptools to look for files 
with a '\r' on the end, due to this piece of code in the distutils 
"sdist" command (which is reused by setuptools):

     def read_manifest (self):
         """Read the manifest file (named by 'self.manifest') and use it to
         fill in 'self.filelist', the list of files to include in the source
         distribution.
         """
         log.info("reading manifest file '%s'", self.manifest)
         manifest = open(self.manifest)
         while 1:
             line = manifest.readline()
             if line == '':              # end of file
                 break
             if line[-1] == '\n':
                 line = line[0:-1]
             self.filelist.append(line)

This code was apparently a platform-safety issue with the distutils 
all along, but it only affects setuptools because we actually ship 
and use the manifest (as SOURCES.txt) with the source distribution.

I've checked in a fix for both the 0.7 trunk and the 0.6 branch; you 
can update with "easy_install setuptools==dev" or "easy_install 
setuptools==dev06".

The fix is a "belt and braces" one; it ensures that Windows will now 
write SOURCES.txt in a way that unfixed setuptools can read it, AND 
it ensures that non-Windows setuptools can read SOURCES.txt written 
by an unfixed setuptools on Windows.  In other words, as long as 
either the sdist creator or the person installing the sdist have the 
fix, it will work.


From b3i4old02 at sneakemail.com  Wed Sep 26 21:25:28 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Wed, 26 Sep 2007 20:25:28 +0100
Subject: [Distutils] install_recommends proposal
In-Reply-To: <fddlqi$pdt$1@sea.gmane.org>
References: <fddlqi$pdt$1@sea.gmane.org>
Message-ID: <fdebnt$d4p$1@sea.gmane.org>

Martijn Faassen wrote:
> Hi there,
> 
> Currently setup.py allows install_requires to specify the dependencies, 
> possibly including exact version numbers of a package.
> 
> It is however beneficial to only specify the *minimum* of expected 
> requirements in there, to retain maximum flexibility of use. This means 
> you'd typically say:
> 
>    install_requires = [
>       'foo',
>       'bar >= 1.3',
>    ]
> 
> if you know you need some version of foo, and at least version 1.3 of 
> your package. This allows developers who use your package to choose 
> themselves which version of foo and bar they want to use, without 
> getting version conflicts as long as you stay within the stated constraints.

Can't you use the extras_require feature for this?


From tseaver at palladion.com  Wed Sep 26 21:47:05 2007
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 26 Sep 2007 15:47:05 -0400
Subject: [Distutils] Enabling local distutils.cfg usage
Message-ID: <46FAB739.2030203@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

When building a virtual environment, I'd like to be able to store
global distutils configuration options which are custom to that
environment.  However, both setuptools and distutils expect to
read / write that file relative to the directory of
'distutils.__file__', which is located in the *source*
environment under virtualenv 0.8.4::

  $ ~/projects/source/bin/virtualenv /tmp/virtual
  New python executable in /tmp/virtual/bin/python
  Installing setuptools.............done.
  [/tmp]
  $ cd /tmp/virtual/
  [/tmp/virtual]
  $ bin/python -c "import distutils; print distutils.__file__"
  /home/tseaver/projects/source/lib/python2.4/distutils/__init__.pyc

The simplest way I can think of to make 'distutils.__file__'
local to the virtual environment is to make an empty local
'distutils' package, which pulls in the source version by hacking
'__path__'.  However, because the virtual environment forces the
'real_prefix' library so high in the path, I also need to hack my
own library in higher (in 'sitecustomize.py')::

  [/tmp/virtual]
  $ patch -p1 < ../local_distutils.patch
  patching file lib/python2.4/distutils/__init__.py
  patching file lib/python2.4/sitecustomize.py
  [/tmp/virtual]
  $ bin/python -c "import distutils; print distutils.__file__"
  /tmp/virtual/lib/python2.4/distutils/__init__.py
  [/tmp/virtual]

Probably it would make more sense to fix the path in the SITE_PY
inside virtualenv's support-files;  virtualenv could then just
make the near-empty local 'distutils' package with the correct
'__path__'.

I have attached the patch I used to fix up the local environment
('local_distutils.patch' in the example above).


Tres.

- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG+rc5+gerLs4ltQ4RAtySAKCR2RYNq/zJUnEAU/eNz5vsyYZ3ZgCeJ7lB
EO0xuEmqiLKZ86ZaK3CqfeU=
=r8N9
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: local_distutils.patch
Type: text/x-patch
Size: 1007 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070926/67fac2bf/attachment-0001.bin 

From faassen at startifact.com  Wed Sep 26 22:38:49 2007
From: faassen at startifact.com (Martijn Faassen)
Date: Wed, 26 Sep 2007 22:38:49 +0200
Subject: [Distutils] install_recommends proposal
In-Reply-To: <fdebnt$d4p$1@sea.gmane.org>
References: <fddlqi$pdt$1@sea.gmane.org> <fdebnt$d4p$1@sea.gmane.org>
Message-ID: <fdeg1v$sb6$1@sea.gmane.org>

Michael Hoffman wrote:
> Martijn Faassen wrote:
>> Hi there,
>>
>> Currently setup.py allows install_requires to specify the dependencies, 
>> possibly including exact version numbers of a package.
>>
>> It is however beneficial to only specify the *minimum* of expected 
>> requirements in there, to retain maximum flexibility of use. This means 
>> you'd typically say:
>>
>>    install_requires = [
>>       'foo',
>>       'bar >= 1.3',
>>    ]
>>
>> if you know you need some version of foo, and at least version 1.3 of 
>> your package. This allows developers who use your package to choose 
>> themselves which version of foo and bar they want to use, without 
>> getting version conflicts as long as you stay within the stated constraints.
> 
> Can't you use the extras_require feature for this?

It is my understanding that extra_requires is used to express *extra*, 
optional requirements beyond those in install_requires. That's not at 
all what I'm trying to talk about, so my apologies for being so unclear.

install_recommends as I proposed it would be used to offer 
recommendations for the core requirements (or, I guess, also 
recommendations of versions for the extra requirements if desired). So, 
the exact same requirements are expressed, just with an additional 
tightening of version numbers.

I understand how the name "install_recommends" could be confusing 
terminology given the way package managers use the term 'recommended 
packages', which means extra that you could install too to get more 
features. "install_prefers" instead?

An alternative would be to expand the syntax of install_requires and 
extra_require to allow the recommended version number hint. Something 
like this:

install_requires = [
    'foo (1.2.1)',
    'bar >= 1.3 (1.3.2)',
   ]

Installation tools are free to ignore the latter number and just install 
something that fits the basic requirement:

install_requires = [
   'foo',
   'bar >= 1.3',
]

But an installation could be taught to take this into account and get 
those versions, and do the equivalent of this:

install_requires = [
    'foo == 1.2.1',
    'bar == 1.3.2',
   ]

The install tool could choose in the case of conflicting version numbers 
to pick the one in the outer packages. So, if foo depends on bar, and 
bar says it would prefer qux 1.3, while foo actually prefers 1.3.2, foo 
would trump bar.

Again, all this would be entirely optional behavior of the installation 
tool. I'm just asking for a way to express this information in the 
package metadata at all, so that install tools could be taught to use it.

Regards,

Martijn


From pje at telecommunity.com  Thu Sep 27 00:36:47 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Sep 2007 18:36:47 -0400
Subject: [Distutils] install_recommends proposal
In-Reply-To: <fdeg1v$sb6$1@sea.gmane.org>
References: <fddlqi$pdt$1@sea.gmane.org> <fdebnt$d4p$1@sea.gmane.org>
	<fdeg1v$sb6$1@sea.gmane.org>
Message-ID: <20070926223412.B317E3A4045@sparrow.telecommunity.com>

At 10:38 PM 9/26/2007 +0200, Martijn Faassen wrote:
>An alternative would be to expand the syntax of install_requires and
>extra_require to allow the recommended version number hint. Something
>like this:
>
>install_requires = [
>     'foo (1.2.1)',
>     'bar >= 1.3 (1.3.2)',
>    ]
>
>Installation tools are free to ignore the latter number and just install
>something that fits the basic requirement:
>
>install_requires = [
>    'foo',
>    'bar >= 1.3',
>]
>
>But an installation could be taught to take this into account and get
>those versions, and do the equivalent of this:
>
>install_requires = [
>     'foo == 1.2.1',
>     'bar == 1.3.2',
>    ]
>
>The install tool could choose in the case of conflicting version numbers
>to pick the one in the outer packages. So, if foo depends on bar, and
>bar says it would prefer qux 1.3, while foo actually prefers 1.3.2, foo
>would trump bar.

Note that the previous proposal for having a way to specify "or'ed" 
conditions would allow this, but it isn't going to happen until I get 
an 0.6 final out so I can focus on refactoring and new features in 0.7.


From marius at pov.lt  Thu Sep 27 00:38:24 2007
From: marius at pov.lt (Marius Gedminas)
Date: Thu, 27 Sep 2007 01:38:24 +0300
Subject: [Distutils] install_recommends proposal
In-Reply-To: <fdeg1v$sb6$1@sea.gmane.org>
References: <fddlqi$pdt$1@sea.gmane.org> <fdebnt$d4p$1@sea.gmane.org>
	<fdeg1v$sb6$1@sea.gmane.org>
Message-ID: <20070926223824.GA31300@fridge.pov.lt>

On Wed, Sep 26, 2007 at 10:38:49PM +0200, Martijn Faassen wrote:
> I understand how the name "install_recommends" could be confusing 
> terminology

Yes it is

> given the way package managers use the term 'recommended 
> packages', which means extra that you could install too to get more 
> features. "install_prefers" instead?
> 
> An alternative would be to expand the syntax of install_requires and 
> extra_require to allow the recommended version number hint. Something 
> like this:
> 
> install_requires = [
>     'foo (1.2.1)',
>     'bar >= 1.3 (1.3.2)',
>    ]

A really human-readable version would be:

  install_requires = [
      'foo (prefer 1.2.1)',
      'bar >= 1.3 (prefer 1.3.2)',
     ]

Marius Gedminas
-- 
Perl is hard for most people to write. They write PERL or Pearl.
        -- Abigail
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070927/52165d6a/attachment.pgp 

From philipp at weitershausen.de  Thu Sep 27 10:49:08 2007
From: philipp at weitershausen.de (Philipp von Weitershausen)
Date: Thu, 27 Sep 2007 10:49:08 +0200
Subject: [Distutils] setuptools fails to install package data from ZIP
	distribution
In-Reply-To: <20070926171103.0A6113A4045@sparrow.telecommunity.com>
References: <fddnjl$vto$1@sea.gmane.org>
	<20070926171103.0A6113A4045@sparrow.telecommunity.com>
Message-ID: <D7A65049-5FCB-4B04-B159-32737BB53AD0@weitershausen.de>

On 26 Sep 2007, at 19:13 , Phillip J. Eby wrote:
> I've checked in a fix for both the 0.7 trunk and the 0.6 branch;  
> you can update with "easy_install setuptools==dev" or "easy_install  
> setuptools==dev06".

Wow, that was quick. Thanks!




From faassen at startifact.com  Thu Sep 27 13:29:52 2007
From: faassen at startifact.com (Martijn Faassen)
Date: Thu, 27 Sep 2007 13:29:52 +0200
Subject: [Distutils] install_recommends proposal
In-Reply-To: <20070926223412.B317E3A4045@sparrow.telecommunity.com>
References: <fddlqi$pdt$1@sea.gmane.org>
	<fdebnt$d4p$1@sea.gmane.org>	<fdeg1v$sb6$1@sea.gmane.org>
	<20070926223412.B317E3A4045@sparrow.telecommunity.com>
Message-ID: <fdg48o$ecg$1@sea.gmane.org>

Phillip J. Eby wrote:
> At 10:38 PM 9/26/2007 +0200, Martijn Faassen wrote:
[snip]
> Note that the previous proposal for having a way to specify "or'ed" 
> conditions would allow this, but it isn't going to happen until I get 
> an 0.6 final out so I can focus on refactoring and new features in 0.7.

I will look for this proposal, any hints as to where to look?

My first response, not based on actually looking at the proposal is that 
a simple 'or' condition would not be sufficient, as the package manager 
doesnt' know which clause in the condition to look at to find the 
preferred version. My second response upon pondering a bit more is that 
the package manager could be told to always prefer the or-ed condition 
that is most specific.

Do you have any (vague) estimates on timeline before a 0.7 with such a 
feature would become available? I'm asking as ... oh, the pain, the pain! :)

Regards,

Martijn


From philipp at weitershausen.de  Thu Sep 27 13:49:58 2007
From: philipp at weitershausen.de (Philipp von Weitershausen)
Date: Thu, 27 Sep 2007 13:49:58 +0200
Subject: [Distutils] install_recommends proposal
In-Reply-To: <20070926223824.GA31300@fridge.pov.lt>
References: <fddlqi$pdt$1@sea.gmane.org>
	<fdebnt$d4p$1@sea.gmane.org>	<fdeg1v$sb6$1@sea.gmane.org>
	<20070926223824.GA31300@fridge.pov.lt>
Message-ID: <46FB98E6.3070909@weitershausen.de>

Marius Gedminas wrote:
> On Wed, Sep 26, 2007 at 10:38:49PM +0200, Martijn Faassen wrote:
>> I understand how the name "install_recommends" could be confusing 
>> terminology
> 
> Yes it is
> 
>> given the way package managers use the term 'recommended 
>> packages', which means extra that you could install too to get more 
>> features. "install_prefers" instead?
>>
>> An alternative would be to expand the syntax of install_requires and 
>> extra_require to allow the recommended version number hint. Something 
>> like this:
>>
>> install_requires = [
>>     'foo (1.2.1)',
>>     'bar >= 1.3 (1.3.2)',
>>    ]
> 
> A really human-readable version would be:
> 
>   install_requires = [
>       'foo (prefer 1.2.1)',
>       'bar >= 1.3 (prefer 1.3.2)',
>      ]

Or make 'prefer' an actual version operator like == or >=:

   install_requires = [
       'foo prefer 1.2.1',
       'bar >=1.3, prefer 1.3.2',
       ]

The advantage of including it into install_requires is that you avoid 
redudancy by not having to repeat the list of required packages twice.

Another advantage might be that the information would be written to a 
place that we already know (EGG-INFO/requires.txt) and have extraction 
tools for. The only problem with that would be that eggs with a "modern" 
EGG-INFO like that would not be installable on older setuptools versions 
that know nothing about the 'prefer' operator. Then again, that would 
also be the case with Martijn's proposed install_recommends parameter.


-- 
http://worldcookery.com -- Professional Zope documentation and training

From faassen at startifact.com  Thu Sep 27 13:48:33 2007
From: faassen at startifact.com (Martijn Faassen)
Date: Thu, 27 Sep 2007 13:48:33 +0200
Subject: [Distutils] install_recommends proposal
In-Reply-To: <fdg48o$ecg$1@sea.gmane.org>
References: <fddlqi$pdt$1@sea.gmane.org>	<fdebnt$d4p$1@sea.gmane.org>	<fdeg1v$sb6$1@sea.gmane.org>	<20070926223412.B317E3A4045@sparrow.telecommunity.com>
	<fdg48o$ecg$1@sea.gmane.org>
Message-ID: <fdg5bp$hfa$2@sea.gmane.org>

Martijn Faassen wrote:
[snip]
> My first response, not based on actually looking at the proposal is that 
> a simple 'or' condition would not be sufficient, as the package manager 
> doesnt' know which clause in the condition to look at to find the 
> preferred version. My second response upon pondering a bit more is that 
> the package manager could be told to always prefer the or-ed condition 
> that is most specific.

This assumes of course that setuptools has a way to let installation tools:

* access the or clauses

* determine which ones are the most specific

Or has a resolution algorithm for this itself.

We also have to resolve situations where we could have multiple specific 
versions:

Package A says (hypothetical syntax):

    install_requires = [
       'B or B == 1.3',
       'C or C == 1.7',
      ]

and then a package B which says:

     install_requires = [
        'C or C == 1.7.1',
     ]

which one to pick? C will do, but if we want to be specific, should we 
pick C 1.7 or C 1.7.1?

I propose we let the outer package (A) break the contention and thus 
decide on C 1.7. The inner package winning would otherwise block 
framework packages from having the ability to make informed decisions to 
diverge from recommendations lower down the dependency tree.

Regards,

Martijn


From pje at telecommunity.com  Thu Sep 27 18:11:01 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 12:11:01 -0400
Subject: [Distutils] What is the legal status of virtual-python.py
In-Reply-To: <cc0c36330709111033p30b3b1b6vaa1de06ad76da49d@mail.gmail.co
 m>
References: <cc0c36330709111033p30b3b1b6vaa1de06ad76da49d@mail.gmail.com>
Message-ID: <20070927160825.BA6523A407B@sparrow.telecommunity.com>

At 08:33 PM 9/11/2007 +0300, Ignas Mikalajunas wrote:
>  Hello, I would like to reuse pieces of code from
>   http://peak.telecommunity.com/dist/virtual-python.py

As far as my contributions to that code, you may consider them public 
domain.  I suspect Ian would say the same.


From pje at telecommunity.com  Thu Sep 27 18:16:18 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 12:16:18 -0400
Subject: [Distutils] Problem adding a new setup command..
In-Reply-To: <819163.3074.qm@web55604.mail.re4.yahoo.com>
References: <819163.3074.qm@web55604.mail.re4.yahoo.com>
Message-ID: <20070927161343.D54CD3A407B@sparrow.telecommunity.com>

At 05:09 PM 9/11/2007 -0400, Etienne Robillard wrote:
>     % (source, command_name, option))
>distutils.errors.DistutilsOptionError: error in
>command line: command 'install_media' has no such
>option 'a     rgs'
>
>Here's the setuptools.Command subclass:
>
>from setuptools import Command
>class install_media(Command):
>     description = "Install app-specific media files"
>     command_consumes_arguments = True

You only set 'command_consumes_arguments' if it accepts *non-option* 
arguments, in which case your instances must have an 'args' attribute 
initialized in initialize_options.


>     # List of options for this command
>     user_options = [
>         # Perhaps the better option is to use the
>'--prefix' arg.
>         # ('with-docroot=', None, 'Install media files
>to this directory'),
>         ('with-apps=', None,
>          "Search theses subdirs for media files
>(comma-separated list)")
>     ]
>     def initialize_options(self):
>         pass
>     def finalize_options(self):
>         pass

These two methods are broken: initialize_options must set the 
instance's with_docroot and with_apps attributes to None, since those 
are options.  The finalize_options method must set these attributes 
to default values if they are still None.

These requirements come from the distutils, and I believe they are 
documented as such.

(Please note that initializing the attributes to None *anywhere* 
other than initialize_options is NOT sufficient to create a correct 
distutils command class, because then it is impossible to 
"reinitialize" a command instance, which the distutils sometimes needs to do.)


>     def run(self):
>         return self._find_media_files()
>     def _find_media_files(self):
>         pass
>
>
>Any insightful hints how to debug this problem?
>The Big Idea was to add a new setup command (named
>install_media) which could presumely allow things
>like:
>
>  $ python setup.py install_media --prefix=/var/www
>--with-apps=foo,bar

Note that you're only taking *options* here, not arguments.  So you 
don't need or want the command_consumes_arguments flag set.


From pje at telecommunity.com  Thu Sep 27 18:34:35 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 12:34:35 -0400
Subject: [Distutils] easy_install should consider the development status
 of packages
In-Reply-To: <46EB7BEC.2040105@behnel.de>
References: <46EB7BEC.2040105@behnel.de>
Message-ID: <20070927163159.92A033A407B@sparrow.telecommunity.com>

At 08:30 AM 9/15/2007 +0200, Stefan Behnel wrote:
>Hi,
>
>currently, lxml has both a stable release series (1.3.x) and an unstable alpha
>release series (2.0alphaX) on PyPI. When you "easy_install lxml", you get the
>2.0alpha version. I don't think that's what users expect and it's definitely
>not what most users want, but at the same time I would like to keep 2.0
>visible and easy_install-able to make people aware of it and to let them
>decide by themselves.
>
>I would like to have easy_install changed to either consider the Trove
>development status of a package and prefer the highest one (or at least those
>from 5 up), and/or to check the version string for the typical alpha/beta
>substrings and ignore them by default. It would then be easy to add a command
>line option like "--unstable" for those who know better.
>
>Any comments on that?

Patches welcome.  I would prefer to see it implemented as the ability 
to specify a level of stability, so that for example I could specify 
that beta is an okay stability level.

The main place where you should be looking for the patch to work is 
in the fetch_distribution method of the PackageIndex class in 
setuptools.package_index.  There is only one call to this method from 
the easy_install command class, so the hookup should be straightforward.

You'll need to scan the distribution object's "parsed_version" to 
determine its compliance with a particular stability level.  And, to 
get the stability level in the first place, you'll need to use 
pkg_resources.parse_version and take the zeroth element for 
comparison purposes:

 >>> from pkg_resources import parse_version as pv
 >>> pv('a')
('*a', '*final')
 >>> pv('b')
('*b', '*final')
 >>> pv('rc')
('*c', '*final')
 >>> pv('dev')
('*@', '*final')
 >>> pv('final')
('*final', '*final')


The parsed_version of a package will look something like:

 >>> pv('2.0a5')
('00000002', '*a', '00000005', '*final')

 >>> pv('1.3.2')
('00000001', '00000003', '00000002', '*final')

So, to look for stable packages, you want to look for a stability of 
'final'.  A package should be disqualified if it contains any 
non-numeric part that is < the acceptable stability.  Thus, if you 
set a stability of "a", 2.0a6dev-r15435 should fail because:

 >>> pv('2.0a6dev-r15435')
('00000002', '*a', '00000006', '*@', '*final-', '*r', '00015435', '*final')

As you can see, it contains a '*@' to represent the 'dev' part, so it 
lacks alpha stability.

Oh, and don't forget that this stability rejection should be a 
*preference*, i.e. if there is no version matching the stability 
setting and the version requirements, the stability requirement 
should be dropped.


From pje at telecommunity.com  Thu Sep 27 18:51:27 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 12:51:27 -0400
Subject: [Distutils] patches: ez_setup.py: don't import setuptools into
 your current address space in order to learn its version number
In-Reply-To: <3E6E0CC5-FF5D-4127-A4EA-730786508910@zooko.com>
References: <3E6E0CC5-FF5D-4127-A4EA-730786508910@zooko.com>
Message-ID: <20070927164853.16CB93A407B@sparrow.telecommunity.com>

At 03:29 PM 9/18/2007 -0600, zooko wrote:
>Folks:
>
>We're gradually converting the allmydata.org tahoe project [1] and
>its spin-off packages to use setuptools.  One problem that comes up
>is that ez_setup.py might require a newer version of setuptools than
>the version of setuptools already installed.
>
>For example, although Tahoe currently uses setuptools to manage its
>dependencies on zfec, foolscap, and nevow [2], but if we execute
>simplejson's ez_setup.py then it requires setuptools v0.6c7 and
>refuses to proceed if an earlier version is installed.
>
>One solution for this problem could be for the packager of simplejson
>(Bob Ippolito) to use the "min_version" patch that we contributed to
>ez_setup.py [3].  This is assuming that simplejson doesn't actually
>*require* the latest version of setuptools, and we could successfully
>install with a slightly older version, but what if a package actually
>does require a newer version of setuptools than the one that is
>already installed?
>
>The following patch was created by Tahoe contributor Arno Washck and
>then modified by me in order to get around such a problem.  The idea
>is simple enough -- reload setuptools.  There may be some subtleties
>that we still need to work out.  This patch hasn't been tested in its
>current form.

It won't work correctly.  First, pkg_resources is part of setuptools, 
so if you install a new version of setuptools, you have to reload() it too.

Second, it's not *safe* to reload it, if it or setuptools were 
already imported at the time the function is called.  That's because 
easy_install runs the setup.py of a package it's building from 
source.  So if you use easy_install to install a package that needs a 
newer version, reloading pkg_resources or setuptools (and note that 
setuptools is a package with lots of submodules!) will break the host 
easy_install process.

Basically, that's why ez_setup.py is written the way it is.  The best 
case scenario here would be to check if setuptools or pkg_resources 
are already imported, and if not, then the pkg_resources version 
check + reload of pkg_resources would work.

But if they are already imported, there is no way to reload them in a 
way that won't hose something that's already in progress in the 
caller's context.

In other words, we could maybe fix this for "setup.py install", but 
not for easy_install.  I'm not sure how useful that is, though, since 
if you have setuptools installed, why download "foo", unpack it, and 
"setup.py install", when you can just "easy_install foo" in one go?

About the only thing that *might* work would be to have easy_install 
detect the error when trying to build the package, and then download 
a new setuptools and run the setup.py in a subprocess with that 
setuptools on the path, and finish up the current install by 
switching to the new setuptools.

But that's just insane, because what if the user is running e.g., 
easy_install -zmaxd to build a collection of eggs for 
distribution?  It makes no sense to install the new setuptools in the 
*distribution* directory.

So, unfortunately, the end user is the only one who can safely 
upgrade their setuptools version (especially if they're doing it via 
.rpm or .deb or some such!).


From pje at telecommunity.com  Thu Sep 27 19:58:32 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 13:58:32 -0400
Subject: [Distutils] patch: ez_setup.py: separate default version and
 minimum version of setuptools
In-Reply-To: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
Message-ID: <20070927175805.9CD533A407B@sparrow.telecommunity.com>

At 11:33 PM 9/14/2007 -0700, zooko wrote:
>Folks:
>
>Thank you for setuptools!  We are using it in the Allmydata-Tahoe
>project [1].  We've patched ez_setup.py so that installation can
>proceed if an older version of setuptools is already present, as long
>as that version is not too old.  This allows ez_setup.py to work
>seamlessly in more situations.  Please apply.
>
>The patch is in-lined and attached.

It's not clear to me why the patch is necessary.  If you reference a 
specific version of setuptools in your call to ez_setup(), the 
behavior is almost identical.  The only difference is that if there 
is no setuptools installed, then the version you specified in your 
setup.py will be used for the installation.


From pje at telecommunity.com  Thu Sep 27 20:01:40 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 14:01:40 -0400
Subject: [Distutils] easy_install should consider the development status
 of packages
In-Reply-To: <525f23e80709150418j4316762au811371c046b96193@mail.gmail.co
 m>
References: <46EB7BEC.2040105@behnel.de>
	<525f23e80709150418j4316762au811371c046b96193@mail.gmail.com>
Message-ID: <20070927175904.2C5E53A407B@sparrow.telecommunity.com>

At 07:18 AM 9/15/2007 -0400, Alexander Michael wrote:
>A related desire of mine is to have both the latest stable release of
>an egg and the latest development release installed with *the latest
>stable release being the default* (i.e. listed, with all its stable
>dependencies, in easy-install.pth). Something along the lines of an -M
>option, which like -m, performs a multi-version install but doesn't
>touch easy-install.pth. I'm not sure if this is feasible (or maybe it
>is even already possible!), but it sounds nice to me. :)

Have you tried:

   easy_install pkg==unstable pkg==stable


From pje at telecommunity.com  Thu Sep 27 20:09:21 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 14:09:21 -0400
Subject: [Distutils] Git plugin for setuptools
In-Reply-To: <877imcrquf.fsf@enceladus.ygingras.net>
References: <877imcrquf.fsf@enceladus.ygingras.net>
Message-ID: <20070927180644.B62DE3A407C@sparrow.telecommunity.com>

(Please direct emails like this to the distutils-sig in future.)

At 12:54 PM 9/27/2007 -0400, Yannick Gingras wrote:

>Hi,
>   I coded a Git plugin for Setuptools.  I posted to the distutils
>mailing list but my message doesn't seem to show up.  A spam filter
>must have eaten it.
>
>The plugin is here:
>
>   http://ygingras.net/files/gitlsfiles-0.1.1.tar.gz
>   http://ygingras.net/files/gitlsfiles-0.1.1-py2.4.egg
>   http://ygingras.net/files/gitlsfiles-0.1.1-py2.5.egg
>
>I tested it, so did a random beta tester on #python.  Could you review
>it?  It's really simple.  If it looks good, should I distribute it
>as-is or would it be best to merge it in the Setuptools distrubution?

Distribution would be best; I don't want setuptools invoking 
executables on the off-chance that they're installed.  It would add 
overhead to every operation.  To be considered for inclusion in 
setuptools proper, I would need it to perform all its operations 
without needing an external program, and a reference to the file 
format it's using so I don't end up having to support it with no idea 
how it works.

For separately distributed file-finder plugins, it's okay for them to 
depend on external programs, since presumably a user who installs the 
plugin has the needed programs.


From pje at telecommunity.com  Thu Sep 27 20:20:30 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 14:20:30 -0400
Subject: [Distutils] licenses for ez_setup and virtual_python
In-Reply-To: <D8177FCD-1E3C-4EC2-B60E-CBEC5588FF19@plope.com>
References: <D8177FCD-1E3C-4EC2-B60E-CBEC5588FF19@plope.com>
Message-ID: <20070927181753.EE25C3A407B@sparrow.telecommunity.com>

At 04:27 AM 9/26/2007 -0400, Chris McDonough wrote:
>We'd like to distribute ez_setup.py and virtual_python.py with a good
>number of our packages.  I suspect ez_setup.py is licensed under the
>same dual ZPL/PSF license as is setuptools.  I'm not sure how
>virtual_python.py is licensed (I got it from From http://
>peak.telecommunity.com/dist/virtual-python.py).
>
>I'd like to do the right thing for licensing and attribution.  What
>would the authors of these packages like to see as far as attribution
>and licensing artifacts in packages that include them?

ez_setup is under the ZPL/PSF, while virtual-python is public domain 
as far as I'm concerned.


From zooko at zooko.com  Thu Sep 27 20:56:47 2007
From: zooko at zooko.com (zooko)
Date: Thu, 27 Sep 2007 12:56:47 -0600
Subject: [Distutils] patch: ez_setup.py: separate default version and
	minimum version of setuptools
In-Reply-To: <20070927175805.9CD533A407B@sparrow.telecommunity.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
Message-ID: <840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>

>> Thank you for setuptools!  We are using it in the Allmydata-Tahoe
>> project [1].  We've patched ez_setup.py so that installation can
>> proceed if an older version of setuptools is already present, as long
>> as that version is not too old.  This allows ez_setup.py to work
>> seamlessly in more situations.  Please apply.

> It's not clear to me why the patch is necessary.  If you reference  
> a specific version of setuptools in your call to ez_setup(), the  
> behavior is almost identical.  The only difference is that if there  
> is no setuptools installed, then the version you specified in your  
> setup.py will be used for the installation.

The situation was that a user had a version of setuptools installed  
(I think it was v0.6c5), and the ez_setup.py specified setuptools  
v0.6c6 (which was at that time the current version).  So, when the  
user executed "./setup.py", then he got an error saying that it  
wasn't possible to upgrade setuptools after setuptools was already  
loaded.

With our patch, this is no longer a problem -- v0.6c5 is new enough  
to work, and so the setup.py execution proceeds after the check  
without attempting to install a newer version of setuptools.

Regards,

Zooko


From zooko at zooko.com  Thu Sep 27 21:29:13 2007
From: zooko at zooko.com (zooko)
Date: Thu, 27 Sep 2007 13:29:13 -0600
Subject: [Distutils] patches: ez_setup.py: don't import setuptools into
	your current address space in order to learn its version number
In-Reply-To: <20070927164853.16CB93A407B@sparrow.telecommunity.com>
References: <3E6E0CC5-FF5D-4127-A4EA-730786508910@zooko.com>
	<20070927164853.16CB93A407B@sparrow.telecommunity.com>
Message-ID: <36D3AB69-911B-4C82-B3CE-87A3A27B91BC@zooko.com>

> In other words, we could maybe fix this for "setup.py install", but  
> not for easy_install.  I'm not sure how useful that is, though,  
> since if you have setuptools installed, why download "foo", unpack  
> it, and "setup.py install", when you can just "easy_install foo" in  
> one go?

Oh, perhaps this also explains why you didn't understand the use case  
for the previous patch.  I love easy_install, in some cases we and  
our users need to execute "setup.py install" instead.  There are a  
few different reasons for this that I would be happy to go over with  
you, but hopefully as far as patching ez_setup.py, if I can write  
patches which improve this use case without harming other use cases,  
you will accept that some people prefer to execute "./setup.py  
install" and not "easy_install"?

Here is the current README for our decentralized file storage project:

http://allmydata.org/trac/tahoe/browser/README

Here is my personal notes on how I like to use setuptools-packaged  
software under GNU stow:

https://zooko.com/log-2007.html#d2007-06-02

Regards,

Zooko


From pje at telecommunity.com  Thu Sep 27 23:31:58 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 17:31:58 -0400
Subject: [Distutils] patch: ez_setup.py: separate default version and
 minimum version of setuptools
In-Reply-To: <840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
	<840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
Message-ID: <20070927212922.117263A407B@sparrow.telecommunity.com>

At 12:56 PM 9/27/2007 -0600, zooko wrote:
>>>Thank you for setuptools!  We are using it in the Allmydata-Tahoe
>>>project [1].  We've patched ez_setup.py so that installation can
>>>proceed if an older version of setuptools is already present, as long
>>>as that version is not too old.  This allows ez_setup.py to work
>>>seamlessly in more situations.  Please apply.
>
>>It's not clear to me why the patch is necessary.  If you reference
>>a specific version of setuptools in your call to ez_setup(), the
>>behavior is almost identical.  The only difference is that if there
>>is no setuptools installed, then the version you specified in your
>>setup.py will be used for the installation.
>
>The situation was that a user had a version of setuptools installed
>(I think it was v0.6c5), and the ez_setup.py specified setuptools
>v0.6c6 (which was at that time the current version).  So, when the
>user executed "./setup.py", then he got an error saying that it
>wasn't possible to upgrade setuptools after setuptools was already
>loaded.
>
>With our patch, this is no longer a problem -- v0.6c5 is new enough
>to work, and so the setup.py execution proceeds after the check
>without attempting to install a newer version of setuptools.

If you invoke 'use_setuptools("0.6c5")', you would get the same 
behavior with an unpatched ez_setup.py.



From pje at telecommunity.com  Thu Sep 27 23:38:30 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 17:38:30 -0400
Subject: [Distutils] patches: ez_setup.py: don't import setuptools into
 your current address space in order to learn its version number
In-Reply-To: <36D3AB69-911B-4C82-B3CE-87A3A27B91BC@zooko.com>
References: <3E6E0CC5-FF5D-4127-A4EA-730786508910@zooko.com>
	<20070927164853.16CB93A407B@sparrow.telecommunity.com>
	<36D3AB69-911B-4C82-B3CE-87A3A27B91BC@zooko.com>
Message-ID: <20070927213554.8F5AF3A407B@sparrow.telecommunity.com>

At 01:29 PM 9/27/2007 -0600, zooko wrote:
> > In other words, we could maybe fix this for "setup.py install", but
> > not for easy_install.  I'm not sure how useful that is, though,
> > since if you have setuptools installed, why download "foo", unpack
> > it, and "setup.py install", when you can just "easy_install foo" in
> > one go?
>
>Oh, perhaps this also explains why you didn't understand the use case
>for the previous patch.  I love easy_install, in some cases we and
>our users need to execute "setup.py install" instead.

Unless you are using --root or --single-version-externally-managed, 
there is actually no difference: "setup.py install" actually invokes 
the rough equivalent of "easy_install .".


>   There are a
>few different reasons for this that I would be happy to go over with
>you, but hopefully as far as patching ez_setup.py, if I can write
>patches which improve this use case without harming other use cases,
>you will accept that some people prefer to execute "./setup.py
>install" and not "easy_install"?

The only reason (besides habit) to do that is to do a single-version 
installation -- in which case trying to make ez_setup do the right 
thing is a red herring.  To do a single-version install you have to 
either know precisely what you're doing, or else you're building a 
system package -- in which case you'd darn well better have a 
compatible setuptools installed.

To put it another way, I'm not interested in making it easier for 
people to shoot themselves in the foot.  The error message is there 
for good reasons, and I've thought long and hard about ways to get 
rid of it.  I'm willing to listen to new ideas, I just don't think a 
safe solution for upgrading setuptools in-place is possible without 
either multiple processes or multiple interpreters in a single process.


From pje at telecommunity.com  Thu Sep 27 23:41:10 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 27 Sep 2007 17:41:10 -0400
Subject: [Distutils] Git plugin for setuptools
In-Reply-To: <873ax0rmfn.fsf@enceladus.ygingras.net>
References: <877imcrquf.fsf@enceladus.ygingras.net>
	<20070927180644.B62DE3A407C@sparrow.telecommunity.com>
	<873ax0rmfn.fsf@enceladus.ygingras.net>
Message-ID: <20070927213834.4F7F63A407B@sparrow.telecommunity.com>

At 02:30 PM 9/27/2007 -0400, Yannick Gingras wrote:
>"Phillip J. Eby" <pje at telecommunity.com> writes:
>
> > Distribution would be best; I don't want setuptools invoking
> > executables on the off-chance that they're installed.  It would add
> > overhead to every operation.  To be considered for inclusion in
> > setuptools proper, I would need it to perform all its operations
> > without needing an external program, and a reference to the file
> > format it's using so I don't end up having to support it with no idea
> > how it works.
> >
> > For separately distributed file-finder plugins, it's okay for them to
> > depend on external programs, since presumably a user who installs the
> > plugin has the needed programs.
>
>Fair enough.  The git file format is a bit ugly and using git to parse
>it is probably the most sensible thing to do.  External distribution
>is a perfectly reasonable solution.
>
>Is there a naming convention for SCM plugins?  Could Setuptools add a
>list of available plugins on it's website?  I know there is a pluging
>for Mercurial:
>
>   http://cheeseshop.python.org/pypi/hg.setuptools/
>
>But without a naming conventions, I would not be likely to find a
>plugin for a particular SCM even it exists.

Well, you can search for your SCM on PyPI, which works for Mercurial 
and bzr, at least.


From zooko at zooko.com  Fri Sep 28 00:20:48 2007
From: zooko at zooko.com (zooko)
Date: Thu, 27 Sep 2007 16:20:48 -0600
Subject: [Distutils] patch: ez_setup.py: separate default version and
	minimum version of setuptools
In-Reply-To: <20070927212922.117263A407B@sparrow.telecommunity.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
	<840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
	<20070927212922.117263A407B@sparrow.telecommunity.com>
Message-ID: <3B10C4B9-0093-4F61-B99E-3EFEF60AB20B@zooko.com>

> If you invoke 'use_setuptools("0.6c5")', you would get the same  
> behavior with an unpatched ez_setup.py.

That's true, but the oldest version of setuptools that we support is  
0.6a9, which comes with Ubuntu Dapper, but that one can't be  
downloaded from pypi.  Therefore there is no version number that we  
can give to the unpatched ez_setup.py that will both allow our dapper  
users to run "./setup.py install" and allow users without setuptools  
to get it automatically installed when they run "./setup.py install".

With our patch we can specify the required minimum version of  
setuptools (v0.6a9) separately from the default version of setuptools  
(v0.6c7), so that users who have setuptools can carry on as usual,  
and users who don't have setuptools at all get the latest version.

Now, since we are using our patch we have solved this problem for our  
users.  However, since the packages that we depend on don't use this  
patch, we sometimes run into the same problem.  Our product --  
"tahoe" -- uses simplejson, and we have a policy of managing source  
tarballs instead of a set of binary packages, so when we tell  
setuptools that tahoe depends on simplejson, then setuptools  
evaluates simplejson's setup.py, and then stops the build saying that  
simplejson requires setuptools >= v0.6c5.  Our current solution to  
this is to ask our users to manually install simplejson.  I'm hoping  
that a patch like the one we're offering will become widespread so  
that our users can get dependencies like simplejson packaged  
automatically.

The other patch that I'm working on -- the attempt to detect  
setuptools version without importing setuptools -- is intended to  
solve a different problem that we have been faced with.  That is:  
what happens when the version of setuptools that is already installed  
on the user's machine is too old -- it is known not to work -- and  
the user has executed "./setup.py install"?

By the way, in my earlier references to pages about how we use  
setuptools, I forgot to include our packaging policy page:

http://allmydata.org/trac/tahoe/wiki/Packaging

Thank you for your time,

Zooko


From pje at telecommunity.com  Fri Sep 28 20:37:34 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 28 Sep 2007 14:37:34 -0400
Subject: [Distutils] patch: ez_setup.py: separate default version and
 minimum version of setuptools
In-Reply-To: <3B10C4B9-0093-4F61-B99E-3EFEF60AB20B@zooko.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
	<840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
	<20070927212922.117263A407B@sparrow.telecommunity.com>
	<3B10C4B9-0093-4F61-B99E-3EFEF60AB20B@zooko.com>
Message-ID: <20070928183459.2BBB63A4045@sparrow.telecommunity.com>

At 04:20 PM 9/27/2007 -0600, zooko wrote:
> > If you invoke 'use_setuptools("0.6c5")', you would get the same
> > behavior with an unpatched ez_setup.py.
>
>That's true, but the oldest version of setuptools that we support is
>0.6a9, which comes with Ubuntu Dapper, but that one can't be
>downloaded from pypi.  Therefore there is no version number that we
>can give to the unpatched ez_setup.py that will both allow our dapper
>users to run "./setup.py install" and allow users without setuptools
>to get it automatically installed when they run "./setup.py install".

Not so: use_setuptools("0.6a9", 
"http://some.where/to/download/eggs/"), where the URL is the base URL 
of a download location.  It's true that this requires you to *have* 
an egg for the version in question, and make it available to download 
somewhere, but at that point you get a usable result.


>With our patch we can specify the required minimum version of
>setuptools (v0.6a9) separately from the default version of setuptools
>(v0.6c7), so that users who have setuptools can carry on as usual,
>and users who don't have setuptools at all get the latest version.
>
>Now, since we are using our patch we have solved this problem for our
>users.  However, since the packages that we depend on don't use this
>patch, we sometimes run into the same problem.

It's not that they don't use the patch, it's that they don't specify 
the minimum version.  If people supplied the minimum specified 
version, then the patch would be unnecessary.  Conversely, having the 
patch will not do anything for all the packages currently distributed 
that don't specify a version.


>The other patch that I'm working on -- the attempt to detect
>setuptools version without importing setuptools -- is intended to
>solve a different problem that we have been faced with.  That is:
>what happens when the version of setuptools that is already installed
>on the user's machine is too old -- it is known not to work -- and
>the user has executed "./setup.py install"?

And as I mentioned on that thread, that's a lost cause if setuptools 
or pkg_resources are already in sys.modules by the time setup.py is executing.


From zooko at zooko.com  Fri Sep 28 21:31:31 2007
From: zooko at zooko.com (zooko)
Date: Fri, 28 Sep 2007 13:31:31 -0600
Subject: [Distutils] patch: ez_setup.py: separate default version and
	minimum version of setuptools
In-Reply-To: <20070928183459.2BBB63A4045@sparrow.telecommunity.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
	<840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
	<20070927212922.117263A407B@sparrow.telecommunity.com>
	<3B10C4B9-0093-4F61-B99E-3EFEF60AB20B@zooko.com>
	<20070928183459.2BBB63A4045@sparrow.telecommunity.com>
Message-ID: <52835798-1E1B-4342-9EA8-9A27C1A49849@zooko.com>

> Not so: use_setuptools("0.6a9", "http://some.where/to/download/ 
> eggs/"), where the URL is the base URL of a download location.   
> It's true that this requires you to *have* an egg for the version  
> in question, and make it available to download somewhere, but at  
> that point you get a usable result.

Agreed.  So in order to automatically satisfy our dependency on  
simplejson using simplejson's source tarballs, we can either ask Bob  
Ippolito (the author and packager of simplejson) to please switch to  
the patched version of ez_setup.py and pass min_version="0.6a9" to  
it, or else we can ask him to please set his version="0.6a9" and  
start hosting setuptools 0.6a9 eggs for his users.

Do I correctly understand those options?

Thank you for your time.

Regards,

Zooko


From pje at telecommunity.com  Fri Sep 28 22:49:21 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 28 Sep 2007 16:49:21 -0400
Subject: [Distutils] patch: ez_setup.py: separate default version and
 minimum version of setuptools
In-Reply-To: <52835798-1E1B-4342-9EA8-9A27C1A49849@zooko.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
	<840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
	<20070927212922.117263A407B@sparrow.telecommunity.com>
	<3B10C4B9-0093-4F61-B99E-3EFEF60AB20B@zooko.com>
	<20070928183459.2BBB63A4045@sparrow.telecommunity.com>
	<52835798-1E1B-4342-9EA8-9A27C1A49849@zooko.com>
Message-ID: <20070928204705.C916D3A4045@sparrow.telecommunity.com>

At 01:31 PM 9/28/2007 -0600, zooko wrote:
> > Not so: use_setuptools("0.6a9", "http://some.where/to/download/
> > eggs/"), where the URL is the base URL of a download location.
> > It's true that this requires you to *have* an egg for the version
> > in question, and make it available to download somewhere, but at
> > that point you get a usable result.
>
>Agreed.  So in order to automatically satisfy our dependency on
>simplejson using simplejson's source tarballs, we can either ask Bob
>Ippolito (the author and packager of simplejson) to please switch to
>the patched version of ez_setup.py and pass min_version="0.6a9" to
>it, or else we can ask him to please set his version="0.6a9" and
>start hosting setuptools 0.6a9 eggs for his users.
>
>Do I correctly understand those options?

Correct.  Although the hosting could certainly be 
shared.  Personally, I have a bit of a conflict of interest in all 
this -- i.e., I'd much rather that people upgrade to later versions.

So bear in mind that since it's already *possible* to do this, it's 
unlikely I'll accept a patch that makes it *easy* for people to keep 
using outdated alphas.  :)


From matt at matt-good.net  Fri Sep 28 23:07:59 2007
From: matt at matt-good.net (Matt Good)
Date: Fri, 28 Sep 2007 14:07:59 -0700
Subject: [Distutils] "activate" script for virtualenv
Message-ID: <930E196B-7BB8-4490-8708-3E9CF41F56D5@matt-good.net>

I just set up a little script so that I can activate a particular  
virtualenv and easily exit it.  Basically it just invokes a new bash  
subshell and sets the PATH to include the current bin dir:

mgood at sjl-dev1:~/python-envs/trac$ cat activate
bash --init-file .bash_init
mgood at sjl-dev1:~/python-envs/trac$ cat .bash_init
. ~/.bash_profile
export PATH="`pwd`/bin:$PATH"

So running ./activate will set up your path in the subshell, and  
exiting will restore your normal PATH.  I don't really like running  
~/.bash_profile again b/c I actually end up with it re-adding some  
stuff that's already in the PATH, but otherwise I lost my normal  
prompt coloring.  Someone with more bash-fu know how to fix that?

mgood at sjl-dev1:~/python-envs/trac$ which python
/usr/local/bin/python
mgood at sjl-dev1:~/python-envs/trac$ ./activate
mgood at sjl-dev1:~/python-envs/trac$ which python
/home/mgood/python-envs/trac/bin/python
mgood at sjl-dev1:~/python-envs/trac$ exit
mgood at sjl-dev1:~/python-envs/trac$ which python
/usr/local/bin/python

-- Matt

From zooko at zooko.com  Sat Sep 29 00:43:36 2007
From: zooko at zooko.com (zooko)
Date: Fri, 28 Sep 2007 16:43:36 -0600
Subject: [Distutils] patches: ez_setup.py: don't import setuptools into
	your current address space in order to learn its version number
In-Reply-To: <20070927164853.16CB93A407B@sparrow.telecommunity.com>
References: <3E6E0CC5-FF5D-4127-A4EA-730786508910@zooko.com>
	<20070927164853.16CB93A407B@sparrow.telecommunity.com>
Message-ID: <63A5DBA9-C3E7-484D-A282-5D64C1D118A9@zooko.com>

On Sep 27, 2007, at 10:51 AM, Phillip J. Eby wrote:

> It won't work correctly.  First, pkg_resources is part of  
> setuptools, so if you install a new version of setuptools, you have  
> to reload() it too.
>
> Second, it's not *safe* to reload it, if it or setuptools were  
> already imported at the time the function is called.  That's  
> because easy_install runs the setup.py of a package it's building  
> from source.  So if you use easy_install to install a package that  
> needs a newer version, reloading pkg_resources or setuptools (and  
> note that setuptools is a package with lots of submodules!) will  
> break the host easy_install process.

Okay.  Attached is a patch that doesn't import pkg_resources and  
doesn't reload anything.  I haven't yet tested it -- I am submitting  
it only to inform the discussion and get any early feedback from you  
on whether the very idea is sound.

Regards,

Zooko

diff -rN -u old-up/setuptools-0.6c7/ez_setup.py new-up/ 
setuptools-0.6c7/ez_setup.py
--- old-up/setuptools-0.6c7/ez_setup.py	2007-09-28 16:41:24.000000000  
-0600
+++ new-up/setuptools-0.6c7/ez_setup.py	2007-09-28 16:41:25.000000000  
-0600
@@ -1,4 +1,4 @@
-#!python
+#!/usr/bin/env python
"""Bootstrap setuptools installation
If you want to use setuptools in your package's setup.py, just  
include this
@@ -13,7 +13,7 @@
This file can also be run as a script to install or upgrade setuptools.
"""
-import sys
+import os, re, subprocess, sys
DEFAULT_VERSION = "0.6c7"
DEFAULT_URL     = "http://pypi.python.org/packages/%s/s/setuptools/"  
% sys.version[:3]
@@ -44,8 +44,6 @@
      'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
}
-import sys, os
-
def _validate_md5(egg_name, data):
      if egg_name in md5_data:
          from md5 import md5
@@ -58,6 +56,42 @@
              sys.exit(2)
      return data
+# The following code to parse versions is copied from  
pkg_resources.py so that
+# we can parse versions without importing that module.
+component_re = re.compile(r'(\d+ | [a-z]+ | \.| -)', re.VERBOSE)
+replace = {'pre':'c',  
'preview':'c','-':'final-','rc':'c','dev':'@'}.get
+
+def _parse_version_parts(s):
+    for part in component_re.split(s):
+        part = replace(part,part)
+        if not part or part=='.':
+            continue
+        if part[:1] in '0123456789':
+            yield part.zfill(8)    # pad for numeric comparison
+        else:
+            yield '*'+part
+
+    yield '*final'  # ensure that alpha/beta/candidate are before final
+
+def parse_version(s):
+    parts = []
+    for part in _parse_version_parts(s.lower()):
+        if part.startswith('*'):
+            if part<'*final':   # remove '-' before a prerelease tag
+                while parts and parts[-1]=='*final-': parts.pop()
+            # remove trailing zeros from each series of numeric parts
+            while parts and parts[-1]=='00000000':
+                parts.pop()
+        parts.append(part)
+    return tuple(parts)
+
+def setuptools_is_new_enough(required_version):
+    """Return True if setuptools is already installed and has a version
+    number >= required_version."""
+    sub = subprocess.Popen([sys.executable, "-c", "import  
setuptools;print setuptools.__version__"], stdout=subprocess.PIPE)
+    verstr = sub.stdout.read().strip()
+    ver = parse_version(verstr)
+    return ver and ver >= parse_version(required_version)
def use_setuptools(
      version=DEFAULT_VERSION, download_base=DEFAULT_URL,  
to_dir=os.curdir,
@@ -74,32 +108,11 @@
      this routine will print a message to ``sys.stderr`` and raise  
SystemExit in
      an attempt to abort the calling script.
      """
-    try:
-        import setuptools
-        if setuptools.__version__ == '0.0.1':
-            print >>sys.stderr, (
-            "You have an obsolete version of setuptools installed.   
Please\n"
-            "remove it from your system entirely before rerunning  
this script."
-            )
-            sys.exit(2)
-    except ImportError:
+    if not setuptools_is_new_enough(version):
          egg = download_setuptools(version, download_base, to_dir,  
download_delay)
          sys.path.insert(0, egg)
          import setuptools; setuptools.bootstrap_install_from = egg
-    import pkg_resources
-    try:
-        pkg_resources.require("setuptools>="+version)
-
-    except pkg_resources.VersionConflict, e:
-        # XXX could we install in a subprocess here?
-        print >>sys.stderr, (
-            "The required version of setuptools (>=%s) is not  
available, and\n"
-            "can't be installed while this script is running. Please  
install\n"
-            " a more recent version first.\n\n(Currently using %r)"
-        ) % (version, e.args[0])
-        sys.exit(2)
-
def download_setuptools(
      version=DEFAULT_VERSION, download_base=DEFAULT_URL,  
to_dir=os.curdir,
      delay = 15
@@ -150,9 +163,14 @@
def main(argv, version=DEFAULT_VERSION):
      """Install or upgrade setuptools and EasyInstall"""
-    try:
-        import setuptools
-    except ImportError:
+    if setuptools_is_new_enough(version):
+        if argv:
+            from setuptools.command.easy_install import main
+            main(argv)
+        else:
+            print "Setuptools version",version,"or greater has been  
installed."
+            print '(Run "ez_setup.py -U setuptools" to reinstall or  
upgrade.)'
+    else:
          egg = None
          try:
              egg = download_setuptools(version, delay=0)
@@ -162,31 +180,6 @@
          finally:
              if egg and os.path.exists(egg):
                  os.unlink(egg)
-    else:
-        if setuptools.__version__ == '0.0.1':
-            # tell the user to uninstall obsolete version
-            use_setuptools(version)
-
-    req = "setuptools>="+version
-    import pkg_resources
-    try:
-        pkg_resources.require(req)
-    except pkg_resources.VersionConflict:
-        try:
-            from setuptools.command.easy_install import main
-        except ImportError:
-            from easy_install import main
-        main(list(argv)+[download_setuptools(delay=0)])
-        sys.exit(0) # try to force an exit
-    else:
-        if argv:
-            from setuptools.command.easy_install import main
-            main(argv)
-        else:
-            print "Setuptools version",version,"or greater has been  
installed."
-            print '(Run "ez_setup.py -U setuptools" to reinstall or  
upgrade.)'
-
-
def update_md5(filenames):
      """Update our built-in md5 registry"""

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ez_setup-subp.patch.txt
Url: http://mail.python.org/pipermail/distutils-sig/attachments/20070928/6555aa54/attachment.txt 

From zooko at zooko.com  Sat Sep 29 00:46:27 2007
From: zooko at zooko.com (zooko)
Date: Fri, 28 Sep 2007 16:46:27 -0600
Subject: [Distutils] patch: ez_setup.py: separate default version and
	minimum version of setuptools
In-Reply-To: <20070928204705.C916D3A4045@sparrow.telecommunity.com>
References: <180CEE29-A2BF-4C88-A4AB-CAA486DD0CCD@zooko.com>
	<20070927175805.9CD533A407B@sparrow.telecommunity.com>
	<840FB88B-8413-4340-AD0B-4F8EABCFAB51@zooko.com>
	<20070927212922.117263A407B@sparrow.telecommunity.com>
	<3B10C4B9-0093-4F61-B99E-3EFEF60AB20B@zooko.com>
	<20070928183459.2BBB63A4045@sparrow.telecommunity.com>
	<52835798-1E1B-4342-9EA8-9A27C1A49849@zooko.com>
	<20070928204705.C916D3A4045@sparrow.telecommunity.com>
Message-ID: <26A75FA7-3B5A-4309-BF61-ECCB8A46C750@zooko.com>

>> Agreed.  So in order to automatically satisfy our dependency on
>> simplejson using simplejson's source tarballs, we can either ask Bob
>> Ippolito (the author and packager of simplejson) to please switch to
>> the patched version of ez_setup.py and pass min_version="0.6a9" to
>> it, or else we can ask him to please set his version="0.6a9" and
>> start hosting setuptools 0.6a9 eggs for his users.

...

> So bear in mind that since it's already *possible* to do this, it's  
> unlikely I'll accept a patch that makes it *easy* for people to  
> keep using outdated alphas.  :)

Oh yes, we too prefer for our users to use newer versions of  
setuptools.  I sympathize with your motivation.  :-)

Actually I think this patch makes it easier in some cases for people  
to gracefully upgrade to newer versions of setuptools.

Consider our case: we are required to service users of Ubuntu dapper  
and to do so without requiring them to first upgrade their version of  
setuptools.  If we use standard ez_setup.py, then what happens is  
that all of our users (not just the Ubuntu dapper using ones) get  
setuptools v0.6a9 installed.  Eventually, our users will upgrade from  
Ubuntu dapper to a newer Ubuntu, and they will upgrade their version  
of setuptools when they upgrade their operating system, but our other  
users -- the ones whose version of setuptools is automatically  
installed by our ez_setup.py script -- will be stuck with v0.6a9.

So in order to support our dapper users while also installing a newer  
version of setuptools for non-dapper-users, we patched our  
ez_setup.py.  Now dapper users proceed as normal, but all our other  
users get setuptools v0.6c7 installed.

This is already good enough for us -- we don't mind maintaining a  
slightly different version of ez_setup.py than the official one.  The  
reason why this patch going into the official ez_setup.py would help  
us is that if packages like simplejson used it (and allowed a  
sufficiently old version of setuptools) then we could add simplejson  
to the list of dependencies that are automatically satisfied by  
setuptools even while satisfying the above-mentioned policies.

By the way, I'm separately submitting a patch which, if it can be  
made to work, might allow us to automatically upgrade people to newer  
versions of setuptools when they follow our installation procedure.

Regards,

Zooko


From me at rpatterson.net  Sun Sep 30 23:45:11 2007
From: me at rpatterson.net (Ross Patterson)
Date: Sun, 30 Sep 2007 14:45:11 -0700
Subject: [Distutils] zc.recipe.egg extension supporting "easy_install
	--editable" and running arbitrary setup.py commands
Message-ID: <87myv3keu0.fsf@superfluity.lefae.org>

An embedded and charset-unspecified text was scrubbed...
Name: editable.txt
Url: http://mail.python.org/pipermail/distutils-sig/attachments/20070930/574bda09/attachment.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: setup.txt
Url: http://mail.python.org/pipermail/distutils-sig/attachments/20070930/574bda09/attachment-0001.txt