From lists at zopyx.com  Sat Mar  1 15:58:55 2008
From: lists at zopyx.com (Andreas Jung)
Date: Sat, 01 Mar 2008 15:58:55 +0100
Subject: [Distutils] [PyPI] Checking reST compliance of long description
Message-ID: <766446A680C4529AFD369804@[192.168.0.24]>

Hi,

I just tried to upload a new package with a pretty long long_description.
After uploading PyPI showed the text as plain/text...obviously an error
within the reST structure. Because of the length it took me half an hour 
for figuring out the error. My local rst2html script did not report any 
errors.
So what is the recommended approach for checking a reST file for PyPI 
compliance instead doing fixing-and-re-registering the package?

Andreas

-- 
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080301/2eed1be3/attachment.pgp 

From jim at zope.com  Sat Mar  1 16:26:01 2008
From: jim at zope.com (Jim Fulton)
Date: Sat, 1 Mar 2008 10:26:01 -0500
Subject: [Distutils] [PyPI] Checking reST compliance of long description
In-Reply-To: <766446A680C4529AFD369804@[192.168.0.24]>
References: <766446A680C4529AFD369804@[192.168.0.24]>
Message-ID: <CC26618E-A978-40B1-AD5C-E76679846E57@zope.com>


On Mar 1, 2008, at 9:58 AM, Andreas Jung wrote:

> Hi,
>
> I just tried to upload a new package with a pretty long  
> long_description.
> After uploading PyPI showed the text as plain/text...obviously an  
> error
> within the reST structure. Because of the length it took me half an  
> hour for figuring out the error. My local rst2html script did not  
> report any errors.

That's very odd.

> So what is the recommended approach for checking a reST file for  
> PyPI compliance instead doing fixing-and-re-registering the package?


I have my setup file write the long description to a file and then I  
test it with rest2html (actually the rst2 script from zc.rst2) before  
uploading it. It's puzzling to me that rest2html didn't give you an  
error when PyPI didn't like the text.

Jim

--
Jim Fulton
Zope Corporation



From ziade.tarek at gmail.com  Sat Mar  1 16:32:29 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 1 Mar 2008 16:32:29 +0100
Subject: [Distutils] [PyPI] Checking reST compliance of long description
In-Reply-To: <CC26618E-A978-40B1-AD5C-E76679846E57@zope.com>
References: <766446A680C4529AFD369804@192.168.0.24>
	<CC26618E-A978-40B1-AD5C-E76679846E57@zope.com>
Message-ID: <94bdd2610803010732q1a57e215kc675c0c37f821715@mail.gmail.com>

On Sat, Mar 1, 2008 at 4:26 PM, Jim Fulton <jim at zope.com> wrote:

>
> On Mar 1, 2008, at 9:58 AM, Andreas Jung wrote:
>
> > Hi,
> >
> > I just tried to upload a new package with a pretty long
> > long_description.
> > After uploading PyPI showed the text as plain/text...obviously an
> > error
> > within the reST structure. Because of the length it took me half an
> > hour for figuring out the error. My local rst2html script did not
> > report any errors.
>
> That's very odd.
>
> > So what is the recommended approach for checking a reST file for
> > PyPI compliance instead doing fixing-and-re-registering the package?
>
>
> I have my setup file write the long description to a file and then I
> test it with rest2html (actually the rst2 script from zc.rst2) before
> uploading it. It's puzzling to me that rest2html didn't give you an
> error when PyPI didn't like the text.
>

Yes I wish Pypi would return some kind of warning when the reST compiling
fails
(see my previous message on this)...

A simple check on client side I do with docutils is :

python setup.py --long-description | rst2html.py > /dev/null




> Jim
>
> --
> Jim Fulton
> Zope Corporation
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080301/9fc5fd92/attachment.htm 

From chrism at plope.com  Sat Mar  1 19:00:58 2008
From: chrism at plope.com (Chris McDonough)
Date: Sat, 01 Mar 2008 13:00:58 -0500
Subject: [Distutils] run zc.recipe.egg tests...
Message-ID: <47C999DA.20404@plope.com>

When I run the zc.recipe.egg-1.0.0 tests via "setup.py test" (after unpacking 
from the distribution tarball), I get the below-mentioned failures.  Is there 
another way I should be running the tests so they pass, or are these failures 
genuine?

- VC

"""
[chrism at vitaminf zc.recipe.egg-1.0.0]$ python setup.py test
running test
running egg_info
writing requirements to src/zc.recipe.egg.egg-info/requires.txt
writing src/zc.recipe.egg.egg-info/PKG-INFO
writing namespace_packages to src/zc.recipe.egg.egg-info/namespace_packages.txt
writing top-level names to src/zc.recipe.egg.egg-info/top_level.txt
writing dependency_links to src/zc.recipe.egg.egg-info/dependency_links.txt
writing entry points to src/zc.recipe.egg.egg-info/entry_points.txt
reading manifest file 'src/zc.recipe.egg.egg-info/SOURCES.txt'
writing manifest file 'src/zc.recipe.egg.egg-info/SOURCES.txt'
running build_ext
Doctest: README.txt ... FAIL
Doctest: api.txt ... ok
Doctest: custom.txt ... ok
Doctest: selecting-python.txt ... 'import site' failed; use -v for traceback
'import site' failed; use -v for traceback
'import site' failed; use -v for traceback
'import site' failed; use -v for traceback
'import site' failed; use -v for traceback
FAIL

======================================================================
FAIL: Doctest: README.txt
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 2323, in runTest
     raise self.failureException(self.format_failure(new.getvalue()))
DocTestFailureException: Failed doctest test for README.txt
   File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/README.txt", 
line 0

----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/README.txt", 
line 78, in README.txt
Failed example:
     ls(sample_buildout, 'eggs')
Expected:
     -  demo-0.2-pyN.N.egg
     -  demoneeded-1.2c1-pyN.N.egg
     setuptools-X-pyN.N.egg
     -  zc.buildout.egg
Got:
     -  demo-0.2-pyN.N.egg
     -  demoneeded-1.2c1-pyN.N.egg
     setuptools-X-pyN.N.egg
     d  zc.buildout.egg
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/README.txt", 
line 249, in README.txt
Failed example:
     ls(sample_buildout, 'eggs')
Expected:
     -  demo-0.2-pyN.N.egg
     -  demoneeded-1.2c1-pyN.N.egg
     setuptools-X-pyN.N.egg
     -  zc.buildout.egg
Got:
     -  demo-0.2-pyN.N.egg
     -  demoneeded-1.2c1-pyN.N.egg
     setuptools-X-pyN.N.egg
     d  zc.buildout.egg
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/README.txt", 
line 266, in README.txt
Failed example:
     ls(sample_buildout, 'eggs')
Expected:
     -  demo-0.2-pyN.N.egg
     -  demo-0.4c1-pyN.N.egg
     -  demoneeded-1.2c1-pyN.N.egg
     setuptools-X-pyN.N.egg
     -  zc.buildout.egg
Got:
     -  demo-0.2-pyN.N.egg
     -  demo-0.4c1-pyN.N.egg
     -  demoneeded-1.2c1-pyN.N.egg
     setuptools-X-pyN.N.egg
     d  zc.buildout.egg


======================================================================
FAIL: Doctest: selecting-python.txt
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 2323, in runTest
     raise self.failureException(self.format_failure(new.getvalue()))
DocTestFailureException: Failed doctest test for selecting-python.txt
   File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 0

----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 12, in selecting-python.txt
Failed example:
     print get(link_server),
Expected:
     <html><body>
     <a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
     <a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
     <a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
     <a href="demo-0.4c1-py2.3.egg">demo-0.4c1-py2.3.egg</a><br>
     <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>
     <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>
     <a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br>
     <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>
     <a href="index/">index/</a><br>
     <a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
     </body></html>
Got:
     <html><body>
     <a href="demo-0.1py2.4.egg">demo-0.1py2.4.egg</a><br>
     <a href="demo-0.2py2.4.egg">demo-0.2py2.4.egg</a><br>
     <a href="demo-0.3py2.4.egg">demo-0.3py2.4.egg</a><br>
     <a href="demo-0.4c1py2.4.egg">demo-0.4c1py2.4.egg</a><br>
     <a href="demoneeded-1.0.zip">demoneeded-1.0.zip</a><br>
     <a href="demoneeded-1.1.zip">demoneeded-1.1.zip</a><br>
     <a href="demoneeded-1.2c1.zip">demoneeded-1.2c1.zip</a><br>
     <a href="extdemo-1.4.zip">extdemo-1.4.zip</a><br>
     <a href="index/">index/</a><br>
     <a href="other-1.0py2.4.egg">other-1.0py2.4.egg</a><br>
     </body></html>
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 29, in selecting-python.txt
Failed example:
     write(sample_buildout, 'buildout.cfg',
     """
     [buildout]
     parts = demo
     eggs-directory = eggs
     index = http://www.python.org/pypi/

     [python2.3]
     executable = %(python23)s

     [demo]
     recipe = zc.recipe.egg
     eggs = demo <0.3
     find-links = %(server)s
     python = python2.3
     interpreter = py-demo
     """ % dict(server=link_server, python23=python2_3_executable))
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[1]>", line 17, in ?
     NameError: name 'python2_3_executable' is not defined
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 52, in selecting-python.txt
Failed example:
     print system(buildout),
Expected:
     Installing demo.
     Getting distribution for 'demo<0.3'.
     Got demo 0.2.
     Getting distribution for 'demoneeded'.
     Getting distribution for 'setuptools'.
     Got setuptools V
     Got demoneeded 1.2c1.
     Generated script '/sample-buildout/bin/demo'.
     Generated interpreter '/sample-buildout/bin/py-demo'.
Got nothing
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 65, in selecting-python.txt
Failed example:
     ls(sample_buildout, 'eggs')
Expected:
     -  demo-0.2-py2.3.egg
     -  demoneeded-1.2c1-py2.3.egg
     setuptools-V-py2.3.egg
     setuptools-Vpy2.4.egg
     -  zc.buildout.egg
Got:
     setuptools-Vpy2.4.egg
     d  zc.buildout.egg
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 79, in selecting-python.txt
Failed example:
     f = open(os.path.join(sample_buildout, 'bin', script_name))
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[9]>", line 1, in ?
     IOError: [Errno 2] No such file or directory: 
'/var/folders/dB/dByJ-qkiE6igZD4Yrm+nMk+++TI/-Tmp-/tmpY4S2YlbuildoutSetUp/_TEST_/sample-buildout/bin/demo'
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 80, in selecting-python.txt
Failed example:
     f.readline().strip() == '#!' + python2_3_executable
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[10]>", line 1, in ?
     NameError: name 'f' is not defined
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 82, in selecting-python.txt
Failed example:
     print f.read(), # doctest: +NORMALIZE_WHITESPACE
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[11]>", line 1, in ?
     NameError: name 'f' is not defined
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 95, in selecting-python.txt
Failed example:
     if sys.platform == 'win32':
         f = open(os.path.join(sample_buildout, 'bin', 'py-demo-script.py'))
     else:
         f = open(os.path.join(sample_buildout, 'bin', 'py-demo'))
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[12]>", line 4, in ?
     IOError: [Errno 2] No such file or directory: 
'/var/folders/dB/dByJ-qkiE6igZD4Yrm+nMk+++TI/-Tmp-/tmpY4S2YlbuildoutSetUp/_TEST_/sample-buildout/bin/py-demo'
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 99, in selecting-python.txt
Failed example:
     f.readline().strip() == '#!' + python2_3_executable
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[13]>", line 1, in ?
     NameError: name 'f' is not defined
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 101, in selecting-python.txt
Failed example:
     print f.read(), # doctest: +NORMALIZE_WHITESPACE
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[14]>", line 1, in ?
     NameError: name 'f' is not defined
----------------------------------------------------------------------
File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/src/zc/recipe/egg/selecting-python.txt", 
line 128, in selecting-python.txt
Failed example:
     f.close()
Exception raised:
     Traceback (most recent call last):
       File 
"/Users/chrism/projects/repoze-devel/tmp/zc.recipe.egg-1.0.0/zope.testing-3.5.1-py2.4.egg/zope/testing/doctest.py", 
line 1356, in __run
         compileflags, 1) in test.globs
       File "<doctest selecting-python.txt[15]>", line 1, in ?
     NameError: name 'f' is not defined


----------------------------------------------------------------------
Ran 4 tests in 41.042s

FAILED (failures=2)
"""


From jim at zope.com  Sat Mar  1 20:44:47 2008
From: jim at zope.com (Jim Fulton)
Date: Sat, 1 Mar 2008 14:44:47 -0500
Subject: [Distutils] run zc.recipe.egg tests...
In-Reply-To: <47C999DA.20404@plope.com>
References: <47C999DA.20404@plope.com>
Message-ID: <488093E6-E314-49B6-9538-66502B0657FC@zope.com>

On Mar 1, 2008, at 1:00 PM, Chris McDonough wrote:

> When I run the zc.recipe.egg-1.0.0 tests via "setup.py test" (after  
> unpacking
> from the distribution tarball), I get the below-mentioned failures.   
> Is there
> another way I should be running the tests so they pass, or are these  
> failures
> genuine?


Unfortunately, the tests can only be run from a buildout of the recipe  
project.  I realize that this is not ideal.

Jim

--
Jim Fulton
Zope Corporation



From stephenemslie at gmail.com  Mon Mar  3 11:58:33 2008
From: stephenemslie at gmail.com (stephen emslie)
Date: Mon, 3 Mar 2008 10:58:33 +0000
Subject: [Distutils] [PyPI] Checking reST compliance of long description
In-Reply-To: <94bdd2610803010732q1a57e215kc675c0c37f821715@mail.gmail.com>
References: <766446A680C4529AFD369804@192.168.0.24>
	<CC26618E-A978-40B1-AD5C-E76679846E57@zope.com>
	<94bdd2610803010732q1a57e215kc675c0c37f821715@mail.gmail.com>
Message-ID: <51f97e530803030258g4d81adcck271c63b49f376bac@mail.gmail.com>

This could be due to a bug in the way that distutils escapes (folds)
whitespace in the long_description field. I have seen that cause
errors when rst2html expects indentation which has been lost in the
escaping process.

For more information see: http://bugs.python.org/issue1923 (meaningful
whitespace can be lost in rfc822_escape). There is also a patch there
which you could apply and see if it solves your problem.

It looks like there may be another issue with the unfolding of this
field: http://bugs.python.org/issue504152.

It would be great to see these bugs addressed.

Stephen Emslie

On Sat, Mar 1, 2008 at 3:32 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>
> On Sat, Mar 1, 2008 at 4:26 PM, Jim Fulton <jim at zope.com> wrote:
> >
> >
> >
> > On Mar 1, 2008, at 9:58 AM, Andreas Jung wrote:
> >
> > > Hi,
> > >
> > > I just tried to upload a new package with a pretty long
> > > long_description.
> > > After uploading PyPI showed the text as plain/text...obviously an
> > > error
> > > within the reST structure. Because of the length it took me half an
> > > hour for figuring out the error. My local rst2html script did not
> > > report any errors.
> >
> > That's very odd.
> >
> >
> >
> > > So what is the recommended approach for checking a reST file for
> > > PyPI compliance instead doing fixing-and-re-registering the package?
> >
> >
> > I have my setup file write the long description to a file and then I
> > test it with rest2html (actually the rst2 script from zc.rst2) before
> > uploading it. It's puzzling to me that rest2html didn't give you an
> > error when PyPI didn't like the text.
> >
>
> Yes I wish Pypi would return some kind of warning when the reST compiling
> fails
> (see my previous message on this)...
>
> A simple check on client side I do with docutils is :
>
> python setup.py --long-description | rst2html.py > /dev/null
>
>
>
> >
> > Jim
> >
> > --
> > Jim Fulton
> > Zope Corporation
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > http://mail.python.org/mailman/listinfo/distutils-sig
> >
>
>
>
> --
> Tarek Ziad? | Association AfPy | www.afpy.org
> Blog FR | http://programmation-python.org
>  Blog EN | http://tarekziade.wordpress.com/
> _______________________________________________
>  Distutils-SIG maillist  -  Distutils-SIG at python.org
>  http://mail.python.org/mailman/listinfo/distutils-sig
>
>

From lists at zopyx.com  Mon Mar  3 12:46:24 2008
From: lists at zopyx.com (Andreas Jung)
Date: Mon, 03 Mar 2008 12:46:24 +0100
Subject: [Distutils] [PyPI] Checking reST compliance of long description
In-Reply-To: <51f97e530803030258g4d81adcck271c63b49f376bac@mail.gmail.com>
References: <766446A680C4529AFD369804@192.168.0.24>
	<CC26618E-A978-40B1-AD5C-E76679846E57@zope.com>
	<94bdd2610803010732q1a57e215kc675c0c37f821715@mail.gmail.com>
	<51f97e530803030258g4d81adcck271c63b49f376bac@mail.gmail.com>
Message-ID: <93E7F05F6E9036C3FE63B2A1@[192.168.0.24]>



--On 3. M?rz 2008 10:58:33 +0000 stephen emslie <stephenemslie at gmail.com> 
wrote:

> This could be due to a bug in the way that distutils escapes (folds)
> whitespace in the long_description field. I have seen that cause
> errors when rst2html expects indentation which has been lost in the
> escaping process.
>
> For more information see: http://bugs.python.org/issue1923 (meaningful
> whitespace can be lost in rfc822_escape). There is also a patch there
> which you could apply and see if it solves your problem.
>
> It looks like there may be another issue with the unfolding of this
> field: http://bugs.python.org/issue504152.
>

As part of my work on haufe.releaser I added new setuptools command
'check_description' that performs a reST check before uploading a
dist file:

<http://pypi.python.org/pypi/haufe.releaser/>

Andreas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080303/c2607307/attachment.pgp 

From pjpj at live.in  Mon Mar  3 18:22:56 2008
From: pjpj at live.in (Pankaj Gupta)
Date: Mon, 3 Mar 2008 22:52:56 +0530
Subject: [Distutils] Python: Distributing Python modules
In-Reply-To: <BLU128-W34160730336B9FAEFB0D25C5160@phx.gbl>
References: <BLU128-W34160730336B9FAEFB0D25C5160@phx.gbl>
Message-ID: <BLU128-W455DAC252D006BD56F7B0CC5170@phx.gbl>




Hi,
 
Windows Installer Executables using distutils
 
I am able to create windows installer executable using 'python setup.py' on command prompt.
 
using below code is written in 'setup.py'
 
       from distutils.core import setup
 
       setup(name='Test',            description="Python Test",            version='0.1',            script_name = 'setup.py',            script_args = ['bdist_wininst','--install-script=post.py'],            scripts = ['Post.py'],            data_files = getDATA())
 
Following are the issues need to be resolved after installing windows installer 'Test.exe'
 
1.It installs at default location and creates a folder 'Test' under 'C:\Python24'
Question: Could you please suggest me how to change the default location using above mentioned code or can specify location on UI in windows installer ?
 
2. When we double click on windows installer 'Test.exe' it shows UI
Question: Another requirement: Can we suppress the UI? I am looking 'how to launch the executable installers (e.g ctypes-1.0.1.win32-py2.4.exe) for the prerequisites silently ? i.e., without user interaction'
 
Could you please help me or send me direct e-mail ids regarding 'Distributing python modules' so that i can communicate accordingly?
 
Thanks
PG
 

Live the life in style with MSN Lifestyle. Check out! Try it now! 
_________________________________________________________________
Post free property ads on Yello Classifieds now! www.yello.in
http://ss1.richmedia.in/recurl.asp?pid=221
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080303/86a19e48/attachment.htm 

From lists at durin42.com  Mon Mar  3 18:38:10 2008
From: lists at durin42.com (Augie Fackler)
Date: Mon, 3 Mar 2008 11:38:10 -0600
Subject: [Distutils] setuptools Fails with Subversion 1.5
In-Reply-To: <66597D62-0263-4078-AFB4-B65049B3E266@cs.toronto.edu>
References: <8FE3CD7E-E1DA-494D-BA12-892F2D1F8B04@durin42.com>
	<66597D62-0263-4078-AFB4-B65049B3E266@cs.toronto.edu>
Message-ID: <5EF35B2D-9C10-48A9-87C6-8ED79C253763@durin42.com>

On Feb 26, 2008, at 9:55 AM, David Wolever wrote:

> I can confirm that, when this change is made, distutils can still  
> get a list of visioned files (I'd like to say "everything works  
> fine", but I suppose all I can really say is "I haven't had a  
> problem with it yet).

My checking indicates that this will continue to work fine - the  
reason the version number in the file got bumped was because they  
added some fields. The existing regular expressions used to parse the  
entries file shouldn't misbehave in any way whatsoever.
Peace,
Augie

> On 25-Feb-08, at 11:20 PM, Augie Fackler wrote:
>
>> Hey all,
>>
>> With Subversion 1.5 entering alpha, I've been using it for more  
>> than just a few projects, and I've noticed that setuptools fails to  
>> run in a 1.5 working copy. For the short term, I've found that I  
>> can just modify the setuptools source to trust that version 9 WCs  
>> will work the same as version 8 WCs. The diff for that is attached.
>>
>> Based on what I've read of subversion's README on the format of the  
>> entries file, the modification I've made should be just fine - the  
>> overall structure of the file is the same, there's just some new  
>> tags possible which will already not get stuck in the regex already  
>> in place. In my local testing, using setuptools for installing  
>> files works just fine with this patch in place.
>>
>> Peace,
>> Augie Fackler<setuptools_patch.txt>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>


From zooko at zooko.com  Tue Mar  4 18:28:43 2008
From: zooko at zooko.com (zooko)
Date: Tue, 4 Mar 2008 10:28:43 -0700
Subject: [Distutils] request: make versions which include svn revision
	numbers look the same with distutils and with setuptools
Message-ID: <4B2E3824-258E-4587-8F44-38C4AB683513@zooko.com>

Folks:

I'm trying to get the Nevow project to accept a patch [1] that makes  
it easier to use Nevow with setuptools, even though the maintainers  
of Nevow passionately hate setuptools.

In such a situation, any tiny unnecessary difference between  
distutils behavior and setuptools behavior is another grain of sand  
in the gears.

So with the patch that I've offered to Nevow, if you run "./setup.py  
sdist" without setuptools installed, you get:

Nevow-0.9.31+r15049.tar.gz

and if you run "./setup.py sdist" with setuptools installed, you get:

Nevow-0.9.31-r15049.tar.gz

Why are the version numbers different?  I personally prefer the '-'  
instead of the '+' in the version number, and I would be happy if  
distutils in Python 2.4.5 used '-'.  However, unless the distutils is  
going to start using '-', then I suggest that setuptools v0.6 final  
also use '+', so that there is one less thing that is different.

Regards,

Zooko

[1] http://www.divmod.org/trac/ticket/2527

From pje at telecommunity.com  Tue Mar  4 20:04:23 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Mar 2008 14:04:23 -0500
Subject: [Distutils] request: make versions which include svn revision
 numbers look the same with distutils and with setuptools
In-Reply-To: <4B2E3824-258E-4587-8F44-38C4AB683513@zooko.com>
References: <4B2E3824-258E-4587-8F44-38C4AB683513@zooko.com>
Message-ID: <20080304190405.A90D73A40A5@sparrow.telecommunity.com>

At 10:28 AM 3/4/2008 -0700, zooko wrote:
>So with the patch that I've offered to Nevow, if you run "./setup.py
>sdist" without setuptools installed, you get:
>
>Nevow-0.9.31+r15049.tar.gz
>
>and if you run "./setup.py sdist" with setuptools installed, you get:
>
>Nevow-0.9.31-r15049.tar.gz
>
>Why are the version numbers different?

Because setuptools treats all runs of characters other than 
[-.A-Za-z0-9] as equal to a single '-', and canonicalizes both 
project and version names accordingly.


From zooko at zooko.com  Tue Mar  4 21:02:16 2008
From: zooko at zooko.com (zooko)
Date: Tue, 4 Mar 2008 13:02:16 -0700
Subject: [Distutils] request: make versions which include svn revision
	numbers look the same with distutils and with setuptools
In-Reply-To: <20080304190405.A90D73A40A5@sparrow.telecommunity.com>
References: <4B2E3824-258E-4587-8F44-38C4AB683513@zooko.com>
	<20080304190405.A90D73A40A5@sparrow.telecommunity.com>
Message-ID: <31D99E12-2BB8-4103-8073-1B53D7BEDC93@zooko.com>

On Mar 4, 2008, at 12:04 PM, Phillip J. Eby wrote:

> Because setuptools treats all runs of characters other than [-.A-Za- 
> z0-9] as equal to a single '-', and canonicalizes both project and  
> version names accordingly.

So why doesn't distutils do that?

I mean: could we please change distutils to do that?

Regards,

Zooko


From p.f.moore at gmail.com  Tue Mar  4 22:05:30 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 4 Mar 2008 21:05:30 +0000
Subject: [Distutils] Easy Install - download an egg
Message-ID: <79990c6b0803041305q5e02c656wfc238995c936600b@mail.gmail.com>

This is a dumb question, but I can't find the answer in the
easy_install documentation (I have to say, I always find that page
hard to follow...)

How do I use easy_install to just locate and download an egg (not install it)?

Suppose I have a directory "eggs" and I want to grab a batch of eggs
from PyPI or wherever. I can simply go to the appropriate download
pages with my web browser, and download the files, but surely I can
get easy_install to grab the files for me?

Thanks,
Paul.

From pje at telecommunity.com  Tue Mar  4 23:32:50 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 04 Mar 2008 17:32:50 -0500
Subject: [Distutils] Easy Install - download an egg
In-Reply-To: <79990c6b0803041305q5e02c656wfc238995c936600b@mail.gmail.co
 m>
References: <79990c6b0803041305q5e02c656wfc238995c936600b@mail.gmail.com>
Message-ID: <20080304223234.F2F4D3A40A5@sparrow.telecommunity.com>

At 09:05 PM 3/4/2008 +0000, Paul Moore wrote:
>This is a dumb question, but I can't find the answer in the
>easy_install documentation (I have to say, I always find that page
>hard to follow...)
>
>How do I use easy_install to just locate and download an egg (not install it)?
>
>Suppose I have a directory "eggs" and I want to grab a batch of eggs
>from PyPI or wherever. I can simply go to the appropriate download
>pages with my web browser, and download the files, but surely I can
>get easy_install to grab the files for me?


easy_install -zmaxd targetdir ...

The -x is to --exclude-scripts, and -d sets the 
--install-directory.  -m means multi-version, which keeps it from 
checking for sys.path compatibility and avoids creating an 
easy-install.pth.  -z means not to unzip the files, and -a means 
--always-copy, so even if the requested egg is already on sys.path, 
it'll be copied to targetdir.


From p.f.moore at gmail.com  Wed Mar  5 00:56:22 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 4 Mar 2008 23:56:22 +0000
Subject: [Distutils] Easy Install - download an egg
In-Reply-To: <20080304223234.F2F4D3A40A5@sparrow.telecommunity.com>
References: <79990c6b0803041305q5e02c656wfc238995c936600b@mail.gmail.com>
	<20080304223234.F2F4D3A40A5@sparrow.telecommunity.com>
Message-ID: <79990c6b0803041556o266cbe0fu248a66bb15de6d64@mail.gmail.com>

On 04/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:

> easy_install -zmaxd targetdir ...

Thanks. I see it in the documentation now. Sorry I missed it.

Paul.

From lukas.polivka at gmail.com  Wed Mar  5 09:00:35 2008
From: lukas.polivka at gmail.com (=?UTF-8?Q?Luk=C3=A1=C5=A1_'Spike'_Pol=C3=ADvka?=)
Date: Wed, 5 Mar 2008 09:00:35 +0100
Subject: [Distutils] distutils.errors.DistutilsFileError: file
	'/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.pyc'
	does not exist
Message-ID: <76a8b5470803050000i5a314a91k1fc136c06fc81211@mail.gmail.com>

Hello!

I'm trying to build an app bundle of Jabbim client [1] for Mac OS X
using py2app & setuptools. I have followed py2app docs [2]. When I
build an 'alias' (see Development with alias mode) mode app, I can run
it. On the other hand, if I build for deployment (without -A flag), I
get the following error/traceback:

Traceback (most recent call last):
  File "/var/folders/Eg/Egf28uy-E94kLa7x5eHVp++++TI/-Tmp-/tmprMzA41.py",
line 229, in <module>
    direct=1)
  File "/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/util.py",
line 204, in byte_compile
    if force or newer(mod.filename, cfile):
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dep_util.py",
line 22, in newer
    raise DistutilsFileError, "file '%s' does not exist" % source
distutils.errors.DistutilsFileError: file
'/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.pyc'
does not exist
Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/build_app.py",
line 548, in _run
    self.run_normal()
  File "/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/build_app.py",
line 619, in run_normal
    self.create_binaries(py_files, pkgdirs, extensions, loader_files)
  File "/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/build_app.py",
line 683, in create_binaries
    dry_run=self.dry_run)
  File "/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/util.py",
line 176, in byte_compile
    spawn(cmd, verbose=verbose, dry_run=dry_run)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/spawn.py",
line 37, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/spawn.py",
line 167, in _spawn_posix
    (cmd[0], exit_status)
DistutilsExecError: command
'/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python'
failed with exit status 1
> /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/spawn.py(167)_spawn_posix()

I don't exactly know how Python Eggs work ? I have only checked that
setuptools-0.6c8-py2.5.egg exists ? only it is a file, whereas e.g.
py2app-0.3.6-py2.5.egg in the same location is a directory/folder.

Any idea why setuptools-0.6c8-py2.5.egg/pkg_resources.pyc does not 'exist'?

When I open Python interpreter, import pkg_resources works OK.

When running the app or building an alias bundle, I have to export
PYTHON_PATH and use --use-pythonpath flag, after that, it works. When
building a deployment bundle, it does not work.

Thanks you very much for any suggestion you can provide, even relevant
links to documentation I might have missed.
Feel free to ask any questions or request my setup.py if needed.

Luk?? 'Spike' Pol?vka

[1] http://dev.jabbim.cz/jabbim/wiki/en/index
[2] http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.htm

-- 
IM (XMPP/Jabber/Google Talk): spike411 at jabber.cz
ICQ, AIM, MSN: Never ever!

From pje at telecommunity.com  Wed Mar  5 14:28:26 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 05 Mar 2008 08:28:26 -0500
Subject: [Distutils] distutils.errors.DistutilsFileError: file
 '/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resou
 rces.pyc' does not exist
In-Reply-To: <76a8b5470803050000i5a314a91k1fc136c06fc81211@mail.gmail.co
 m>
References: <76a8b5470803050000i5a314a91k1fc136c06fc81211@mail.gmail.com>
Message-ID: <20080305132809.01D123A4097@sparrow.telecommunity.com>

At 09:00 AM 3/5/2008 +0100, Luk??? 'Spike' Pol??vka wrote:
>Traceback (most recent call last):   File 
>"/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/build_app.py", 
>line 548, in _run     self.run_normal()   File 
>"/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/build_app.py", 
>line 619, in run_normal     self.create_binaries(py_files, pkgdirs, 
>extensions, loader_files)   File 
>"/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/build_app.py", 
>line 683, in create_binaries     dry_run=self.dry_run)   File 
>"/Library/Python/2.5/site-packages/py2app-0.3.6-py2.5.egg/py2app/util.py", 
>line 176, in byte_compile     spawn(cmd, verbose=verbose, 
>dry_run=dry_run)   File 
>"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/spawn.py", 
>line 37, in spawn     _spawn_posix(cmd, search_path, 
>dry_run=dry_run)   File 
>"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/spawn.py", 
>line 167, in _spawn_posix     (cmd[0], exit_status) 
>DistutilsExecError: command 
>'/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python' 
>failed with exit status 1 > 
>/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/spawn.py(167)_spawn_posix() 
>

This error appears to indicate that py2app is having trouble creating 
a .pyc or .pyo file, due to not being able to run 
"/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python".


From zooko at zooko.com  Wed Mar  5 15:20:47 2008
From: zooko at zooko.com (zooko)
Date: Wed, 5 Mar 2008 07:20:47 -0700
Subject: [Distutils] wanted: a catalog of setuptools plugins
In-Reply-To: <FA3E00A1-E220-4555-B69A-B169F220E89D@acm.org>
References: <8E7A32F1-9C65-401B-A424-E590D6ECCD0E@zooko.com>
	<4F152FFE-C3B7-46B2-8A37-544D50C0A494@zooko.com>
	<FA3E00A1-E220-4555-B69A-B169F220E89D@acm.org>
Message-ID: <AC54D315-93D8-4230-BFD1-4A42C46141C6@zooko.com>


On Jan 28, 2008, at 8:28 AM, Fred Drake wrote:

> On Jan 28, 2008, at 9:41 AM, zooko wrote:
>> And this morning I just noticed the "keywords" field in the
>> metadata.  Perhaps the setuptools docs could encourage authors of
>> setuptools plugins to put the keyword "plugin" into their keywords?
>
> Another option, for which there's precedent, is to define a  
> classifier for the setuptools framework, similar to the classifiers  
> used for the various web frameworks.

Okay, let's call it "Frameworks :: Setuptools Plugin".

I just tried to add that classifier to my "darcsver" plugin, but got  
this result from pypi:

Submitting dist/darcsver-1.0.3.tar.gz to http://www.python.org/pypi
Upload failed (500): There's been a problem with your request
------------------------------------------------------------------------ 
--- Error...

There's been a problem with your request

exceptions.TypeError: unsubscriptable object  
------------------------------------------------------------------------ 
---


Rerunning "./setup.py register" tells me this:

Using PyPI login from /Users/wonwinmcbrootles/.pypirc
Server response (400): Invalid classifier "Framework :: Setuptools  
Plugin"


Regards,

Zooko


From ziade.tarek at gmail.com  Wed Mar  5 15:30:47 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 5 Mar 2008 15:30:47 +0100
Subject: [Distutils] wanted: a catalog of setuptools plugins
In-Reply-To: <AC54D315-93D8-4230-BFD1-4A42C46141C6@zooko.com>
References: <8E7A32F1-9C65-401B-A424-E590D6ECCD0E@zooko.com>
	<4F152FFE-C3B7-46B2-8A37-544D50C0A494@zooko.com>
	<FA3E00A1-E220-4555-B69A-B169F220E89D@acm.org>
	<AC54D315-93D8-4230-BFD1-4A42C46141C6@zooko.com>
Message-ID: <94bdd2610803050630j9a9bc80ne2bc9e9d780b4f49@mail.gmail.com>

On Wed, Mar 5, 2008 at 3:20 PM, zooko <zooko at zooko.com> wrote:

>
> On Jan 28, 2008, at 8:28 AM, Fred Drake wrote:
>
> > On Jan 28, 2008, at 9:41 AM, zooko wrote:
> >> And this morning I just noticed the "keywords" field in the
> >> metadata.  Perhaps the setuptools docs could encourage authors of
> >> setuptools plugins to put the keyword "plugin" into their keywords?
> >
> > Another option, for which there's precedent, is to define a
> > classifier for the setuptools framework, similar to the classifiers
> > used for the various web frameworks.
>
> Okay, let's call it "Frameworks :: Setuptools Plugin".
>
> I just tried to add that classifier to my "darcsver" plugin, but got
> this result from pypi:
>
> Submitting dist/darcsver-1.0.3.tar.gz to http://www.python.org/pypi
> Upload failed (500): There's been a problem with your request
> ------------------------------------------------------------------------
> --- Error...
>
> There's been a problem with your request
>
> exceptions.TypeError: unsubscriptable object
> ------------------------------------------------------------------------
> ---
>
>
> Rerunning "./setup.py register" tells me this:
>
> Using PyPI login from /Users/wonwinmcbrootles/.pypirc
> Server response (400): Invalid classifier "Framework :: Setuptools
> Plugin"
>


Yes the classifier has to be in the TROVE list maintained at Pypi
(see http://pypi.python.org/pypi?%3Aaction=list_classifiers)

It will have to be added for your package to be registerable.
You may request for this inclusion on catalog-sig ML

In the meantime, we have discussed lately on the fact that Pypi should
not return an error when a classifier does not exist but a warning
(see http://wiki.python.org/moin/EnhancedPyPI)

I didn't find the time yet to submit and propose a patch for this, but it is

on my pile.

Tarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080305/a31dbe8b/attachment.htm 

From ziade.tarek at gmail.com  Wed Mar  5 16:00:06 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 5 Mar 2008 16:00:06 +0100
Subject: [Distutils] easy_install and https
Message-ID: <94bdd2610803050700ga606715od539c08dc6944ff@mail.gmail.com>

Hello,

If I do :

$ easy_install -U somepackage

The script will scan the PyPI simple page to look for an egg.

But if it fails on an url page that is starting with https, and if the user
is behind a
proxy that blocks it, it will block there before any egg has a chance to
be downloaded.

I guess a good practice is to avoid https URLs in any metadata,
I guess the user can also call easy_install with the direct egg url in that
case,

but maybe easy_install could do something particular on https links to avoid

this issue, if doable ? Maybe by ordering the reading, to put https ones at
the end ?
(I don't know how https failures works when a proxy blocks them)

Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080305/8c8d593c/attachment.htm 

From lukas.polivka at gmail.com  Wed Mar  5 16:22:01 2008
From: lukas.polivka at gmail.com (=?UTF-8?Q?Luk=C3=A1=C5=A1_'Spike'_Pol=C3=ADvka?=)
Date: Wed, 5 Mar 2008 16:22:01 +0100
Subject: [Distutils] distutils.errors.DistutilsFileError: file
	'/Library/Python/2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resou
	rces.pyc' does not exist
In-Reply-To: <20080305132809.01D123A4097@sparrow.telecommunity.com>
References: <76a8b5470803050000i5a314a91k1fc136c06fc81211@mail.gmail.com>
	<20080305132809.01D123A4097@sparrow.telecommunity.com>
Message-ID: <76a8b5470803050722h123a2a4ex7cc063bc17549e37@mail.gmail.com>

On Wed, Mar 5, 2008 at 2:28 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
>  This error appears to indicate that py2app is having trouble creating
>  a .pyc or .pyo file, due to not being able to run
>  "/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python".
>
>

It seems to start the interpreter without problem:
$ /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Any further suggestions?
I'm running Mac OS X 10.5.2. I think I had MacPython (or
ActivePython?) 2.5 installed on 10.4.10, before I upgraded to Leopard.
I then deleted the non-bundled version by hand.

Thanks,
Luk?? 'Spike' Pol?vka
-- 
IM (XMPP/Jabber/Google Talk): spike411 at jabber.cz
ICQ, AIM, MSN: Never ever!

From jtk at yahoo.com  Fri Mar  7 13:11:30 2008
From: jtk at yahoo.com (Jeff Kowalczyk)
Date: Fri, 07 Mar 2008 07:11:30 -0500
Subject: [Distutils] PyPI registration and ownership of package names
Message-ID: <pan.2008.03.07.12.11.30.149333@yahoo.com>

Are PyPI names exclusive to the first registration?

Not to single out a particular developer, but does the presence of
http://pypi.python.org/pypi/Reporting/1.0 mean that the top-level name
'Reporting' always and forever belongs to a proprietary package with no
upload?

Thanks,
Jeff


From pje at telecommunity.com  Fri Mar  7 13:58:34 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Mar 2008 07:58:34 -0500
Subject: [Distutils] PyPI registration and ownership of package  names
In-Reply-To: <pan.2008.03.07.12.11.30.149333@yahoo.com>
References: <pan.2008.03.07.12.11.30.149333@yahoo.com>
Message-ID: <20080307130431.C62FC3A4097@sparrow.telecommunity.com>

At 07:11 AM 3/7/2008 -0500, Jeff Kowalczyk wrote:
>Are PyPI names exclusive to the first registration?
>
>Not to single out a particular developer, but does the presence of
>http://pypi.python.org/pypi/Reporting/1.0 mean that the top-level name
>'Reporting' always and forever belongs to a proprietary package with no
>upload?

This is a topic for the Catalog-SIG, actually.


From r.ritz at biologie.hu-berlin.de  Fri Mar  7 14:26:35 2008
From: r.ritz at biologie.hu-berlin.de (Raphael Ritz)
Date: Fri, 07 Mar 2008 14:26:35 +0100
Subject: [Distutils] Tahoe
Message-ID: <fqrfqa$1hc$1@ger.gmane.org>

http://allmydata.org/trac/tahoe

something for the software center?

Raphael


From r.ritz at biologie.hu-berlin.de  Fri Mar  7 14:29:50 2008
From: r.ritz at biologie.hu-berlin.de (Raphael Ritz)
Date: Fri, 07 Mar 2008 14:29:50 +0100
Subject: [Distutils] Sorry [Re: Tahoe]
In-Reply-To: <fqrfqa$1hc$1@ger.gmane.org>
References: <fqrfqa$1hc$1@ger.gmane.org>
Message-ID: <fqrg0d$1hc$2@ger.gmane.org>

Sorry guys this wasn't meant to go here.
My apologies :-(

Raphael

Raphael Ritz wrote:
> http://allmydata.org/trac/tahoe
> 
> something for the software center?
> 
> Raphael
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 


From lists at zopyx.com  Fri Mar  7 14:26:36 2008
From: lists at zopyx.com (Andreas Jung)
Date: Fri, 07 Mar 2008 14:26:36 +0100
Subject: [Distutils] [zc.buildout] find-links does not deal with two-level
	hierarchy
Message-ID: <46A583988116C8D0DC43BB79@[192.168.0.24]>

Hi,

we have a local fileserver with a two-level hierarchy for egg -something 
like:

haufe.eggserver
  +--- haufe.eggserver-0.1.0.egg
  +---
haufe.recipe.svnsetup
  +--- ....

easy_install -i http://our-server/egg haufe.recipe.svnsetup
works fine with this two-level hierarchy.

Using the URL within the find-links option of my buildout.cfg
does not work  (Error: Could not find a distribution...) for a part
with recipe=haufe.recipe.svnsetup.

Bug or feature?

Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080307/5b59be70/attachment.pgp 

From pje at telecommunity.com  Fri Mar  7 18:25:21 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Mar 2008 12:25:21 -0500
Subject: [Distutils] [zc.buildout] find-links does not deal with
 two-level hierarchy
In-Reply-To: <46A583988116C8D0DC43BB79@[192.168.0.24]>
References: <46A583988116C8D0DC43BB79@[192.168.0.24]>
Message-ID: <20080307172504.A81F63A4097@sparrow.telecommunity.com>

At 02:26 PM 3/7/2008 +0100, Andreas Jung wrote:
>Hi,
>
>we have a local fileserver with a two-level hierarchy for egg -something like:
>
>haufe.eggserver
>  +--- haufe.eggserver-0.1.0.egg
>  +---
>haufe.recipe.svnsetup
>  +--- ....
>
>easy_install -i http://our-server/egg haufe.recipe.svnsetup
>works fine with this two-level hierarchy.
>
>Using the URL within the find-links option of my buildout.cfg
>does not work  (Error: Could not find a distribution...) for a part
>with recipe=haufe.recipe.svnsetup.
>
>Bug or feature?

By design -- easy_install doesn't want to be a general-purpose web spider.

If you want your index to work with find-links, you'll need to either 
dump it all in one level (and use -f instead of -i), or else put 
2nd-level links in your find-links.


From tseaver at palladion.com  Fri Mar  7 18:58:12 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 07 Mar 2008 12:58:12 -0500
Subject: [Distutils] [zc.buildout] find-links does not deal with
	two-level hierarchy
In-Reply-To: <20080307172504.A81F63A4097@sparrow.telecommunity.com>
References: <46A583988116C8D0DC43BB79@[192.168.0.24]>
	<20080307172504.A81F63A4097@sparrow.telecommunity.com>
Message-ID: <47D18234.2030808@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 02:26 PM 3/7/2008 +0100, Andreas Jung wrote:
>> Hi,
>>
>> we have a local fileserver with a two-level hierarchy for egg -something like:
>>
>> haufe.eggserver
>>  +--- haufe.eggserver-0.1.0.egg
>>  +---
>> haufe.recipe.svnsetup
>>  +--- ....
>>
>> easy_install -i http://our-server/egg haufe.recipe.svnsetup
>> works fine with this two-level hierarchy.
>>
>> Using the URL within the find-links option of my buildout.cfg
>> does not work  (Error: Could not find a distribution...) for a part
>> with recipe=haufe.recipe.svnsetup.
>>
>> Bug or feature?
> 
> By design -- easy_install doesn't want to be a general-purpose web spider.
> 
> If you want your index to work with find-links, you'll need to either 
> dump it all in one level (and use -f instead of -i), or else put 
> 2nd-level links in your find-links.

When you use 'easy_install -i URL', you are using an *index*, which is
different than using *find links* (the '-f' options).  If zc.buildout
were able to express "use this package index, but only for this target
egg", then that would be the equivalent spelling.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH0YI0+gerLs4ltQ4RAkD7AJ4o5ZjaEDzxpo8N+TS/sHbMJNh0OwCgw36a
yJBtNehNUiVZauZmsRybzq8=
=JZ5f
-----END PGP SIGNATURE-----


From jim at zope.com  Fri Mar  7 19:08:41 2008
From: jim at zope.com (Jim Fulton)
Date: Fri, 7 Mar 2008 13:08:41 -0500
Subject: [Distutils] [zc.buildout] find-links does not deal with
	two-level hierarchy
In-Reply-To: <47D18234.2030808@palladion.com>
References: <46A583988116C8D0DC43BB79@[192.168.0.24]>
	<20080307172504.A81F63A4097@sparrow.telecommunity.com>
	<47D18234.2030808@palladion.com>
Message-ID: <EBCF0E41-5C9D-4137-8157-F77893D7A858@zope.com>


On Mar 7, 2008, at 12:58 PM, Tres Seaver wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Phillip J. Eby wrote:
>> At 02:26 PM 3/7/2008 +0100, Andreas Jung wrote:
>>> Hi,
>>>
>>> we have a local fileserver with a two-level hierarchy for egg - 
>>> something like:
>>>
>>> haufe.eggserver
>>> +--- haufe.eggserver-0.1.0.egg
>>> +---
>>> haufe.recipe.svnsetup
>>> +--- ....
>>>
>>> easy_install -i http://our-server/egg haufe.recipe.svnsetup
>>> works fine with this two-level hierarchy.
>>>
>>> Using the URL within the find-links option of my buildout.cfg
>>> does not work  (Error: Could not find a distribution...) for a part
>>> with recipe=haufe.recipe.svnsetup.
>>>
>>> Bug or feature?
>>
>> By design -- easy_install doesn't want to be a general-purpose web  
>> spider.
>>
>> If you want your index to work with find-links, you'll need to either
>> dump it all in one level (and use -f instead of -i), or else put
>> 2nd-level links in your find-links.
>
> When you use 'easy_install -i URL', you are using an *index*, which is
> different than using *find links* (the '-f' options).  If zc.buildout
> were able to express "use this package index, but only for this target
> egg", then that would be the equivalent spelling.


zc.buildout does let you specify an index for an individual part/egg.

I fail to see how that is an equivalent spelling.

Jim

--
Jim Fulton
Zope Corporation



From tseaver at palladion.com  Fri Mar  7 19:27:15 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 07 Mar 2008 13:27:15 -0500
Subject: [Distutils] [zc.buildout] find-links does not deal with
	two-level hierarchy
In-Reply-To: <EBCF0E41-5C9D-4137-8157-F77893D7A858@zope.com>
References: <46A583988116C8D0DC43BB79@[192.168.0.24]>	<20080307172504.A81F63A4097@sparrow.telecommunity.com>	<47D18234.2030808@palladion.com>
	<EBCF0E41-5C9D-4137-8157-F77893D7A858@zope.com>
Message-ID: <47D18903.8060404@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Fulton wrote:
> On Mar 7, 2008, at 12:58 PM, Tres Seaver wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Phillip J. Eby wrote:
>>> At 02:26 PM 3/7/2008 +0100, Andreas Jung wrote:
>>>> Hi,
>>>>
>>>> we have a local fileserver with a two-level hierarchy for egg - 
>>>> something like:
>>>>
>>>> haufe.eggserver
>>>> +--- haufe.eggserver-0.1.0.egg
>>>> +---
>>>> haufe.recipe.svnsetup
>>>> +--- ....
>>>>
>>>> easy_install -i http://our-server/egg haufe.recipe.svnsetup
>>>> works fine with this two-level hierarchy.
>>>>
>>>> Using the URL within the find-links option of my buildout.cfg
>>>> does not work  (Error: Could not find a distribution...) for a part
>>>> with recipe=haufe.recipe.svnsetup.
>>>>
>>>> Bug or feature?
>>> By design -- easy_install doesn't want to be a general-purpose web  
>>> spider.
>>>
>>> If you want your index to work with find-links, you'll need to either
>>> dump it all in one level (and use -f instead of -i), or else put
>>> 2nd-level links in your find-links.
>> When you use 'easy_install -i URL', you are using an *index*, which is
>> different than using *find links* (the '-f' options).  If zc.buildout
>> were able to express "use this package index, but only for this target
>> egg", then that would be the equivalent spelling.
> 
> 
> zc.buildout does let you specify an index for an individual part/egg.

I didn't know that:  I believed the "there can be only one index" meme
which I heard during the KGS discussion.

> I fail to see how that is an equivalent spelling.

If each a buildout can customize the index URL for each egg it installs,
then Andreas can have what he wants, which is a way to install an egg in
the same way that 'easy_install -i' does.  'find-links' is a
distraction, in this case, because it is not the same as 'easy_install
- -i' (it is equivalent to 'easy_install -f').


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH0YkD+gerLs4ltQ4RAgToAJ9Fl3SZJ5o6J84jfgZThByHKcWpkgCfR/6G
AeWSGp50fKxt5ZAFZdYdNf4=
=8br0
-----END PGP SIGNATURE-----


From pje at telecommunity.com  Sat Mar  8 03:42:51 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 07 Mar 2008 21:42:51 -0500
Subject: [Distutils] Shared libs with setuptools/distutils ?
In-Reply-To: <Pine.OSX.4.64.0803071800280.47913@yuzu.local>
References: <Pine.OSX.4.64.0803071800280.47913@yuzu.local>
Message-ID: <20080308024302.760403A40C1@sparrow.telecommunity.com>

At 06:11 PM 3/7/2008 -0800, Andi Vajda wrote:
>If I want to support multiple Python extension modules built with 
>JCC to run in the same Python process and hence share the same Java 
>VM, there is some code between these modules that needs to be shared.
>
>Currently, the JCC runtime code (not much, really) is compiled into 
>each JCC-built extension module. There is one C++ class, JCCEnv, 
>that really needs to be shared among all JCC-built modules. Building 
>that class into a shared library (instead of a python module 
>library) could be to be the way to go. Then every such module would 
>be linked against this shared library.
>
>I gave this a quick manual try on Mac and it seems to work.
>
>But do distutils or setuptools have built-in ways of building a 
>regular shared library (in particular, on Mac OS X, a .dylib, not a 
>bundle) or is there some hacking needed there still ? (google shows 
>that there have been nibbles of interest over the past years about 
>this but it's unclear whether this was done or not). It would make 
>the JCC build considerably more complicated if I had to add 
>hand-crafted code for each and every platform/compiler that is 
>supported by JCC (that list is growing). Having a distutils or 
>setuptools way of building a regular shared library would be very 
>helpful here. Has support for that that been added yet ?

Not really, but there is some experimental code in the setuptools 
trunk, IIRC.  If you look at the 'tests/test_shlib' subdirectory 
there is some sample code for building a shared library.  I don't 
know if it works right on OS X...  and there were some hairy things 
involved in doing it on Linux.  Patches welcome.  :)

(Chances are good, though, that you'll be better off using the import 
+ API struct approach to sharing stuff.  IIRC, there are still a lot 
of hoops to jump through to get shared libs working on OS X and Linux.)


From vajda at osafoundation.org  Sat Mar  8 04:34:07 2008
From: vajda at osafoundation.org (Andi Vajda)
Date: Fri, 7 Mar 2008 19:34:07 -0800
Subject: [Distutils] Shared libs with setuptools/distutils ?
In-Reply-To: <20080308024302.760403A40C1@sparrow.telecommunity.com>
References: <Pine.OSX.4.64.0803071800280.47913@yuzu.local>
	<20080308024302.760403A40C1@sparrow.telecommunity.com>
Message-ID: <5459CED6-5C0C-46ED-8157-B9A83FDD77C6@osafoundation.org>


On Mar 7, 2008, at 18:42, "Phillip J. Eby" <pje at telecommunity.com>  
wrote:

> At 06:11 PM 3/7/2008 -0800, Andi Vajda wrote:
>> If I want to support multiple Python extension modules built with  
>> JCC to run in the same Python process and hence share the same Java  
>> VM, there is some code between these modules that needs to be shared.
>>
>> Currently, the JCC runtime code (not much, really) is compiled into  
>> each JCC-built extension module. There is one C++ class, JCCEnv,  
>> that really needs to be shared among all JCC-built modules.  
>> Building that class into a shared library (instead of a python  
>> module library) could be to be the way to go. Then every such  
>> module would be linked against this shared library.
>>
>> I gave this a quick manual try on Mac and it seems to work.
>>
>> But do distutils or setuptools have built-in ways of building a  
>> regular shared library (in particular, on Mac OS X, a .dylib, not a  
>> bundle) or is there some hacking needed there still ? (google shows  
>> that there have been nibbles of interest over the past years about  
>> this but it's unclear whether this was done or not). It would make  
>> the JCC build considerably more complicated if I had to add hand- 
>> crafted code for each and every platform/compiler that is supported  
>> by JCC (that list is growing). Having a distutils or setuptools way  
>> of building a regular shared library would be very helpful here.  
>> Has support for that that been added yet ?
>
> Not really, but there is some experimental code in the setuptools  
> trunk, IIRC.  If you look at the 'tests/test_shlib' subdirectory  
> there is some sample code for building a shared library.  I don't  
> know if it works right on OS X...  and there were some hairy things  
> involved in doing it on Linux.  Patches welcome.  :)
>
> (Chances are good, though, that you'll be better off using the  
> import + API struct approach to sharing stuff.  IIRC, there are  
> still a lot of hoops to jump through to get shared libs working on  
> OS X and Linux.)
>

Yes, I thought of that first but how do I do this with a C++ class and  
all its methods ? We're not talking about a single function pointer  
here :(

Andi..


From lists at zopyx.com  Sat Mar  8 07:48:47 2008
From: lists at zopyx.com (Andreas Jung)
Date: Sat, 08 Mar 2008 07:48:47 +0100
Subject: [Distutils] Encoding of metadata?
Message-ID: <A8F9629810894D00C931B825@[192.168.0.24]>

Hi,

are there any recommendations or constraints for the encoding of
package metadata - both for PyPI and the setuptools internals?
Although most of us write their metadata in English, we do have author 
names with accented characters...it's unclear if iso-8859-15 or utf-8 is 
preferred in this case (or even unicode strings)...what's the 
recommendation for
the non-ascii world?

Andreas

-- 
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 T?bingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Gesch?ftsf?hrer/Gesellschafter: ZOPYX Limited, Birmingham, UK
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080308/780e0b9e/attachment.pgp 

From bignose+hates-spam at benfinney.id.au  Sat Mar  8 08:26:54 2008
From: bignose+hates-spam at benfinney.id.au (Ben Finney)
Date: Sat, 08 Mar 2008 18:26:54 +1100
Subject: [Distutils] Encoding of metadata?
References: <A8F9629810894D00C931B825@[192.168.0.24]>
Message-ID: <8763vx7k1t.fsf@benfinney.id.au>

Andreas Jung <lists at zopyx.com> writes:

> what's the recommendation for the non-ascii world?

I'm not sure what body you're expecting a recommendation from. My own
recommendation is to encode using UTF-8.

-- 
 \     "If you can't annoy somebody there is little point in writing." |
  `\                                                  -- Kingsley Amis |
_o__)                                                                  |
Ben Finney


From lists at zopyx.com  Sat Mar  8 08:35:45 2008
From: lists at zopyx.com (Andreas Jung)
Date: Sat, 08 Mar 2008 08:35:45 +0100
Subject: [Distutils] Encoding of metadata?
In-Reply-To: <8763vx7k1t.fsf@benfinney.id.au>
References: <A8F9629810894D00C931B825@[192.168.0.24]>
	<8763vx7k1t.fsf@benfinney.id.au>
Message-ID: <FB62785C44BB375596FCD51E@[192.168.0.24]>



--On 8. M?rz 2008 18:26:54 +1100 Ben Finney 
<bignose+hates-spam at benfinney.id.au> wrote:

> Andreas Jung <lists at zopyx.com> writes:
>
>> what's the recommendation for the non-ascii world?
>
> I'm not sure what body you're expecting a recommendation from.

Strange question. I expect the metadata having a clear and well-defined 
encoding in order to deal with it in a reasonable way within a unicode
application. Guessing or smelling encoding is the _wrong_ way.

-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080308/30b56de6/attachment.pgp 

From tseaver at palladion.com  Sat Mar  8 19:51:09 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 08 Mar 2008 13:51:09 -0500
Subject: [Distutils] Encoding of metadata?
In-Reply-To: <FB62785C44BB375596FCD51E@[192.168.0.24]>
References: <A8F9629810894D00C931B825@[192.168.0.24]>	<8763vx7k1t.fsf@benfinney.id.au>
	<FB62785C44BB375596FCD51E@[192.168.0.24]>
Message-ID: <47D2E01D.6010907@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andreas Jung wrote:
> 
> --On 8. M?rz 2008 18:26:54 +1100 Ben Finney 
> <bignose+hates-spam at benfinney.id.au> wrote:
> 
>> Andreas Jung <lists at zopyx.com> writes:
>>
>>> what's the recommendation for the non-ascii world?
>> I'm not sure what body you're expecting a recommendation from.
> 
> Strange question. I expect the metadata having a clear and well-defined 
> encoding in order to deal with it in a reasonable way within a unicode
> application. Guessing or smelling encoding is the _wrong_ way.

Unless you want to inflict pain on future users, use UTF8.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH0uAd+gerLs4ltQ4RAthOAKCir1ESYZFMVt2tZFDkhSVK5J0QLwCeONRo
jg+QbrSNhvj6Vtd1LRuxTBo=
=8Rgz
-----END PGP SIGNATURE-----


From zooko at zooko.com  Sun Mar  9 05:01:04 2008
From: zooko at zooko.com (zooko)
Date: Sat, 8 Mar 2008 21:01:04 -0700
Subject: [Distutils] [Catalog-sig] requested Trove Classifier:
	"Framework :: Setuptools Plugin"
In-Reply-To: <200803091357.19180.richardjones@optushome.com.au>
References: <3D18B7BD-5B13-4171-B573-943866240545@zooko.com>
	<200803091103.13519.richardjones@optushome.com.au>
	<9F191149-1E82-43B2-B935-53967A905030@zooko.com>
	<200803091357.19180.richardjones@optushome.com.au>
Message-ID: <9EE5F72C-3A55-460D-B1E4-1856D69E97BD@zooko.com>

On Mar 8, 2008, at 7:57 PM, Richard Jones wrote:

> OK, I've added "Framework :: Setuptools Plugin"

Hooray!

I've added the first three setuptools plugins to be marked with that  
classifier.  This search shows all packages on pypi which have the  
"Framework :: Setuptools Plugin" classifier:

http://pypi.python.org/pypi?:action=browse&c=524

Regards,

Zooko


From pje at telecommunity.com  Mon Mar 10 17:57:25 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 10 Mar 2008 12:57:25 -0400
Subject: [Distutils] pywin32 egg?
In-Reply-To: <040e01c7c27b$5b8bc100$090a0a0a@enfoldsystems.local>
References: <20070707122727.4A95D3A4046@sparrow.telecommunity.com>
	<040e01c7c27b$5b8bc100$090a0a0a@enfoldsystems.local>
Message-ID: <20080310165729.914963A40A5@sparrow.telecommunity.com>

I recently discovered that one reason why pywin32 didn't work so well 
with easy_install was that easy_install wasn't processing pywin32's 
.pth file due to a case-sensitivity bug!

I've checked in a fix that appears to handle everything correctly, 
except for the previously-discussed issues with the post-install 
script, and a minor issue in pywintypes.py.  Adding the following at 
line 91 in pywintypes.py fixed it for me:

         if found is None:
             # Not in the Python directory?  Maybe we were installed via
             # easy_install...
             if 
os.path.isfile(os.path.join(os.path.dirname(__file__), filename)):
                 found = os.path.join(os.path.dirname(__file__), filename)

With this addition and my fix to easy_install, it's possible to 
"easy_install pywin32" and get a working pywin32 install, minus the 
various registrations and shortcuts handled by the post-install script.

Fixing the post-install script to work with an easy_install-supplied 
payload is a bit more difficult, since it can't find its library 
directory and various other things.  However, my personal use case 
for having a pywin32 egg is simply to be able to install it into a 
Chandler Python installation, which is not registered in the 
registry.  (The .exe installer won't work with unregistered Pythons, 
so I'm win32-less otherwise.)


From michael.milne at gmail.com  Mon Mar 10 21:11:32 2008
From: michael.milne at gmail.com (Michael NT Milne)
Date: Mon, 10 Mar 2008 20:11:32 +0000 (UTC)
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
Message-ID: <loom.20080310T200733-922@post.gmane.org>

Been told to try this post from Plone users on this list....

This is infuriating. Each time we install a new egg we have to run
buildout in online mode as buildout -o or buildout -No just produces
an invalid command error.

buildout ?c c:\Plone\'yoursitename'\buildout.cfg

The above works but it re-installs new eggs etc alongside the old eggs
and the site breaks as below. When re-running buildout, Zope uses the
eggs with the higher numbers. Do you need to change anything to make
this work? Also has anyone got Windows buildouts running in offline
mode? It seems the only way to fix this is to delete the new eggs etc
that have been fetched.


2008-03-10T16:45:09 ERROR Application Couldn't install Five
Traceback (most recent call last):
 File "D:\Plone3\Zope\lib\python\OFS\Application.py", line 786, in
install_product
   initmethod(context)
 File "D:\Plone3\Zope\lib\python\Products\Five\__init__.py", line 28,
in initialize
   zcml.load_site()
 File "D:\Plone3\Zope\lib\python\Products\Five\zcml.py", line 53, in load_site
   _context = xmlconfig.file(file)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 579, in file
   include(context, name, package)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 515, in include
   processxmlfile(f, context)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 370, in processxmlfile
   parser.parse(src)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 109, in parse
   xmlreader.IncrementalParser.parse(self, source)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\xmlreader.py",
line 123, in parse
   self.feed(buffer)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 216, in feed
   self._parser.Parse(data, isFinal)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 364, in end_element_ns
   self._cont_handler.endElementNS(pair, None)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 349, in endElementNS
   self.context.end()
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
544, in end
   self.stack.pop().finish()
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
692, in finish
   actions = self.handler(context, **args)
 File "D:\Plone3\Zope\lib\python\Products\Five\fiveconfigure.py",
line 78, in loadProducts
   handleBrokenProduct(product)
 File "D:\Plone3\Zope\lib\python\Products\Five\fiveconfigure.py",
line 76, in loadProducts
   xmlconfig.include(_context, zcml, package=product)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 515, in include
   processxmlfile(f, context)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 370, in processxmlfile
   parser.parse(src)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 109, in parse
   xmlreader.IncrementalParser.parse(self, source)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\xmlreader.py",
line 123, in parse
   self.feed(buffer)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 216, in feed
   self._parser.Parse(data, isFinal)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 364, in end_element_ns
   self._cont_handler.endElementNS(pair, None)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 349, in endElementNS
   self.context.end()
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
544, in end
   self.stack.pop().finish()
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
692, in finish
   actions = self.handler(context, **args)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 515, in include
   processxmlfile(f, context)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 370, in processxmlfile
   parser.parse(src)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 109, in parse
   xmlreader.IncrementalParser.parse(self, source)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\xmlreader.py",
line 123, in parse
   self.feed(buffer)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 216, in feed
   self._parser.Parse(data, isFinal)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 364, in end_element_ns
   self._cont_handler.endElementNS(pair, None)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 349, in endElementNS
   self.context.end()
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
544, in end
   self.stack.pop().finish()
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
692, in finish
   actions = self.handler(context, **args)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 515, in include
   processxmlfile(f, context)
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 370, in processxmlfile
   parser.parse(src)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 109, in parse
   xmlreader.IncrementalParser.parse(self, source)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\xmlreader.py",
line 123, in parse
   self.feed(buffer)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 216, in feed
   self._parser.Parse(data, isFinal)
 File "D:\Plone3\Python\Lib\site-packages\_xmlplus\sax\expatreader.py",
line 353, in start_element_ns
   AttributesNSImpl(newattrs, qnames))
 File "D:\Plone3\Zope\lib\python\zope\configuration\xmlconfig.py",
line 222, in startElementNS
   self.context.begin(name, data, info)
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
541, in begin
   self.stack.append(self.stack[-1].contained(__name, __data, __info))
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
842, in contained
   return RootStackItem.contained(self, name, data, info)
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
710, in contained
   factory = self.context.factory(self.context, name)
 File "D:\Plone3\Zope\lib\python\zope\configuration\config.py", line
487, in factory
   raise ConfigurationError("Unknown directive", ns, n)
ZopeXMLConfigurationError: File
"d:\Plone3\Insite2\parts\instance\etc\site.zcml", line 15.2-15.23
   ZopeXMLConfigurationError: File
"d:\Plone3\Data\Products\CMFPlone\configure.zcml", line 8.2-8.46
   ZopeXMLConfigurationError: File
"d:\plone3\insite2\eggs\plone.app.contentrules-1.1.0a1-
py2.4.egg\plone\app\contentrules\configure.zcml",
line 22.4-22.39
   ZopeXMLConfigurationError: File
"d:\plone3\insite2\eggs\plone.app.contentrules-1.1.0a1-
py2.4.egg\plone\app\contentrules\exportimport\configure.zcml",
line 16.4
   ConfigurationError: ('Unknown directive',
u'http://namespaces.zope.org/genericsetup', u'importStep')




From jim at zope.com  Mon Mar 10 21:28:08 2008
From: jim at zope.com (Jim Fulton)
Date: Mon, 10 Mar 2008 16:28:08 -0400
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <loom.20080310T200733-922@post.gmane.org>
References: <loom.20080310T200733-922@post.gmane.org>
Message-ID: <813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>


On Mar 10, 2008, at 4:11 PM, Michael NT Milne wrote:

> Been told to try this post from Plone users on this list....
>
> This is infuriating. Each time we install a new egg we have to run
> buildout in online mode as buildout -o or buildout -No just produces
> an invalid command error.

I don't understand what you are saying.  What exactly are you trying  
to do?

> buildout ?c c:\Plone\'yoursitename'\buildout.cfg
>
> The above works but it re-installs new eggs etc alongside the old eggs
> and the site breaks as below. When re-running buildout, Zope uses the
> eggs with the higher numbers. Do you need to change anything to make
> this work?

What work? I don't know what your desired outcome is.


> Also has anyone got Windows buildouts running in offline
> mode? It seems the only way to fix this is to delete the new eggs etc
> that have been fetched.

The error below is a Plone error:

> 2008-03-10T16:45:09 ERROR Application Couldn't install Five
> Traceback (most recent call last):
...
> line 22.4-22.39
>   ZopeXMLConfigurationError: File
> "d:\plone3\insite2\eggs\plone.app.contentrules-1.1.0a1-
> py2.4.egg\plone\app\contentrules\exportimport\configure.zcml",
> line 16.4
>   ConfigurationError: ('Unknown directive',
> u'http://namespaces.zope.org/genericsetup', u'importStep')


This looks like plone\app\contentrules\exportimport\configure.zcml" is  
missing a namespace declaration, but this has nothing to do with  
distutils.  I suggest going back to the plone list for help with this  
error. :)

Jim

--
Jim Fulton
Zope Corporation



From michael.milne at gmail.com  Mon Mar 10 21:40:42 2008
From: michael.milne at gmail.com (michael nt milne)
Date: Mon, 10 Mar 2008 20:40:42 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
Message-ID: <134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>

Hi

My question is simply, is there a command for running buildout on
Windows in offline and not newest mode? In the docs buildout -o or -No
or -Nov is suggested but none of these work on Windows. We can only
get buildout to work using buildout -c on Windows.

The further error is probably the result of getting the new Plone 3.1
through easy_install

Thanks

On Mon, Mar 10, 2008 at 8:28 PM, Jim Fulton <jim at zope.com> wrote:
>
>  On Mar 10, 2008, at 4:11 PM, Michael NT Milne wrote:
>
>  > Been told to try this post from Plone users on this list....
>  >
>  > This is infuriating. Each time we install a new egg we have to run
>  > buildout in online mode as buildout -o or buildout -No just produces
>  > an invalid command error.
>
>  I don't understand what you are saying.  What exactly are you trying
>  to do?
>
>  > buildout ?c c:\Plone\'yoursitename'\buildout.cfg
>  >
>  > The above works but it re-installs new eggs etc alongside the old eggs
>  > and the site breaks as below. When re-running buildout, Zope uses the
>  > eggs with the higher numbers. Do you need to change anything to make
>  > this work?
>
>  What work? I don't know what your desired outcome is.
>
>
>  > Also has anyone got Windows buildouts running in offline
>  > mode? It seems the only way to fix this is to delete the new eggs etc
>  > that have been fetched.
>
>  The error below is a Plone error:
>
>  > 2008-03-10T16:45:09 ERROR Application Couldn't install Five
>  > Traceback (most recent call last):
>  ...
>  > line 22.4-22.39
>  >   ZopeXMLConfigurationError: File
>  > "d:\plone3\insite2\eggs\plone.app.contentrules-1.1.0a1-
>  > py2.4.egg\plone\app\contentrules\exportimport\configure.zcml",
>  > line 16.4
>  >   ConfigurationError: ('Unknown directive',
>  > u'http://namespaces.zope.org/genericsetup', u'importStep')
>
>
>  This looks like plone\app\contentrules\exportimport\configure.zcml" is
>  missing a namespace declaration, but this has nothing to do with
>  distutils.  I suggest going back to the plone list for help with this
>  error. :)
>
>  Jim
>
>  --
>  Jim Fulton
>  Zope Corporation
>
>
>



-- 
michael

From optilude at gmx.net  Mon Mar 10 21:49:30 2008
From: optilude at gmx.net (Martin Aspeli)
Date: Mon, 10 Mar 2008 20:49:30 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
 work...any alternatives?
In-Reply-To: <813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
Message-ID: <fr46su$41u$1@ger.gmane.org>

Jim Fulton wrote:
> On Mar 10, 2008, at 4:11 PM, Michael NT Milne wrote:
> 
>> Been told to try this post from Plone users on this list....
>>
>> This is infuriating. Each time we install a new egg we have to run
>> buildout in online mode as buildout -o or buildout -No just produces
>> an invalid command error.
> 
> I don't understand what you are saying.  What exactly are you trying  
> to do?
> 
>> buildout ?c c:\Plone\'yoursitename'\buildout.cfg
>>
>> The above works but it re-installs new eggs etc alongside the old eggs
>> and the site breaks as below. When re-running buildout, Zope uses the
>> eggs with the higher numbers. Do you need to change anything to make
>> this work?
> 
> What work? I don't know what your desired outcome is.
> 
> 
>> Also has anyone got Windows buildouts running in offline
>> mode? It seems the only way to fix this is to delete the new eggs etc
>> that have been fetched.
> 
> The error below is a Plone error:
> 
>> 2008-03-10T16:45:09 ERROR Application Couldn't install Five
>> Traceback (most recent call last):
> ...
>> line 22.4-22.39
>>   ZopeXMLConfigurationError: File
>> "d:\plone3\insite2\eggs\plone.app.contentrules-1.1.0a1-
>> py2.4.egg\plone\app\contentrules\exportimport\configure.zcml",
>> line 16.4
>>   ConfigurationError: ('Unknown directive',
>> u'http://namespaces.zope.org/genericsetup', u'importStep')
> 
> 
> This looks like plone\app\contentrules\exportimport\configure.zcml" is  
> missing a namespace declaration, but this has nothing to do with  
> distutils.  I suggest going back to the plone list for help with this  
> error. :)

I think what's going on is that non-newest mode is trying to download 
things and update things, and offline mode is giving an error.

Some packages have gotten upgraded accidentally, including 
plone.app.contentrules, to 3.1 beta 1 versions, so Michael has a version 
of Plone that's half 3.0.6 and half 3.1.

The problem with -o downloading things is a buildout issue, though - 
whether a bug or user error, I can't tell.

Martin

-- 
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


From jim at zope.com  Mon Mar 10 21:52:52 2008
From: jim at zope.com (Jim Fulton)
Date: Mon, 10 Mar 2008 16:52:52 -0400
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
Message-ID: <13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>


On Mar 10, 2008, at 4:40 PM, michael nt milne wrote:

> Hi
>
> My question is simply, is there a command for running buildout on
> Windows in offline and not newest mode?

I'm 99% sure they work just find on Windows. I have test for them and  
the tests pass on Windows.

> In the docs buildout -o or -No
> or -Nov is suggested but none of these work on Windows.

I think you are mistaken.  What makes you think they don't work?

> We can only
> get buildout to work using buildout -c on Windows.

The -c option and the -N and -o options are unrelated.

If you don't want to use the buildout.cfg from the current working  
directory, then you need to use -c to specify an alternate buildout  
configuration.

I suggest using -c and -No (or just -o) together.

Jim

--
Jim Fulton
Zope Corporation



From optilude at gmx.net  Mon Mar 10 21:50:08 2008
From: optilude at gmx.net (Martin Aspeli)
Date: Mon, 10 Mar 2008 20:50:08 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
 work...any alternatives?
In-Reply-To: <134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
References: <loom.20080310T200733-922@post.gmane.org>	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
Message-ID: <fr46u3$41u$2@ger.gmane.org>

michael nt milne wrote:
> Hi
> 
> My question is simply, is there a command for running buildout on
> Windows in offline and not newest mode? In the docs buildout -o or -No
> or -Nov is suggested but none of these work on Windows. We can only
> get buildout to work using buildout -c on Windows.
> 
> The further error is probably the result of getting the new Plone 3.1
> through easy_install

Did you manually install bits of Plone 3.1 into your system Python with 
easy_install? If so, all bets are off...

Martin

-- 
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


From jim at zope.com  Mon Mar 10 21:56:35 2008
From: jim at zope.com (Jim Fulton)
Date: Mon, 10 Mar 2008 16:56:35 -0400
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <fr46su$41u$1@ger.gmane.org>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<fr46su$41u$1@ger.gmane.org>
Message-ID: <18C68CEB-1664-4902-AA93-909D61D6B3C9@zope.com>


On Mar 10, 2008, at 4:49 PM, Martin Aspeli wrote:
> I think what's going on is that non-newest mode is trying to download
> things and update things, and offline mode is giving an error.

There error is from zope.configuration. It has nothing to do with  
buildout. Non-newest mode will specifically avoid downloading  
distributions unless there is an unmet requirement.  No evidence has  
been given that this is failing and there are tests of this machinery  
that pass on Windows.

> Some packages have gotten upgraded accidentally, including
> plone.app.contentrules, to 3.1 beta 1 versions, so Michael has a  
> version
> of Plone that's half 3.0.6 and half 3.1.
>
> The problem with -o downloading things is a buildout issue, though -
> whether a bug or user error, I can't tell.


-o prevents downloads by any recipes that pay attention to it.

I'd be wildly surprised if there was a failure in the -o option.  I  
certainly haven't seen any evidence that there is. The only output  
that's been given in this thread was from running buildout without the  
option.

JIm

--
Jim Fulton
Zope Corporation



From jim at zope.com  Mon Mar 10 21:58:16 2008
From: jim at zope.com (Jim Fulton)
Date: Mon, 10 Mar 2008 16:58:16 -0400
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <fr46u3$41u$2@ger.gmane.org>
References: <loom.20080310T200733-922@post.gmane.org>	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<fr46u3$41u$2@ger.gmane.org>
Message-ID: <F9807003-AC3A-4F73-9BC3-6C3E628A009A@zope.com>


On Mar 10, 2008, at 4:50 PM, Martin Aspeli wrote:

> michael nt milne wrote:
>> Hi
>>
>> My question is simply, is there a command for running buildout on
>> Windows in offline and not newest mode? In the docs buildout -o or - 
>> No
>> or -Nov is suggested but none of these work on Windows. We can only
>> get buildout to work using buildout -c on Windows.
>>
>> The further error is probably the result of getting the new Plone 3.1
>> through easy_install
>
> Did you manually install bits of Plone 3.1 into your system Python  
> with
> easy_install? If so, all bets are off...


He's been running buildout without -N or -o so I'm sure he has the  
latest eggs.

Perhaps someone should give him a working versions.cfg.

Jim

--
Jim Fulton
Zope Corporation



From michael.milne at gmail.com  Mon Mar 10 21:59:08 2008
From: michael.milne at gmail.com (michael nt milne)
Date: Mon, 10 Mar 2008 20:59:08 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
Message-ID: <134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>

>I suggest using -c and -No (or just -o) together

Ok thanks for the advice. I will try this.

On Mon, Mar 10, 2008 at 8:52 PM, Jim Fulton <jim at zope.com> wrote:
>
>  On Mar 10, 2008, at 4:40 PM, michael nt milne wrote:
>
>  > Hi
>  >
>  > My question is simply, is there a command for running buildout on
>  > Windows in offline and not newest mode?
>
>  I'm 99% sure they work just find on Windows. I have test for them and
>  the tests pass on Windows.
>
>
>  > In the docs buildout -o or -No
>  > or -Nov is suggested but none of these work on Windows.
>
>  I think you are mistaken.  What makes you think they don't work?
>
>
>  > We can only
>  > get buildout to work using buildout -c on Windows.
>
>  The -c option and the -N and -o options are unrelated.
>
>  If you don't want to use the buildout.cfg from the current working
>  directory, then you need to use -c to specify an alternate buildout
>  configuration.
>
>  I suggest using -c and -No (or just -o) together.
>
>
>
>  Jim
>
>  --
>  Jim Fulton
>  Zope Corporation
>
>
>



-- 
michael

From jim at zope.com  Mon Mar 10 22:07:59 2008
From: jim at zope.com (Jim Fulton)
Date: Mon, 10 Mar 2008 17:07:59 -0400
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
Message-ID: <85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>


On Mar 10, 2008, at 4:59 PM, michael nt milne wrote:

>> I suggest using -c and -No (or just -o) together
>
> Ok thanks for the advice. I will try this.


Note that this won't help unless you have a working starting point. If  
you don't, you'll need to manually remove unwanted eggs from your eggs  
directory.

Alternatively, you can use a versions option to specify which versions  
to use.

Jim

--
Jim Fulton
Zope Corporation



From optilude at gmx.net  Mon Mar 10 22:19:34 2008
From: optilude at gmx.net (Martin Aspeli)
Date: Mon, 10 Mar 2008 21:19:34 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
 work...any alternatives?
In-Reply-To: <85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
References: <loom.20080310T200733-922@post.gmane.org>	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
	<85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
Message-ID: <fr48la$bu8$1@ger.gmane.org>

Jim Fulton wrote:
> On Mar 10, 2008, at 4:59 PM, michael nt milne wrote:
> 
>>> I suggest using -c and -No (or just -o) together
>> Ok thanks for the advice. I will try this.
> 
> 
> Note that this won't help unless you have a working starting point. If  
> you don't, you'll need to manually remove unwanted eggs from your eggs  
> directory.
> 
> Alternatively, you can use a versions option to specify which versions  
> to use.

Michael, you may want to remove your .installed.cfg (to start from 
scratch) and then change the [plone] block to:

[plone]
recipe=plone.recipe.plone==3.0.6

That'll pin all eggs to the 3.0.6 version.

Martin

-- 
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


From michael.milne at gmail.com  Tue Mar 11 12:53:53 2008
From: michael.milne at gmail.com (michael nt milne)
Date: Tue, 11 Mar 2008 11:53:53 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
	<85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
Message-ID: <134b8b710803110453w540075daw1512a53060fe1e05@mail.gmail.com>

Hi

I've tried the following commands to get the offline mode to work with
no success

D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o buildout.cfg

Error: invalid command: buildout.cfg

D:\Plone3\Insite2\bin>buildout -c-o d:\Plone3\Insite2\buildout.cfg

Error: invalid command: d:\Plone3\Insite2\buildout.cfg

The only one which will work is this

D:\Plone3\Insite2\bin>buildout -c d:\Plone3\Insite2\buildout.cfg

The path is completely correct

On Mon, Mar 10, 2008 at 9:07 PM, Jim Fulton <jim at zope.com> wrote:
>
>  On Mar 10, 2008, at 4:59 PM, michael nt milne wrote:
>
>  >> I suggest using -c and -No (or just -o) together
>  >
>  > Ok thanks for the advice. I will try this.
>
>
>  Note that this won't help unless you have a working starting point. If
>  you don't, you'll need to manually remove unwanted eggs from your eggs
>  directory.
>
>  Alternatively, you can use a versions option to specify which versions
>  to use.
>
>
>
>  Jim
>
>  --
>  Jim Fulton
>  Zope Corporation
>
>
>



-- 
michael

From tim.terlegard at valentinewebsystems.se  Tue Mar 11 14:27:45 2008
From: tim.terlegard at valentinewebsystems.se (=?ISO-8859-1?Q?Tim_Terleg=E5rd?=)
Date: Tue, 11 Mar 2008 14:27:45 +0100
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <134b8b710803110453w540075daw1512a53060fe1e05@mail.gmail.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
	<85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
	<134b8b710803110453w540075daw1512a53060fe1e05@mail.gmail.com>
Message-ID: <E2B018CA-0534-4BA3-98F7-9001147FB2A5@valentinewebsystems.se>

On Mar 11, 2008, at 12:53 PM, michael nt milne wrote:

> I've tried the following commands to get the offline mode to work with
> no success
>
> D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o buildout.cfg
>
> Error: invalid command: buildout.cfg

You're using the arguments wrong. "buildout -h" will give you the
details of the options and arguments. Either you don't specify a
configuration file at all and buildout.cfg will automatically be used,
or you specify a configuration file with "-c someconf.cfg".

So try:

     D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o

/Tim

From michael.milne at gmail.com  Tue Mar 11 14:37:18 2008
From: michael.milne at gmail.com (michael nt milne)
Date: Tue, 11 Mar 2008 13:37:18 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <E2B018CA-0534-4BA3-98F7-9001147FB2A5@valentinewebsystems.se>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
	<85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
	<134b8b710803110453w540075daw1512a53060fe1e05@mail.gmail.com>
	<E2B018CA-0534-4BA3-98F7-9001147FB2A5@valentinewebsystems.se>
Message-ID: <134b8b710803110637k3178666dv114d45ebf8b0625b@mail.gmail.com>

thanks

ok so why doesn't

D:\Plone3\Insite2\bin>buildout -c-o d:\Plone3\Insite2\buildout.cfg

work then?

I'll give D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o

a try

On Tue, Mar 11, 2008 at 1:27 PM, Tim Terleg?rd
<tim.terlegard at valentinewebsystems.se> wrote:
> On Mar 11, 2008, at 12:53 PM, michael nt milne wrote:
>
>  > I've tried the following commands to get the offline mode to work with
>  > no success
>  >
>  > D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o buildout.cfg
>  >
>  > Error: invalid command: buildout.cfg
>
>  You're using the arguments wrong. "buildout -h" will give you the
>  details of the options and arguments. Either you don't specify a
>  configuration file at all and buildout.cfg will automatically be used,
>  or you specify a configuration file with "-c someconf.cfg".
>
>  So try:
>
>
>
>      D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o
>
>  /Tim
>



-- 
michael

From jim at zope.com  Tue Mar 11 14:45:43 2008
From: jim at zope.com (Jim Fulton)
Date: Tue, 11 Mar 2008 09:45:43 -0400
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <134b8b710803110637k3178666dv114d45ebf8b0625b@mail.gmail.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
	<85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
	<134b8b710803110453w540075daw1512a53060fe1e05@mail.gmail.com>
	<E2B018CA-0534-4BA3-98F7-9001147FB2A5@valentinewebsystems.se>
	<134b8b710803110637k3178666dv114d45ebf8b0625b@mail.gmail.com>
Message-ID: <CE3DB1C7-859D-4B71-BD6A-B59D496ABA3A@zope.com>


On Mar 11, 2008, at 9:37 AM, michael nt milne wrote:

> thanks
>
> ok so why doesn't
>
> D:\Plone3\Insite2\bin>buildout -c-o d:\Plone3\Insite2\buildout.cfg
>
> work then?

Because the config file name is an argument to the -c option, *not*  
the -o option.

Jim

--
Jim Fulton
Zope Corporation



From michael.milne at gmail.com  Tue Mar 11 14:57:28 2008
From: michael.milne at gmail.com (michael nt milne)
Date: Tue, 11 Mar 2008 13:57:28 +0000
Subject: [Distutils] offline mode for Windows buildout -o or -No doesn't
	work...any alternatives?
In-Reply-To: <CE3DB1C7-859D-4B71-BD6A-B59D496ABA3A@zope.com>
References: <loom.20080310T200733-922@post.gmane.org>
	<813EDB2D-4234-401C-AE05-B07DCD090541@zope.com>
	<134b8b710803101340i39fe41c3w2dbc590c2b2de5e2@mail.gmail.com>
	<13DB3016-1A5E-4ED0-89B2-571D4CF797DC@zope.com>
	<134b8b710803101359h760a5f9nefe56d9731d4418d@mail.gmail.com>
	<85CA5EA2-68BB-4788-996A-15E117C1E148@zope.com>
	<134b8b710803110453w540075daw1512a53060fe1e05@mail.gmail.com>
	<E2B018CA-0534-4BA3-98F7-9001147FB2A5@valentinewebsystems.se>
	<134b8b710803110637k3178666dv114d45ebf8b0625b@mail.gmail.com>
	<CE3DB1C7-859D-4B71-BD6A-B59D496ABA3A@zope.com>
Message-ID: <134b8b710803110657u542ef132od1aa6b70803a14e7@mail.gmail.com>

ok thanks

D:\Plone3\Insite2>d:\Plone3\Insite2\bin\buildout -o

works but for some reason adding d:\Plone3\Insite2\bin to the path
environment variable doesn't work so can't do an easy buildout -o from
within D:\Plone3\Insite2\

Will just use the long winded methond..

On Tue, Mar 11, 2008 at 1:45 PM, Jim Fulton <jim at zope.com> wrote:
>
>  On Mar 11, 2008, at 9:37 AM, michael nt milne wrote:
>
>  > thanks
>  >
>  > ok so why doesn't
>  >
>  > D:\Plone3\Insite2\bin>buildout -c-o d:\Plone3\Insite2\buildout.cfg
>  >
>  > work then?
>
>  Because the config file name is an argument to the -c option, *not*
>  the -o option.
>
>
>
>  Jim
>
>  --
>  Jim Fulton
>  Zope Corporation
>
>
>



-- 
michael

From michael.milne at gmail.com  Thu Mar 13 17:41:17 2008
From: michael.milne at gmail.com (michael nt milne)
Date: Thu, 13 Mar 2008 16:41:17 +0000
Subject: [Distutils] moving a development egg over to a production server
	not using buildout...
Message-ID: <134b8b710803130941h708c9a39n657983198ca23f11@mail.gmail.com>

Hi

Is there some special command that is required for packing up an egg
which has been in development and is sitting in a buildout /src
directory, over to a live server. This server isn't using a buildout
config so it will go straight into site-packages. Does the egg need to
be zipped up before it can be easy_installed?

I've had a look at this page

http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options

What I am getting is a single .egg file and no folder structure for
the egg within site-packages.

Thanks

-- 
michael

From skip at pobox.com  Sun Mar 16 17:38:04 2008
From: skip at pobox.com (skip at pobox.com)
Date: Sun, 16 Mar 2008 11:38:04 -0500
Subject: [Distutils] *ugh* - failure to easy_install numpy
Message-ID: <18397.19692.881850.757972@montanaro-dyndns-org.local>


I have a new-ish MacBook Pro.  When I installed it I used Apple's migration
assitant to transfer content from my PowerBook.  That left me with lots of
Mach-O PPC shared object files.  I've been sort of working my way through
replacing things.

Today I tried reinstalling numpy using easy_install (had been 1.0.1 in
Mach-O PPC format, should wind up with 1.0.4 in Mach-O Intel format).  I got
this rather ugly output:

    % easy_install numpy
    /Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/package_index.py:7: DeprecationWarning: the md5 module is deprecated; use hashlib instead
      from md5 import md5
    Searching for numpy
    Reading http://pypi.python.org/simple/numpy/
    Reading http://numeric.scipy.org
    Reading http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
    Reading http://numpy.scipy.org
    Best match: numpy 1.0.4
    Downloading http://downloads.sourceforge.net/numpy/numpy-1.0.4.tar.gz?modtime=1194536674&big_mirror=0
    Processing numpy-1.0.4.tar.gz
    Running numpy-1.0.4/setup.py -q bdist_egg --dist-dir /var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/egg-dist-tmp-X0k4SP
    Running from numpy source directory.
    non-existing path in '/private/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils': 'site.cfg'
    /var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py:1278: DeprecationWarning: os.popen4 is deprecated.  Use the subprocess module.
      sin, sout = os.popen4('svnversion')
    Traceback (most recent call last):
      File "/Users/skip/local/bin/easy_install", line 8, in <module>
        load_entry_point('setuptools==0.6c7', 'console_scripts', 'easy_install')()
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 1670, in main
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 1659, in with_ei_usage
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 1674, in <lambda>
      File "/Users/skip/local/lib/python2.6/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/Users/skip/local/lib/python2.6/distutils/dist.py", line 974, in run_commands
        self.run_command(cmd)
      File "/Users/skip/local/lib/python2.6/distutils/dist.py", line 994, in run_command
        cmd_obj.run()
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 211, in run
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 446, in easy_install
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 471, in install_item
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 655, in install_eggs
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 930, in build_and_install
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 919, in run_setup
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/sandbox.py", line 27, in run_setup
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/sandbox.py", line 63, in run
      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-0.6c7-py2.6.egg/setuptools/sandbox.py", line 29, in <lambda>
      File "setup.py", line 89, in <module>
      File "setup.py", line 82, in setup_package
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/core.py", line 145, in setup
      File "setup.py", line 49, in configuration
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py", line 834, in add_subpackage
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py", line 817, in get_subpackage
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py", line 764, in _get_configuration_from_setup_py
      File "numpy/setup.py", line 9, in configuration
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py", line 834, in add_subpackage
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py", line 817, in get_subpackage
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/misc_util.py", line 764, in _get_configuration_from_setup_py
      File "/private/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/core/setup.py", line 246, in configuration
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/system_info.py", line 267, in get_info
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/system_info.py", line 366, in __init__
      File "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-1.0.4/numpy/distutils/system_info.py", line 374, in parse_config_files
      File "/Users/skip/local/lib/python2.6/ConfigParser.py", line 242, in add_section
        raise ValueError, 'Invalid section name: %s' % section
    ValueError: Invalid section name: DEFAULT

Any idea what the problem might be?  I'm using easy_install 0.6c7 with
Python 2.6 (Subversion checkout).  I don't know if 0.6c7 is the most recent
version or not.

Thanks,

-- 
Skip Montanaro - skip at pobox.com - http://www.webfast.com/~skip/

From dpeterson at enthought.com  Sun Mar 16 18:06:07 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Sun, 16 Mar 2008 12:06:07 -0500
Subject: [Distutils] *ugh* - failure to easy_install numpy
In-Reply-To: <18397.19692.881850.757972@montanaro-dyndns-org.local>
References: <18397.19692.881850.757972@montanaro-dyndns-org.local>
Message-ID: <47DD537F.7080101@enthought.com>

skip at pobox.com wrote:
> Any idea what the problem might be?  I'm using easy_install 0.6c7 with
> Python 2.6 (Subversion checkout).  I don't know if 0.6c7 is the most recent
> version or not.
>   

Sorry, I've got no idea on the problem but I can answer your other 
question.  There is at least a 0.6c8 that has been released.
I'm pretty sure you'll always be able to check for the latest version here:
     http://pypi.python.org/pypi/setuptools

-- Dave


From ziade.tarek at gmail.com  Sun Mar 16 18:29:40 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 16 Mar 2008 18:29:40 +0100
Subject: [Distutils] *ugh* - failure to easy_install numpy
In-Reply-To: <18397.19692.881850.757972@montanaro-dyndns-org.local>
References: <18397.19692.881850.757972@montanaro-dyndns-org.local>
Message-ID: <94bdd2610803161029p40013eddqa3b8146762f9a8ba@mail.gmail.com>

On Sun, Mar 16, 2008 at 5:38 PM, <skip at pobox.com> wrote:

>
> I have a new-ish MacBook Pro.  When I installed it I used Apple's
> migration
> assitant to transfer content from my PowerBook.  That left me with lots of
> Mach-O PPC shared object files.  I've been sort of working my way through
> replacing things.
>
> Today I tried reinstalling numpy using easy_install (had been 1.0.1 in
> Mach-O PPC format, should wind up with 1.0.4 in Mach-O Intel format).  I
> got
> this rather ugly output:
>

It looks like numpy tries to add a DEFAULT section in the site.cfgconfiguration
file, which seems now forbidden in Python 2.6 ConfigParser (reserved
section).

...
        if section.lower() == "default":
            raise ValueError, 'Invalid section name: %s' % section
...

So I'd say that numpy (I don't know this lib) is not compatible with 2.6 yet
(it works in 2.5)



>
>    % easy_install numpy
>    /Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/package_index.py:7: DeprecationWarning: the md5
> module is deprecated; use hashlib instead
>      from md5 import md5
>    Searching for numpy
>    Reading http://pypi.python.org/simple/numpy/
>    Reading http://numeric.scipy.org
>    Reading
> http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103
>    Reading http://numpy.scipy.org
>    Best match: numpy 1.0.4
>    Downloading
> http://downloads.sourceforge.net/numpy/numpy-1.0.4.tar.gz?modtime=1194536674&big_mirror=0
>    Processing numpy-1.0.4.tar.gz
>    Running numpy-1.0.4/setup.py -q bdist_egg --dist-dir
> /var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/egg-dist-tmp-X0k4SP
>    Running from numpy source directory.
>    non-existing path in
> '/private/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils': 'site.cfg'
>
>  /var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py:1278: DeprecationWarning: os.popen4 is
> deprecated.  Use the subprocess module.
>      sin, sout = os.popen4('svnversion')
>    Traceback (most recent call last):
>      File "/Users/skip/local/bin/easy_install", line 8, in <module>
>        load_entry_point('setuptools==0.6c7', 'console_scripts',
> 'easy_install')()
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 1670, in main
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 1659, in
> with_ei_usage
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 1674, in
> <lambda>
>      File "/Users/skip/local/lib/python2.6/distutils/core.py", line 151,
> in setup
>        dist.run_commands()
>      File "/Users/skip/local/lib/python2.6/distutils/dist.py", line 974,
> in run_commands
>        self.run_command(cmd)
>      File "/Users/skip/local/lib/python2.6/distutils/dist.py", line 994,
> in run_command
>        cmd_obj.run()
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 211, in run
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 446, in
> easy_install
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 471, in
> install_item
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 655, in
> install_eggs
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 930, in
> build_and_install
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/command/easy_install.py", line 919, in
> run_setup
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/sandbox.py", line 27, in run_setup
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/sandbox.py", line 63, in run
>      File "/Users/skip/local/lib/python2.6/site-packages/setuptools-
> 0.6c7-py2.6.egg/setuptools/sandbox.py", line 29, in <lambda>
>      File "setup.py", line 89, in <module>
>      File "setup.py", line 82, in setup_package
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/core.py", line 145, in setup
>      File "setup.py", line 49, in configuration
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py", line 834, in add_subpackage
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py", line 817, in get_subpackage
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py", line 764, in
> _get_configuration_from_setup_py
>      File "numpy/setup.py", line 9, in configuration
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py", line 834, in add_subpackage
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py", line 817, in get_subpackage
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/misc_util.py", line 764, in
> _get_configuration_from_setup_py
>      File
> "/private/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/core/setup.py", line 246, in configuration
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/system_info.py", line 267, in get_info
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/system_info.py", line 366, in __init__
>      File
> "/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/easy_install-dLgdmR/numpy-
> 1.0.4/numpy/distutils/system_info.py", line 374, in parse_config_files
>      File "/Users/skip/local/lib/python2.6/ConfigParser.py", line 242, in
> add_section
>        raise ValueError, 'Invalid section name: %s' % section
>    ValueError: Invalid section name: DEFAULT
>
> Any idea what the problem might be?  I'm using easy_install 0.6c7 with
> Python 2.6 (Subversion checkout).  I don't know if 0.6c7 is the most
> recent
> version or not.
>
> Thanks,
>
> --
> Skip Montanaro - skip at pobox.com - http://www.webfast.com/~skip/<http://www.webfast.com/%7Eskip/>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080316/740aab58/attachment.htm 

From skip at pobox.com  Sun Mar 16 18:53:09 2008
From: skip at pobox.com (skip at pobox.com)
Date: Sun, 16 Mar 2008 12:53:09 -0500
Subject: [Distutils] *ugh* - failure to easy_install numpy
In-Reply-To: <47DD537F.7080101@enthought.com>
References: <18397.19692.881850.757972@montanaro-dyndns-org.local>
	<47DD537F.7080101@enthought.com>
Message-ID: <18397.24197.127753.39731@montanaro-dyndns-org.local>


    Dave> Sorry, I've got no idea on the problem but I can answer your other
    Dave> question.  There is at least a 0.6c8 that has been released.  I'm
    Dave> pretty sure you'll always be able to check for the latest version
    Dave> here:
    Dave>      http://pypi.python.org/pypi/setuptools

Thanks.  I was trying to 

    easy_install easy_install

:-)  Maybe "easy_install" could be an alias for "setuptools"?

Skip

From pje at telecommunity.com  Mon Mar 17 01:06:24 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 16 Mar 2008 20:06:24 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
Message-ID: <20080317000630.735823A40B0@sparrow.telecommunity.com>

Quick summary of the below: I'm definitely fine with doing a simpler, 
pure-bootstrap module, if there's some consensus on what should go in 
it.  I just wish we could've had this discussion last year, when OSAF 
was still able to fund the work...  ;-)


At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote:
>Phillip asked me to give an opinion on his pkg_resources PEP. While
>the PEP is short and sweet, the pkg_resources module itself is huge
>(1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67
>names in total according to __all__). And pkg_resources.txt is another
>1700 lines of documentation. I find that hard to swallow. Is there
>anyone besides Phillip who can claim he understands this module?

Bob Ippolito actually wrote the very first version of 
pkg_resources.  Others, such as Philip Jenvey of the Jython project, 
have provided patches.  From previous discussions on the 
distutils-sig, I know that Jim Fulton has in-depth knowledge of both 
pkg_resources and easy_install.

Of course, that's not the same as any of these guys volunteering to 
be maintainers.  :)


>If its inclusion is really meant just as a bootstrap to simplify
>installing other package management solutions, as the PEP claims, I
>would prefer to see something with a much smaller footprint.

Actually, the PEP says:

"pkg_resources is a module used to find and manage Python 
package/version dependencies and access bundled files and resources, 
including those inside of zipped .egg files. Currently, pkg_resources 
is only available through installing the entire setuptools 
distribution, but it does not depend on any other part of setuptools; 
in effect, it comprises the entire runtime support library for Python 
Eggs, and is independently useful."

This kind of glosses over the part where this is also for runtime 
support of projects that use eggs.  Which, these days, is, well, 
almost any large Python project, from Chandler to Enthought to Zope.


>  Surely
>there is no need for example to have support for C extensions inside
>zip files *as part of the bootstrap module*?

It's a runtime; the PEP actually merely proposes that a further 
addition to be made to support bootstrapping, *also*.  Otherwise, the 
PEP would be even shorter.  :)

The reason I proposed it this way was for simplicity -- and politics.

Currently, people using setuptools in their setup.py have to include 
a similar bootstrap module to download setuptools if it's not 
available, and pkg_resources already has version checking logic and 
everything needed to find dependencies and download them.  (Plus, I 
figured it'd be easier to just use what was already there and stable, 
rather than creating something different.)

That was the simplicity part.  The politics part was that:

1. I thought it would be less controversial to include the "runtime 
for eggs" than to include something that's just a bootstrapper for 
setuptools.  However, MvL surprised me by actually being in *favor* 
of including a setuptools bootstrapper.

2. I thought that it would have broader acceptance if it was oriented 
towards bootstrapping *any* package, not just setuptools.

So, if the consensus is that it would be better to have a module that 
only does bootstrap installs of pure-Python eggs from PyPI, I'm 
totally fine with that.


>Unless I find someone besides Phillip who is interested in having this
>included and is willing to help maintain it, I don't really think it
>would be wise to accept this into the standard library.
>
>Phillip, in the PEP you mention that there are several other package
>management tools that also like to use pkg_resources. Maybe you can
>get some folks from those tools to speak up and explain what
>pkg_resources means to them, and maybe even volunteer to co-own it
>once it's in the standard library?

The distutils-sig is the de facto place for discussions regarding 
those tools, so I've cc'd this there.  Hopefully, one or more 
volunteers will step up if they want this.


From noah.gift at gmail.com  Mon Mar 17 03:09:42 2008
From: noah.gift at gmail.com (Noah Gift)
Date: Sun, 16 Mar 2008 21:09:42 -0500
Subject: [Distutils] virtualenv not respecting --no-site-packages for ZODB3?
Message-ID: <F0775826-F035-42FF-9CD4-BAA08ADCBB00@gmail.com>

I am getting the following behavior when using virtualenv to  
easy_install ZODB3.

[ngift at Macintosh-7][H:10256][J:0]# virtualenv --no-site-packages zodbENV
New python executable in zodbENV/bin/python
Installing setuptools..........................done.
[ngift at Macintosh-7][H:10257][J:0]# zodbENV/bin/easy_install ZODB3
Searching for ZODB3
Best match: ZODB3 3.8.0
Processing ZODB3-3.8.0-py2.5-macosx-10.5-i386.egg
Adding ZODB3 3.8.0 to easy-install.pth file
Installing mkzeoinst script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing fstail script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing zeopack script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing runzeo script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing zeopasswd script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing zeoctl script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing fsdump script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing fsrefs script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing repozo script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin
Installing fsoids script to /Users/ngift/Documents/writing/IBM/ZODB/ 
zodbENV/bin

Using /Library/Python/2.5/site-packages/ZODB3-3.8.0-py2.5-macosx-10.5- 
i386.egg
Processing dependencies for ZODB3
Searching for zope.interface
Reading http://pypi.python.org/simple/zope.interface/
Reading http://zope.org/Wikis/Interfaces/FrontPage
Best match: zope.interface 3.4.1
Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.4.1.tar.gz#md5 
=b085f4a774adab688e037ad32fbbf08e
Processing zope.interface-3.4.1.tar.gz
Running zope.interface-3.4.1/setup.py -q bdist_egg --dist-dir /var/ 
folders/LZ/LZFo5h8JEW4Jzr+ydkXfI++++TI/-Tmp-/easy_install-2eJgo2/ 
zope.interface-3.4.1/egg-dist-tmp-qzXWo6
Adding zope.interface 3.4.1 to easy-install.pth file

Installed /Users/ngift/Documents/writing/IBM/ZODB/zodbENV/lib/ 
python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.5-i386.egg
Finished processing dependencies for ZODB3


From eikeon at eikeon.com  Mon Mar 17 02:16:22 2008
From: eikeon at eikeon.com (Daniel Krech)
Date: Sun, 16 Mar 2008 21:16:22 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317000630.735823A40B0@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
Message-ID: <FA817330-AFBD-46BF-A671-3DBE134767F1@eikeon.com>


On Mar 16, 2008, at 8:06 PM, Phillip J. Eby wrote:

> Quick summary of the below: I'm definitely fine with doing a simpler,
> pure-bootstrap module, if there's some consensus on what should go in
> it.  I just wish we could've had this discussion last year, when OSAF
> was still able to fund the work...  ;-)
>
>
> At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote:
>> Phillip asked me to give an opinion on his pkg_resources PEP. While
>> the PEP is short and sweet, the pkg_resources module itself is huge
>> (1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67
>> names in total according to __all__). And pkg_resources.txt is  
>> another
>> 1700 lines of documentation. I find that hard to swallow. Is there
>> anyone besides Phillip who can claim he understands this module?
>
> Bob Ippolito actually wrote the very first version of
> pkg_resources.  Others, such as Philip Jenvey of the Jython project,
> have provided patches.  From previous discussions on the
> distutils-sig, I know that Jim Fulton has in-depth knowledge of both
> pkg_resources and easy_install.
>
> Of course, that's not the same as any of these guys volunteering to
> be maintainers.  :)
>
>
>> If its inclusion is really meant just as a bootstrap to simplify
>> installing other package management solutions, as the PEP claims, I
>> would prefer to see something with a much smaller footprint.
>
> Actually, the PEP says:
>
> "pkg_resources is a module used to find and manage Python
> package/version dependencies and access bundled files and resources,
> including those inside of zipped .egg files. Currently, pkg_resources
> is only available through installing the entire setuptools
> distribution, but it does not depend on any other part of setuptools;
> in effect, it comprises the entire runtime support library for Python
> Eggs, and is independently useful."
>
> This kind of glosses over the part where this is also for runtime
> support of projects that use eggs.  Which, these days, is, well,
> almost any large Python project, from Chandler to Enthought to Zope.
>
>
>> Surely
>> there is no need for example to have support for C extensions inside
>> zip files *as part of the bootstrap module*?
>
> It's a runtime; the PEP actually merely proposes that a further
> addition to be made to support bootstrapping, *also*.  Otherwise, the
> PEP would be even shorter.  :)
>
> The reason I proposed it this way was for simplicity -- and politics.
>
> Currently, people using setuptools in their setup.py have to include
> a similar bootstrap module to download setuptools if it's not
> available, and pkg_resources already has version checking logic and
> everything needed to find dependencies and download them.  (Plus, I
> figured it'd be easier to just use what was already there and stable,
> rather than creating something different.)
>
> That was the simplicity part.  The politics part was that:
>
> 1. I thought it would be less controversial to include the "runtime
> for eggs" than to include something that's just a bootstrapper for
> setuptools.  However, MvL surprised me by actually being in *favor*
> of including a setuptools bootstrapper.
>
> 2. I thought that it would have broader acceptance if it was oriented
> towards bootstrapping *any* package, not just setuptools.
>
> So, if the consensus is that it would be better to have a module that
> only does bootstrap installs of pure-Python eggs from PyPI, I'm
> totally fine with that.
>
>
>> Unless I find someone besides Phillip who is interested in having  
>> this
>> included and is willing to help maintain it, I don't really think it
>> would be wise to accept this into the standard library.
>>
>> Phillip, in the PEP you mention that there are several other package
>> management tools that also like to use pkg_resources. Maybe you can
>> get some folks from those tools to speak up and explain what
>> pkg_resources means to them, and maybe even volunteer to co-own it
>> once it's in the standard library?
>
> The distutils-sig is the de facto place for discussions regarding
> those tools, so I've cc'd this there.  Hopefully, one or more
> volunteers will step up if they want this.

I'd like to see it in and am willing to help maintain it. Especially  
if it only does the bootstrap installs of pure-Python egg bit. I've  
dug into pkg_resource some, but can't claim to understand it all.


From guido at python.org  Mon Mar 17 14:48:51 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 08:48:51 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317000630.735823A40B0@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
Message-ID: <ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>

On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
>  So, if the consensus is that it would be better to have a module that
>  only does bootstrap installs of pure-Python eggs from PyPI, I'm
>  totally fine with that.

Let's just do this; it will avoid a protracted discussion of the
merits of eggs, pkg_resources, and setuptools.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Mon Mar 17 15:19:11 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 10:19:11 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
Message-ID: <20080317142225.7C7D83A40B1@sparrow.telecommunity.com>

At 08:48 AM 3/17/2008 -0500, Guido van Rossum wrote:
>On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> >  So, if the consensus is that it would be better to have a module that
> >  only does bootstrap installs of pure-Python eggs from PyPI, I'm
> >  totally fine with that.
>
>Let's just do this; it will avoid a protracted discussion of the
>merits of eggs, pkg_resources, and setuptools.

Well, it might be replaced by a protracted discussion of how the 
module should work and what its API should be, but perhaps that would 
be a better one to have.  :)

So, the original proposal (from the previous thread about this) was 
that the module be named easy_install, and that it simply downloads 
setuptools and delegates to the "real" easy_install.  That way, 
people can simply use "python -m easy_install ...", without worrying 
about whether setuptools has been installed yet.

IIRC, other package management tools such as zc.buildout and 
workingenv can then be installed using easy_install.

Any objections?  Should I revise the PEP?


From martin at v.loewis.de  Mon Mar 17 15:45:25 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Mar 2008 09:45:25 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<20080317000630.735823A40B0@sparrow.telecommunity.com>	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
Message-ID: <47DE8405.2030201@v.loewis.de>

> Well, it might be replaced by a protracted discussion of how the 
> module should work and what its API should be, but perhaps that would 
> be a better one to have.  :)

Indeed, that's likely to happen :-)

> So, the original proposal (from the previous thread about this) was 
> that the module be named easy_install, and that it simply downloads 
> setuptools and delegates to the "real" easy_install.  That way, 
> people can simply use "python -m easy_install ...", without worrying 
> about whether setuptools has been installed yet.

I thought the original proposal was to install a *binary* easy_install
that takes that function.

> IIRC, other package management tools such as zc.buildout and 
> workingenv can then be installed using easy_install.
> 
> Any objections?  Should I revise the PEP?

I'm fine with the module, but would really like to see a command
line utility in addition.

This, of course, would raise the issue who "owns" the easy_install
script name; ideally, the script would not have to be overwritten
when setuptools gets installed.

Regards,
Martin

From pje at telecommunity.com  Mon Mar 17 16:10:36 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 11:10:36 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47DE8405.2030201@v.loewis.de>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
Message-ID: <20080317151637.532D23A409D@sparrow.telecommunity.com>

At 09:45 AM 3/17/2008 -0500, Martin v. L?wis wrote:
> > Well, it might be replaced by a protracted discussion of how the
> > module should work and what its API should be, but perhaps that would
> > be a better one to have.  :)
>
>Indeed, that's likely to happen :-)
>
> > So, the original proposal (from the previous thread about this) was
> > that the module be named easy_install, and that it simply downloads
> > setuptools and delegates to the "real" easy_install.  That way,
> > people can simply use "python -m easy_install ...", without worrying
> > about whether setuptools has been installed yet.
>
>I thought the original proposal was to install a *binary* easy_install
>that takes that function.

What do you mean by "binary"?  I thought we were talking about a 
module.  Do you mean a script to be installed alongside Python itself 
in e.g. /usr/bin?

In the original discussion, it was a module to be added alongside 
pkg_resources, which would use pkg_resources to find and/or install 
setuptools.  I also personally like the use of -m instead of a script 
because it makes it quite clear that this is a Python-specific 
installation tool, and *which* version of Python, as well, without 
having to have easy_install-2.5, easy_install-2.6, etc.


> > IIRC, other package management tools such as zc.buildout and
> > workingenv can then be installed using easy_install.
> >
> > Any objections?  Should I revise the PEP?
>
>I'm fine with the module, but would really like to see a command
>line utility in addition.
>
>This, of course, would raise the issue who "owns" the easy_install
>script name; ideally, the script would not have to be overwritten
>when setuptools gets installed.

It won't have to.  The module will attempt to import the 
setuptools-supplied version of easy_install, and delegate to it if 
possible, before trying to do any download.


From martin at v.loewis.de  Mon Mar 17 16:51:28 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Mar 2008 10:51:28 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317151637.532D23A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
Message-ID: <47DE9380.5000809@v.loewis.de>

>> I thought the original proposal was to install a *binary* easy_install
>> that takes that function.
> 
> What do you mean by "binary"?  I thought we were talking about a 
> module.  Do you mean a script to be installed alongside Python itself in 
> e.g. /usr/bin?

Exactly so.

> In the original discussion, it was a module to be added alongside 
> pkg_resources, which would use pkg_resources to find and/or install 
> setuptools.  I also personally like the use of -m instead of a script 
> because it makes it quite clear that this is a Python-specific 
> installation tool, and *which* version of Python, as well, without 
> having to have easy_install-2.5, easy_install-2.6, etc.

If that becomes the official interface to easy_install, that's fine
with me. I'm worried about web instructions that tell people that
there is an "easy_install" utility, so that people never find out
the module actually exists.

Regards,
Martin


From guido at python.org  Mon Mar 17 16:53:21 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 10:53:21 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317151637.532D23A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
Message-ID: <ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>

I don't think this should play games with scripts being overridden or
whatever. If a bootstrap script is to be installed it should have a
separate name. I'm not sure what the advantage is of a bootstrap
script over "python -m bootstrap_module ..." though.

The PEP suggests that other package managers also benefit. How do they
benefit if the bootstrap script installs setuptools? That sounds like
the bootstrap script would make setuptools the de-facto standard,
which I'd like to avoid given the politics around this topic.

I'd also like to avoid the specific name "easy_install" for any of
this. That's a "brand name" (and a misleading one if you ask me, but
that's politics again :-).

--Guido

On Mon, Mar 17, 2008 at 10:10 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 09:45 AM 3/17/2008 -0500, Martin v. L?wis wrote:
>  > > Well, it might be replaced by a protracted discussion of how the
>  > > module should work and what its API should be, but perhaps that would
>  > > be a better one to have.  :)
>  >
>  >Indeed, that's likely to happen :-)
>  >
>  > > So, the original proposal (from the previous thread about this) was
>  > > that the module be named easy_install, and that it simply downloads
>  > > setuptools and delegates to the "real" easy_install.  That way,
>  > > people can simply use "python -m easy_install ...", without worrying
>  > > about whether setuptools has been installed yet.
>  >
>  >I thought the original proposal was to install a *binary* easy_install
>  >that takes that function.
>
>  What do you mean by "binary"?  I thought we were talking about a
>  module.  Do you mean a script to be installed alongside Python itself
>  in e.g. /usr/bin?
>
>  In the original discussion, it was a module to be added alongside
>  pkg_resources, which would use pkg_resources to find and/or install
>  setuptools.  I also personally like the use of -m instead of a script
>  because it makes it quite clear that this is a Python-specific
>  installation tool, and *which* version of Python, as well, without
>  having to have easy_install-2.5, easy_install-2.6, etc.
>
>
>
>  > > IIRC, other package management tools such as zc.buildout and
>  > > workingenv can then be installed using easy_install.
>  > >
>  > > Any objections?  Should I revise the PEP?
>  >
>  >I'm fine with the module, but would really like to see a command
>  >line utility in addition.
>  >
>  >This, of course, would raise the issue who "owns" the easy_install
>  >script name; ideally, the script would not have to be overwritten
>  >when setuptools gets installed.
>
>  It won't have to.  The module will attempt to import the
>  setuptools-supplied version of easy_install, and delegate to it if
>  possible, before trying to do any download.
>
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Mon Mar 17 17:12:45 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 12:12:45 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
Message-ID: <20080317161305.22B183A409D@sparrow.telecommunity.com>

At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote:
>I don't think this should play games with scripts being overridden or
>whatever. If a bootstrap script is to be installed it should have a
>separate name. I'm not sure what the advantage is of a bootstrap
>script over "python -m bootstrap_module ..." though.

And -m also makes explicit:

1. that it's a Python-specific tool
2. which Python version it will apply to


>The PEP suggests that other package managers also benefit. How do they
>benefit if the bootstrap script installs setuptools?

Because those other package managers depend, in fact, on setuptools, 
or at least pkg_resources...  which was why the original proposal was 
to just include pkg_resources in the first place.  :)


>I'd also like to avoid the specific name "easy_install" for any of
>this. That's a "brand name" (and a misleading one if you ask me, but
>that's politics again :-).

Ok, so if someone will propose a name and API for the thing, I'll 
implement it.  (Assuming the proposed API is sane and reasonably 
implementable, of course.)


From guido at python.org  Mon Mar 17 17:31:59 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 11:31:59 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317161305.22B183A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
Message-ID: <ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>

On Mon, Mar 17, 2008 at 11:12 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote:
>  >I don't think this should play games with scripts being overridden or
>  >whatever. If a bootstrap script is to be installed it should have a
>  >separate name. I'm not sure what the advantage is of a bootstrap
>  >script over "python -m bootstrap_module ..." though.
>
>  And -m also makes explicit:
>
>  1. that it's a Python-specific tool
>  2. which Python version it will apply to

Right!

>  >The PEP suggests that other package managers also benefit. How do they
>  >benefit if the bootstrap script installs setuptools?
>
>  Because those other package managers depend, in fact, on setuptools,
>  or at least pkg_resources...  which was why the original proposal was
>  to just include pkg_resources in the first place.  :)

On how much of pkg_resources do they depend? Or is that an
unanswerable question?

>  >I'd also like to avoid the specific name "easy_install" for any of
>  >this. That's a "brand name" (and a misleading one if you ask me, but
>  >that's politics again :-).
>
>  Ok, so if someone will propose a name and API for the thing, I'll
>  implement it.  (Assuming the proposed API is sane and reasonably
>  implementable, of course.)

Perhaps it can be called package_bootstrap? I don't know enough about
the required functionality to propose an API, but here goes anyway.

Would be reasonable for it to have a command line that allows you to
specify a package name, optionally a version string, and (very)
optionally a server to contact (specified as an URL?). It should
default to the highest non-experimental version that's applicable to
the current Python version (assuming there's an easy way to determine
this; I don't want it to try and download different versions until one
works). It should not handle any kind of dependency management or
package management administration.

It should be able to download a Python-only module or package and
install it into site-packages (or perhaps elsewhere if so directed via
another optional command line flag). It should support zip, tar and
tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the
zip/tar file using the zip or tar modules, and extract the
package/module into site-packages in such a way that it can be
imported directly without messing with sys.path. It should not mess
with .pth files, setup.py scripts, or eggs. If the contents of the
tar/zip file has a toplevel directory with version numbers in its name
(e.g. Django-0.96.1) it should skip that and just use the subdirectory
whose name is the "pure" package name (e.g. django).

Does this make sense? I'm happy to take push-back, this is just
something I cooked up off the top of my head based on my experience
with manually installing packages.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From p.f.moore at gmail.com  Mon Mar 17 17:35:24 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 17 Mar 2008 16:35:24 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317161305.22B183A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
Message-ID: <79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>

On 17/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> >The PEP suggests that other package managers also benefit. How do they
> >benefit if the bootstrap script installs setuptools?
>
> Because those other package managers depend, in fact, on setuptools,
> or at least pkg_resources...  which was why the original proposal was
> to just include pkg_resources in the first place.  :)

I'm puzzled. We seem to be talking about adding a module to the stdlib
whose basic function is to download and install setuptools? How is
this better than just including setuptools in the stdlib?

Personally, I have no problem per se with including setuptools in the
stdlib. Maybe that means I miss the subtle benefit of this approach...

I'm -1 on having a module which just installs setuptools.
I'm +0 on including pkg_resources (as described in PEP 365) in the stdlib.

I'm +lots on someone giving a clear explanation of the meaning and
interrelationship of the various terms involved in this discussion
(setuptools, easy_install, pkg_resources, eggs, "package managers" as
distinct from setuptools, etc etc) so that the discussion gets some
much-needed clarity :-(

I'm -1 on adding anything until PEP 365 is updated to match what is
being proposed, and then that amended PEP is submitted for discussion.

Paul.

From guido at python.org  Mon Mar 17 17:40:21 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 11:40:21 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
Message-ID: <ca471dc20803170940p41af33bbl8218e8e23479477c@mail.gmail.com>

On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 17/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
>  > >The PEP suggests that other package managers also benefit. How do they
>  > >benefit if the bootstrap script installs setuptools?
>  >
>  > Because those other package managers depend, in fact, on setuptools,
>  > or at least pkg_resources...  which was why the original proposal was
>  > to just include pkg_resources in the first place.  :)
>
>  I'm puzzled. We seem to be talking about adding a module to the stdlib
>  whose basic function is to download and install setuptools? How is
>  this better than just including setuptools in the stdlib?

I'm not in favor of this either.

>  Personally, I have no problem per se with including setuptools in the
>  stdlib. Maybe that means I miss the subtle benefit of this approach...
>
>  I'm -1 on having a module which just installs setuptools.
>  I'm +0 on including pkg_resources (as described in PEP 365) in the stdlib.
>
>  I'm +lots on someone giving a clear explanation of the meaning and
>  interrelationship of the various terms involved in this discussion
>  (setuptools, easy_install, pkg_resources, eggs, "package managers" as
>  distinct from setuptools, etc etc) so that the discussion gets some
>  much-needed clarity :-(

Right. But finding someone who can explain all this is apparently
hard. All the owners of package managers seem busy...

>  I'm -1 on adding anything until PEP 365 is updated to match what is
>  being proposed, and then that amended PEP is submitted for discussion.

Well, if we can reach consensus here first on what to put into PEP 365
first I'd be fine with updating the PEP as an afterthought, especially
of the consensus also comes with working code (hopefully less than
2500 lines :-).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Mon Mar 17 17:48:09 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Mar 2008 11:48:09 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	
	<20080317000630.735823A40B0@sparrow.telecommunity.com>	
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>	
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>	
	<47DE8405.2030201@v.loewis.de>	
	<20080317151637.532D23A409D@sparrow.telecommunity.com>	
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>	
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
Message-ID: <47DEA0C9.1020300@v.loewis.de>

> I'm puzzled. We seem to be talking about adding a module to the stdlib
> whose basic function is to download and install setuptools? How is
> this better than just including setuptools in the stdlib?

I can do a review of such a module in an hour. To review setuptools
(which I would have to do if it were to be included), I would likely
need a week or so, full time.

> Personally, I have no problem per se with including setuptools in the
> stdlib. Maybe that means I miss the subtle benefit of this approach...

Did you review setuptools and can vouch that it is written cleanly,
follows the coding style, has correct documentation, and so on?

Regards,
Martin

From stefan_ml at behnel.de  Mon Mar 17 17:55:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 17 Mar 2008 17:55:17 +0100
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<20080317000630.735823A40B0@sparrow.telecommunity.com>	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>	<47DE8405.2030201@v.loewis.de>	<20080317151637.532D23A409D@sparrow.telecommunity.com>	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
Message-ID: <47DEA275.4080306@behnel.de>

Guido van Rossum wrote:
> It should be able to download a Python-only module or package and
> install it into site-packages (or perhaps elsewhere if so directed via
> another optional command line flag). It should support zip, tar and
> tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the
> zip/tar file using the zip or tar modules, and extract the
> package/module into site-packages in such a way that it can be
> imported directly without messing with sys.path. It should not mess
> with .pth files, setup.py scripts, or eggs.

Do you mean "existing eggs" or does that include the (potential .egg) package
that is being installed? If I understood correctly, this bootstrap module
currently supports installing eggs (although I'm not sure how they are
supposed to work without the current way of keeping a .pth file).

Is it *wanted* that eggs are being supported by core Python?

Stefan


From p.f.moore at gmail.com  Mon Mar 17 18:12:40 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 17 Mar 2008 17:12:40 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47DEA0C9.1020300@v.loewis.de>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
	<47DEA0C9.1020300@v.loewis.de>
Message-ID: <79990c6b0803171012uaae8e9cmf6e0f4f189fa7ade@mail.gmail.com>

On 17/03/2008, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > Personally, I have no problem per se with including setuptools in the
> > stdlib. Maybe that means I miss the subtle benefit of this approach...
>
> Did you review setuptools and can vouch that it is written cleanly,
> follows the coding style, has correct documentation, and so on?

No, I concede that when I say "I have no problem" with including
setuptools, I'm assuming that someone does that review - and there's
no reason to assume that anyone will do that (that's why I say "I have
no problem with" rather than "I want").

But I don't see any practical difference with including setuptools and
including a module that installs setuptools. Would you be happy with
the standard library including a module whose sole function was to
install a package that you weren't happy to include directly in the
standard library? I wouldn't, personally.

Paul

From guido at python.org  Mon Mar 17 18:17:49 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 12:17:49 -0500
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <47DEA275.4080306@behnel.de>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
Message-ID: <ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>

On Mon, Mar 17, 2008 at 11:55 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Guido van Rossum wrote:
>  > It should be able to download a Python-only module or package and
>  > install it into site-packages (or perhaps elsewhere if so directed via
>  > another optional command line flag). It should support zip, tar and
>  > tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the
>  > zip/tar file using the zip or tar modules, and extract the
>  > package/module into site-packages in such a way that it can be
>  > imported directly without messing with sys.path. It should not mess
>  > with .pth files, setup.py scripts, or eggs.
>
>  Do you mean "existing eggs" or does that include the (potential .egg) package
>  that is being installed?

The latter.

> If I understood correctly, this bootstrap module
>  currently supports installing eggs (although I'm not sure how they are
>  supposed to work without the current way of keeping a .pth file).

My proposal is that the boostrap module only be able to install
non-egg archives.

>  Is it *wanted* that eggs are being supported by core Python?

No. There will be no egg support in the standard library.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Mar 17 18:18:41 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 12:18:41 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803171012uaae8e9cmf6e0f4f189fa7ade@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
	<47DEA0C9.1020300@v.loewis.de>
	<79990c6b0803171012uaae8e9cmf6e0f4f189fa7ade@mail.gmail.com>
Message-ID: <ca471dc20803171018w1c0c3607l3dfa8806d75d3b6f@mail.gmail.com>

On Mon, Mar 17, 2008 at 12:12 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>  But I don't see any practical difference with including setuptools and
>  including a module that installs setuptools. Would you be happy with
>  the standard library including a module whose sole function was to
>  install a package that you weren't happy to include directly in the
>  standard library? I wouldn't, personally.

Actually it's quite different -- the user or administrator is in
control and they can choose not to install setuptools.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Mon Mar 17 18:45:23 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 13:45:23 -0400
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
Message-ID: <20080317174542.89A463A409D@sparrow.telecommunity.com>

At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:
>There will be no egg support in the standard library.

Are there any qualifications on that statement, or is this in the 
same category as "from __future__ import braces"?


From p.f.moore at gmail.com  Mon Mar 17 18:48:45 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 17 Mar 2008 17:48:45 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
Message-ID: <79990c6b0803171048t140a1dbfvcb1fa4dc005e5cb5@mail.gmail.com>

On 17/03/2008, Guido van Rossum <guido at python.org> wrote:
> On Mon, Mar 17, 2008 at 11:55 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> >  Is it *wanted* that eggs are being supported by core Python?
>
> No. There will be no egg support in the standard library.

This bothers me somewhat. At a certain level, all that egg files are
is zip files with a different extension - and the zipimport module and
PEP 302 certainly do support them. There is a lot more conceptual
baggage associated with the egg "brand name", but unless that extra is
clearly specified, a statement like "there will be no egg support"
doesn't mean much.

My view on PEP 365 is that it offers a standard way of doing

    from pkg_resources import resource_string
    foo_config = resource_string(__name__, 'foo.conf')

which is basically a version of

    foo_config = open(os.path.join(os.path.dirname(__file__),'foo.conf').read()

which also supports PEP 302 importers such as zipimport. This has
nothing to do with eggs, and everything to do with solidifying the
support for cleanly handling PEP 302 importers.

For me, that would be far more useful that a package
download&installer (as I'm on Windows, I tend to use bdist_wininst
installers, and a bootstrap module which gave no uninstall capability
would suck for me).

The sticking point here, is deciding what parts of pkg_resources are
OK, and which constitute "egg support". It may not be possible to come
to a clear understanding of this.

But I remain -1 on any module that just does download and install,
with no uninstall capability. I don't like *eggs* for precisely this
reason!

Paul.

From martin at v.loewis.de  Mon Mar 17 18:49:25 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 17 Mar 2008 12:49:25 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803171012uaae8e9cmf6e0f4f189fa7ade@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	
	<20080317000630.735823A40B0@sparrow.telecommunity.com>	
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>	
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>	
	<47DE8405.2030201@v.loewis.de>	
	<20080317151637.532D23A409D@sparrow.telecommunity.com>	
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>	
	<20080317161305.22B183A409D@sparrow.telecommunity.com>	
	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>	
	<47DEA0C9.1020300@v.loewis.de>
	<79990c6b0803171012uaae8e9cmf6e0f4f189fa7ade@mail.gmail.com>
Message-ID: <47DEAF25.20101@v.loewis.de>

> But I don't see any practical difference with including setuptools and
> including a module that installs setuptools. Would you be happy with
> the standard library including a module whose sole function was to
> install a package that you weren't happy to include directly in the
> standard library? I wouldn't, personally.

If the actual code is not in Python, I'm not responsible for it. I
don't have to deal with bug reports, and I don't have to help users
with that feature. As users want to get the functionality anyway,
and out-of-the-box, I consider it a useful compromise.

Regards,
Martin




From guido at python.org  Mon Mar 17 18:59:46 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 12:59:46 -0500
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <20080317174542.89A463A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
Message-ID: <ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>

On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:
>  >There will be no egg support in the standard library.
>
>  Are there any qualifications on that statement, or is this in the
>  same category as "from __future__ import braces"?

IIUC eggs are a method of package management that includes support for
dependencies, multiple versions, and C extensions in zip files, as
well as conventions for naming these and for encoding metadata (e.g.
how to find out the version or the dependencies). This whole set of
conventions is IMO too much to include into the stdlib ATM -- if only
because it has proved controversial in the past. Maybe a few years
from now it will be no longer controversial and then my objections
will disappear.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From pje at telecommunity.com  Mon Mar 17 19:45:40 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 14:45:40 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
Message-ID: <20080317184553.913413A409D@sparrow.telecommunity.com>

At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote:
>On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby 
><pje at telecommunity.com> wrote:
> > At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:
> >  >There will be no egg support in the standard library.
> >
> >  Are there any qualifications on that statement, or is this in the
> >  same category as "from __future__ import braces"?
>
>IIUC eggs are a method of package management that includes support for
>dependencies, multiple versions, and C extensions in zip files, as
>well as conventions for naming these and for encoding metadata (e.g.
>how to find out the version or the dependencies). This whole set of
>conventions is IMO too much to include into the stdlib ATM -- if only
>because it has proved controversial in the past. Maybe a few years
>from now it will be no longer controversial and then my objections
>will disappear.

So, does this mean that the bootstrap tool must not use eggs?  That 
seems a little bit odd, in that setuptools will at least need its 
.egg-info directory to get installed, and all of the people who'll be 
using this initially will be using it precisely in order to have 
support for eggs...

So, it might be simpler all around to just clear up the 
"controversy".  To the best of my recollection, only MAL and MvL have 
ever objected on Python-Dev to the idea of supporting eggs.

Note: I'm specifically segregating "egg support" from the topic of 
including setuptools or easy_install in the stdlib directly.  There 
are many legitimate reservations and open questions about the latter, 
including availability of volunteer support, choice of defaults, 
whether to replace distutils with setuptools, etc. etc.  I recognize 
and respect the validity of those issues, which is precisely why I 
withdrew setuptools from inclusion in Python 2.5.

However, regarding support for eggs, my understanding is that there 
were only two objections to eggs -- even at the time of the 2.5 
setuptools discussions.  And even though MvL objects to the idea of 
eggs in *principle*, I didn't read his recent posts as objecting to 
having the bootstrap tool download and install eggs in 
*practice*.  (Although I hope he will clarify that stance one way or 
the other.)

That leaves MAL, whose objections to PEP 365 centered on the API (he 
said he was "+1 on the concepts being added to the stdlib, -1 on 
adding the module in its current state").  Among other concerns, he 
wanted pkg_resources to be split into pkgutil and a new "egglib" 
module.  I don't have a problem with this in principle, if there were 
a pkg_resources module that reconstituted the merged API.  (But there 
are some practical problems with that approach, such as trying to 
split namespace package support between two theoretically-unrelated modules.)

I would guess, however, that MAL's issues with the pkg_resources API 
would not apply to a bootstrap module whose sole purpose was to 
download eggs and put them on sys.path.  Or, perhaps he would object 
*more*, I don't know.  We could certainly ask him, though.  :)

So, was there anyone else you were counting towards 
"controversy"?  The only other person I recall objecting to 
setuptools in any way on Python-Dev was effbot, and IIUC his 
objections were practical/administrative re: supporting easy_install 
and setuptools, not to the idea of .egg support in general.

In summary, I think the controversy on Python-Dev regarding .egg 
support has actually been over for some time now.


From guido at python.org  Mon Mar 17 19:59:38 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 17 Mar 2008 13:59:38 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317184553.913413A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
Message-ID: <ca471dc20803171159j69762b21w8cf6e4194bb3608f@mail.gmail.com>

On Mon, Mar 17, 2008 at 1:45 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote:
>  >On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby
>  ><pje at telecommunity.com> wrote:
>  > > At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:
>  > >  >There will be no egg support in the standard library.
>  > >
>  > >  Are there any qualifications on that statement, or is this in the
>  > >  same category as "from __future__ import braces"?
>  >
>  >IIUC eggs are a method of package management that includes support for
>  >dependencies, multiple versions, and C extensions in zip files, as
>  >well as conventions for naming these and for encoding metadata (e.g.
>  >how to find out the version or the dependencies). This whole set of
>  >conventions is IMO too much to include into the stdlib ATM -- if only
>  >because it has proved controversial in the past. Maybe a few years
>  >from now it will be no longer controversial and then my objections
>  >will disappear.
>
>  So, does this mean that the bootstrap tool must not use eggs?

Correct.

>  That
>  seems a little bit odd, in that setuptools will at least need its
>  .egg-info directory to get installed, and all of the people who'll be
>  using this initially will be using it precisely in order to have
>  support for eggs...

I'm okay if setuptools, once it's been installed, runs some setup code
that creates the .egg-info directory and whatever else. This means I'm
also okay with the bootstrap module finding and invoking that setup
code. But I'm *not* okay with building any kind of egg management into
the bootstrap module. The bootstrap module must be be neutral w.r.t.
the package management style.

>  So, it might be simpler all around to just clear up the
>  "controversy".  To the best of my recollection, only MAL and MvL have
>  ever objected on Python-Dev to the idea of supporting eggs.

You can add my name to the list. I've heard plenty of people speak
highly of eggs, but I've *also* heard from plenty of people (besides
MAL and MvL) who have serious difficulties with the concept of eggs. I
have certainly personally encountered plenty of situations where I
wasn't able to complete an egg-based install because some dependency
was broken (e.g. not available for the Python version I was using).

>  Note: I'm specifically segregating "egg support" from the topic of
>  including setuptools or easy_install in the stdlib directly.  There
>  are many legitimate reservations and open questions about the latter,
>  including availability of volunteer support, choice of defaults,
>  whether to replace distutils with setuptools, etc. etc.  I recognize
>  and respect the validity of those issues, which is precisely why I
>  withdrew setuptools from inclusion in Python 2.5.
>
>  However, regarding support for eggs, my understanding is that there
>  were only two objections to eggs -- even at the time of the 2.5
>  setuptools discussions.  And even though MvL objects to the idea of
>  eggs in *principle*, I didn't read his recent posts as objecting to
>  having the bootstrap tool download and install eggs in
>  *practice*.  (Although I hope he will clarify that stance one way or
>  the other.)

You can do it in two stages. The bootstrap can download and install
egg support, even though the bootstrap itself knows nothing about
eggs. Then another bootstrap can download and install eggs.

>  That leaves MAL, whose objections to PEP 365 centered on the API (he
>  said he was "+1 on the concepts being added to the stdlib, -1 on
>  adding the module in its current state").  Among other concerns, he
>  wanted pkg_resources to be split into pkgutil and a new "egglib"
>  module.  I don't have a problem with this in principle, if there were
>  a pkg_resources module that reconstituted the merged API.  (But there
>  are some practical problems with that approach, such as trying to
>  split namespace package support between two theoretically-unrelated modules.)

Well, you've heard my position now.

>  I would guess, however, that MAL's issues with the pkg_resources API
>  would not apply to a bootstrap module whose sole purpose was to
>  download eggs and put them on sys.path.  Or, perhaps he would object
>  *more*, I don't know.  We could certainly ask him, though.  :)

No need. I object to this myself.

>  So, was there anyone else you were counting towards
>  "controversy"?  The only other person I recall objecting to
>  setuptools in any way on Python-Dev was effbot, and IIUC his
>  objections were practical/administrative re: supporting easy_install
>  and setuptools, not to the idea of .egg support in general.
>
>  In summary, I think the controversy on Python-Dev regarding .egg
>  support has actually been over for some time now.

Not really.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tseaver at palladion.com  Mon Mar 17 20:05:15 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 17 Mar 2008 15:05:15 -0400
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<20080317000630.735823A40B0@sparrow.telecommunity.com>	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>	<47DE8405.2030201@v.loewis.de>	<20080317151637.532D23A409D@sparrow.telecommunity.com>	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
Message-ID: <47DEC0EB.3000404@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guido van Rossum wrote:
> On Mon, Mar 17, 2008 at 11:12 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
>> At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote:
>>  >I don't think this should play games with scripts being overridden or
>>  >whatever. If a bootstrap script is to be installed it should have a
>>  >separate name. I'm not sure what the advantage is of a bootstrap
>>  >script over "python -m bootstrap_module ..." though.
>>
>>  And -m also makes explicit:
>>
>>  1. that it's a Python-specific tool
>>  2. which Python version it will apply to
> 
> Right!
> 
>>  >The PEP suggests that other package managers also benefit. How do they
>>  >benefit if the bootstrap script installs setuptools?
>>
>>  Because those other package managers depend, in fact, on setuptools,
>>  or at least pkg_resources...  which was why the original proposal was
>>  to just include pkg_resources in the first place.  :)
> 
> On how much of pkg_resources do they depend? Or is that an
> unanswerable question?
> 
>>  >I'd also like to avoid the specific name "easy_install" for any of
>>  >this. That's a "brand name" (and a misleading one if you ask me, but
>>  >that's politics again :-).
>>
>>  Ok, so if someone will propose a name and API for the thing, I'll
>>  implement it.  (Assuming the proposed API is sane and reasonably
>>  implementable, of course.)
> 
> Perhaps it can be called package_bootstrap? I don't know enough about
> the required functionality to propose an API, but here goes anyway.
> 
> Would be reasonable for it to have a command line that allows you to
> specify a package name, optionally a version string, and (very)
> optionally a server to contact (specified as an URL?). It should
> default to the highest non-experimental version that's applicable to
> the current Python version (assuming there's an easy way to determine
> this; I don't want it to try and download different versions until one
> works). It should not handle any kind of dependency management or
> package management administration.
> 
> It should be able to download a Python-only module or package and
> install it into site-packages (or perhaps elsewhere if so directed via
> another optional command line flag). It should support zip, tar and
> tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the
> zip/tar file using the zip or tar modules, and extract the
> package/module into site-packages in such a way that it can be
> imported directly without messing with sys.path. It should not mess
> with .pth files, setup.py scripts, or eggs. If the contents of the
> tar/zip file has a toplevel directory with version numbers in its name
> (e.g. Django-0.96.1) it should skip that and just use the subdirectory
> whose name is the "pure" package name (e.g. django).
>
> Does this make sense? I'm happy to take push-back, this is just
> something I cooked up off the top of my head based on my experience
> with manually installing packages.
>

I would prefer to see it just:

 - Find a source distribution (the variants you specified) on the
   given server which matches the supplied version string, using the
   same semantics as the current 'pkg_resources' constraints.

 - Unpack the source distribution to a temp directory.

 - cd into that directory and use sys.executable to invoke
   'setup.py install'.  Any extra command-line arguments beyond those
   used to find the source distribution would be passed on to the
   'install' command, which would allow alternate locations, etc.

That makes the installation as much like a manual one as possible, and
means that existing pacakges will be installable whether or not they
know about setuptools, etc.



- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH3sDr+gerLs4ltQ4RAhjWAKCbFP87mJN4UnVt0pzDs81JovVpoACdGI7A
tohpRJnXah0/QnyQeYiqoYY=
=9Cif
-----END PGP SIGNATURE-----


From pje at telecommunity.com  Mon Mar 17 20:36:49 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 15:36:49 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803171159j69762b21w8cf6e4194bb3608f@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<ca471dc20803171159j69762b21w8cf6e4194bb3608f@mail.gmail.com>
Message-ID: <20080317194030.D6BAF3A409D@sparrow.telecommunity.com>

At 01:59 PM 3/17/2008 -0500, Guido van Rossum wrote:
>I have certainly personally encountered plenty of situations where I
>wasn't able to complete an egg-based install because some dependency
>was broken (e.g. not available for the Python version I was using).

That's odd -- setuptools-based installs should be able to find and 
install packages from source.  I have noticed a recent phenomenon 
where new developers upload *only* an egg to PyPI, without the 
source, but that's usually short-lived until someone points it out to 
them.  Do you happen to know what packages you had this problem with?


>I'm okay if setuptools, once it's been installed, runs some setup code
>that creates the .egg-info directory and whatever else. This means I'm
>also okay with the bootstrap module finding and invoking that setup
>code. But I'm *not* okay with building any kind of egg management into
>the bootstrap module. The bootstrap module must be be neutral w.r.t.
>the package management style.

Ok, well then we'll have to invent a new kind of binary package, 
whose name isn't 'egg'.  Supporting distutils source packages is 
almost certainly a non-starter, if you want to avoid bringing the 
rest of setuptools into play.

The only way to correctly determine what a source package contains is 
to run its setup script...  and running unboxed setup scripts isn't 
safe because there are people who hardcode paths (or more precisely, 
use bad ways of computing them) in their setup scripts.

I'm not saying the tool needs to guard against *malicious* scripts, 
just badly-written ones.  (Setuptools does this with its "sandboxing" 
module, when running source packages' setup scripts.)

So, if source is out, then some binary format is needed, which means 
defining the conventions for said format...  i.e. "eggs lite" or "egg 
substitutes".  :)


> >  So, it might be simpler all around to just clear up the
> >  "controversy".  To the best of my recollection, only MAL and MvL have
> >  ever objected on Python-Dev to the idea of supporting eggs.
>
>You can add my name to the list. I've heard plenty of people speak
>highly of eggs, but I've *also* heard from plenty of people (besides
>MAL and MvL) who have serious difficulties with the concept of eggs.

I did say "on Python-Dev", and you implied that it was not 
controversial with you, except for the maintenance-related 
concerns.  I'm not fighting about this, but I would rather you were 
straight-up with your objections rather than deferring it to a 
controversy that "might go away in a few years".  That way, I could 
at least attempt to do something about the concerns.  OTOH, if your 
objections were non-specific and likely to stay that way, then I 
could have at least not wasted your time with any of this.




From jeff at taupro.com  Mon Mar 17 20:44:07 2008
From: jeff at taupro.com (Jeff Rush)
Date: Mon, 17 Mar 2008 14:44:07 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the
	pkg_resources	module)
In-Reply-To: <ca471dc20803170940p41af33bbl8218e8e23479477c@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<20080317000630.735823A40B0@sparrow.telecommunity.com>	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>	<47DE8405.2030201@v.loewis.de>	<20080317151637.532D23A409D@sparrow.telecommunity.com>	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>	<20080317161305.22B183A409D@sparrow.telecommunity.com>	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
	<ca471dc20803170940p41af33bbl8218e8e23479477c@mail.gmail.com>
Message-ID: <47DECA07.6000600@taupro.com>

Guido van Rossum wrote:
> On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>>  I'm +lots on someone giving a clear explanation of the meaning and
>>  interrelationship of the various terms involved in this discussion
>>  (setuptools, easy_install, pkg_resources, eggs, "package managers" as
>>  distinct from setuptools, etc etc) so that the discussion gets some
>>  much-needed clarity :-(
> 
> Right. But finding someone who can explain all this is apparently
> hard. All the owners of package managers seem busy...

In preparing for my PyCon 2008 tutorial on eggs and buildout, I spent three 
full-time weeks carefully going over sources for distutils, setuptools and 
buildout to discover those aspects not documented.  I can explain how they 
work, although I'm not sure this is the correct forum.  I'd like to first 
offer my slides from my tutorial, 150 of them with detailed handout notes on 
many of them.

   http://wiki.python.org/moin/buildout/pycon2008_tutorial

I'm happy to answer questions after that.  I'm in the Hanada B room for OLPC 
at PyCon and on IRC #pycon.

-Jeff

From pje at telecommunity.com  Mon Mar 17 21:09:23 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 16:09:23 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the
 pkg_resources	module)
In-Reply-To: <47DECA07.6000600@taupro.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317000630.735823A40B0@sparrow.telecommunity.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<79990c6b0803170935v59129ac8k3b0ff4ece23fe591@mail.gmail.com>
	<ca471dc20803170940p41af33bbl8218e8e23479477c@mail.gmail.com>
	<47DECA07.6000600@taupro.com>
Message-ID: <20080317200941.372403A409D@sparrow.telecommunity.com>

At 02:44 PM 3/17/2008 -0500, Jeff Rush wrote:
>Guido van Rossum wrote:
> > On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >>
> >>  I'm +lots on someone giving a clear explanation of the meaning and
> >>  interrelationship of the various terms involved in this discussion
> >>  (setuptools, easy_install, pkg_resources, eggs, "package managers" as
> >>  distinct from setuptools, etc etc) so that the discussion gets some
> >>  much-needed clarity :-(
> >
> > Right. But finding someone who can explain all this is apparently
> > hard. All the owners of package managers seem busy...
>
>In preparing for my PyCon 2008 tutorial on eggs and buildout, I spent three
>full-time weeks carefully going over sources for distutils, setuptools and
>buildout to discover those aspects not documented.  I can explain how they
>work, although I'm not sure this is the correct forum.  I'd like to first
>offer my slides from my tutorial, 150 of them with detailed handout notes on
>many of them.
>
>    http://wiki.python.org/moin/buildout/pycon2008_tutorial

Wow.   I am skimming over the 44-page one on setuptools, and that is 
definitely the most comprehensive doc anyone has produced on it, 
aside from the official docs.  Thank you!


From p.f.moore at gmail.com  Mon Mar 17 21:19:11 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 17 Mar 2008 20:19:11 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080317184553.913413A409D@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
Message-ID: <79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>

On 17/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
>  That leaves MAL, whose objections to PEP 365 centered on the API (he
>  said he was "+1 on the concepts being added to the stdlib, -1 on
>  adding the module in its current state").  Among other concerns, he
>  wanted pkg_resources to be split into pkgutil and a new "egglib"
>  module.  I don't have a problem with this in principle, if there were
>  a pkg_resources module that reconstituted the merged API.  (But there
>  are some practical problems with that approach, such as trying to
>  split namespace package support between two theoretically-unrelated modules.)

I would personally like to see such a separation. As one of the
authors of PEP 302 (well, the documentation - Just did all of the
implementation) I have an interest in strengthening the standard
library's support for transparent use of PEP 302 importers. To that
end, I would like to see such functions as
pkg_resources.resource_string() standardised.

That covers the pkgutil aspect of pkg_resources.

The "egglib" side of things is where the controversy lies, IMHO. I
have a (somewhat unfocussed) dislike of eggs, largely because of the
lack of a package management tool to handle them. I can live with them
or without them, and it doesn't bother me if others use them, but the
thing that really, really bothers me is that the controversy (and yes,
there is such) over eggs is hampering the adoption of the pkgutil side
of pkg_resources.

So from me, there's a strong +1 for the split of pkg_resources into
additions to the existing pkgutil module, and an independent egglib.
You say there are practical problems to doing this. OK, but could we
not have a discussion on how to resolve those issues as they come up?
Could the split not be initially into 3 parts - pkgutil (eg,
resource_string), egglib, and "difficult"? Then there would be
something concrete to discuss and resolve.

Paul.

From zooko at zooko.com  Mon Mar 17 22:19:31 2008
From: zooko at zooko.com (zooko)
Date: Mon, 17 Mar 2008 15:19:31 -0600
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <47DEC0EB.3000404@palladion.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<20080317000630.735823A40B0@sparrow.telecommunity.com>	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>	<47DE8405.2030201@v.loewis.de>	<20080317151637.532D23A409D@sparrow.telecommunity.com>	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEC0EB.3000404@palladion.com>
Message-ID: <440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com>

Folks:

(By the way, it was a pleasure to meet many of you in Real Life for  
the first time at Pycon.)

Here is what I want:

1.  The standard Python build tools by default produce machine- 
parseable package metadata, which can include package dependency  
information with reasonably well-defined semantics.

Oh good!  I already have this, since distutils in Python >= 2.5  
produces .egg-info metadata in an easy-to-parse format.

2.  This machine-parseable metadata is widely supported and  
understood by the Python community.

In retrospect, it's too bad that it isn't named ".pkg-info" instead  
of ".egg-info", in order to avoid the fraught politics around the  
concept of "eggs".  A concrete example of such a misunderstanding is  
the sad fact that many Linux distributions were in the habit of  
deleting this information from their Python packages, perhaps because  
they were under the impression that it was obviated by their  
packaging tools.  The major distributions have all stopped doing this  
now.

Unifying the created-by-default PKG-INFO files and the created-by- 
default .egg-info directories would be nice, too.

3.  The standard Python library includes a tool to find and parse  
this metadata, so that I can write programmatic tests of my  
dependencies, like this:

http://allmydata.org/trac/tahoe/browser/_auto_deps.py?rev=2062

This is one of the improvements that I was anticipating from  
pkg_resources going into the standard library.

4.  The standard Python library includes a tool to find and read  
resources (other than Python modules) that came bundled in a Python  
package.

Consider, for example, this snippets of code in Nevow:

http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786#L10
http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786
http://divmod.org/trac/browser/trunk/Nevow/setup_egg.py?rev=2406

When Nevow uses pkg_resources to import its files such as  
"default.css", then it is able to find at runtime, even if is being  
imported from a py2exe or py2app zip, or on other platforms where its  
homegrown setup script and homegrown "find my file" function fail.   
So using pkg_resources (and setuptools to install it) makes  
"test_nevow" pass on all of the allmydata.org buildslaves:

http://allmydata.org/buildbot/waterfall?show_events=false


Regards,

Zooko


From jeff at taupro.com  Mon Mar 17 23:10:51 2008
From: jeff at taupro.com (Jeff Rush)
Date: Mon, 17 Mar 2008 17:10:51 -0500
Subject: [Distutils] Capsule Summary of Some Packaging/Deployment Technology
	Concerns
Message-ID: <47DEEC6B.5040602@taupro.com>

I was in a Packaging BoF yesterday and, although not very relevant to the 
packager bootstrap thread, Guido has asked me to post some of the concerns.

The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu 
and such.  Everyone had strong expressions of frustration with the status quo 
and most had tried to resolve their issues but had their patches rejected.  I 
am not taking either side and whether those rejections were justified I cannot 
say, but the general feeling of their concerns intentionally not being 
addressed isn't healthy.  Several had abandoned setuptools, deeming it a 
failed solution and others called for a fork.

To start, I am not a leader of the group nor do I claim I accurately captured 
and expressed all their concerns.  I apologize to those in the BoF for any 
misrepresentations.

1. Many felt the existing dependency resolver was not correct.  They wanted a
    full tree traversal resulting in an intersection of all restrictions,
    instead of a first-acceptable-solution approach taking now, which can
    result in top-level dependencies not being enforced upon lower levels.  The
    latter is faster however.  One solution would be to make the resolver
    pluggable.

2. People want a solution for the handling of documentation.  The distutils
    module has had commented out sections related to this for several years.

3. A more flexible internal handing of the different types of files is needed.
    Currently the code, data, lib, etc. files are aggregated at build time and
    people would like them to be kept separate until install/packaging time.

    They also want greater flexibility in the kinds of files identified for
    packaging.  There is currently a single plugin entrypoint for file_finding,
    so people have resorted to abusing the setuptools function find_packages()
    again and again with different include/exclude args.  A solution is to
    expand the set of entrypoints into finer grained categories.  They also
    want a way to expand the set of categories rather than a fixed set, which
    can be easily done with entrypoint groups and names.

    People also want a greater variety of file_finders to be included with
    setuptools.  Instead of just CVS and SVN, they want it to comprehend
    Mercurial, Bazaar, Git and so forth.

4. They want an uninstall setuptools command.  Adding one to remove a specific
    egg isn't difficult but correctly removing those dependencies that came in
    with that egg, without breaking later installs can be tricky.

    This is complicated because there isn't a single global package namespace
    to manage, when you factor in virtualenv and buildout sandboxes and
    per-user package areas.  This differs from how RPMs and .debs are viewed.

5. There was concern over the .pth mechanism used by setuptools re activation.
    First, there is a (perceived) performance issue with increasingly adding
    every ZIP file explicitly onto sys.path.  This may or may not be a red
    herring.

    The other is the use of a single .pth file to control the list of activated
    packages.  Those who produce distributions would prefer a magic directory
    into which links to distributions could be dropped, similar to the current
    best practices for Linux, with /etc/conf.d/, /etc/profile.d/,
    /etc/xinetd.d/ and so forth.

6. There is a need for more extensibility hooks.  People want places to plug
    in special handling.  For example:

    a) setuptools has a --record option to capture the list of files installed
       for use by subsequent packaging tools.  Some want that list to be
       available to a setuptools plugin.

    b) some want hooks for post-build/post-install actions, instead of the
       current approach of writing a custom build class that handles it all.

7. Many wanted to ability to install files anywhere in the install tree and
    not just under the Python package.  Under distutils this was possible but
    it was removed in setuptools for security reasons.  Custom code can still
    be written to do this explicitly but this is not popular.  Neither
    setuptools nor distutils has the ability to rename files at install time.

A fair question is whether it is the job of setuptools (or any Python 
packaging solution) to cover all these bases.  The risk of not doing the job 
is that some of those in attendance were rolling their own solutions which do 
not play well with packages installed using other means, not seeing them. 
Distutils has intentionally tried to -not- be a general replacement packaging 
solution, with its support of the "bdist" command for various 
platform-specific distribution formats.  We should continue not trying to 
replace platform-specific packaging technologies but perhaps improve our 
control of their creation.

As mentioned, some of these concerns can be resolved by adding 
customization-pressure-release entrypoints to setuptools, and some can be 
handled with much better documentation of use cases and what to do.  And some 
of it is confusion over packaging libraries versus applications, where 
setuptools focuses on the former and zc.buildout focuses on the latter.  But 
buildout is very young, maintains isolation from the system Python and was not 
known to many of the packaging BoF attendees.

Some of this may seem down on eggs, but I think they are really cool and would 
like to see them adopted as the standard for packaging Python software.  There 
are rough spots on setuptools and buildout that would benefit from opening up 
the process and bringing in more developers, and communicating what they are 
and more importantly, what they are not.  I believe the lack of a coherent 
packaging and deployment story for Python is hurting its uptake in many 
sectors and would like to work with others to strengthen this area of Python.

-Jeff


From pje at telecommunity.com  Tue Mar 18 01:37:30 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 17 Mar 2008 20:37:30 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47DEEC6B.5040602@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
Message-ID: <20080318004718.56E173A40FF@sparrow.telecommunity.com>

At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote:
>I was in a Packaging BoF yesterday and, although not very relevant to the
>packager bootstrap thread, Guido has asked me to post some of the concerns.
>
>The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu
>and such.  Everyone had strong expressions of frustration with the status quo
>and most had tried to resolve their issues but had their patches rejected.  I
>am not taking either side and whether those rejections were 
>justified I cannot
>say, but the general feeling of their concerns intentionally not being
>addressed isn't healthy.  Several had abandoned setuptools, deeming it a
>failed solution and others called for a fork.
>
>To start, I am not a leader of the group nor do I claim I accurately captured
>and expressed all their concerns.  I apologize to those in the BoF for any
>misrepresentations.

I'm actually happy to hear that there's this much energy available -- 
hopefully some of it can be harnessed towards positive solutions.

When I began developing setuptools, I often asked for the input of 
packagers, developers, etc., through the distutils-sig...  and was 
met with overwhelming silence.  So the fact that there is now a group 
of people who are ready to work for some solutions seems like a 
positive change, to me.

It's hard to make design decisions regarding itches you don't 
personally have, and which other people won't help 
scratch.  Unfortunately, a lot of the proposals from packaging system 
people have been of the form of, "fix this for us by breaking things 
for other people".  Not all of them, though.  Many have been very 
helpful, contributing troubleshooting help and good patches.

That some of those good patches took nearly a year to get into 
setuptools (some from Fedora just got into 0.6c8 that were sent to me 
almost a year ago) is because I'm the only person reviewing 
setuptools patches, and I've spent only a few days in the last year 
doing focused development work on setuptools (as opposed to answering 
questions about it  on the SIG).

It's never a good thing when people's patches sit around, regardless 
of where they come from.  But that's not the same thing as 
*rejecting* the patches.


>1. Many felt the existing dependency resolver was not correct.  They wanted a
>     full tree traversal resulting in an intersection of all restrictions,
>     instead of a first-acceptable-solution approach taking now, which can
>     result in top-level dependencies not being enforced upon lower 
> levels.  The
>     latter is faster however.  One solution would be to make the resolver
>     pluggable.

Patches welcome, on both counts.  Personally, Bob and I originally 
wanted a full-tree intersection, too, but it turned out to be hairier 
to implement than it seems at first.  My guess is that none of the 
people who want it, have actually tried to implement it without a 
factorial or exponential O().  But that doesn't mean I'll be unhappy 
if somebody succeeds.  :)

Intuitively, it seems easy, just gather the requirements and 
intersect.  In practice, different versions of a package may have 
different dependencies, so the intersection is not nearly as simple 
as that.  We ended up just going for a depth-first version of the 
current algorithm (switched to breadth-first later, after field tests 
showed some problems with that), being greedy by testing 
latest-version-first, on the assumption that more recent versions 
would be likely to have the most-restrictive version requirements.

In other words, we attempt to achieve heuristically what's being 
proposed to do algorithmically.  And my guess is that whatever cases 
the heuristic is failing at, would probably not be helped by an 
algorithmic approach either.  But I would welcome some actual data, either way.

Again, though, patches are welcome.  :)  (Specifically, for the 
trunk; I don't see a resolver overhaul as being suitable for the 0.6 
stable branch.)


>2. People want a solution for the handling of documentation.  The distutils
>     module has had commented out sections related to this for several years.

As with so many other things, this gets tossed around the 
distutils-sig every now and then.  A couple of times I've thrown out 
some options for how this might be done, but then the conversation 
peters out around the time anybody would have to actually do some 
work on it.  (Me included, since I don't have an itch that needs 
scratching in this area.)

In particular, if somebody wants to come up with a metadata standard 
for including documentation in eggs, we've got a boatload of hooks by 
which it could be done.  Nothing's stopping anybody from proposing a 
standard and building a tool, here.  (e.g. using the setuptools 
command hook, .egg-info writer hook, etc.)


>3. A more flexible internal handing of the different types of files is needed.
>     Currently the code, data, lib, etc. files are aggregated at 
> build time and
>     people would like them to be kept separate until install/packaging time.

I don't know what this means, exactly.


>     They also want greater flexibility in the kinds of files identified for
>     packaging.  There is currently a single plugin entrypoint for 
> file_finding,
>     so people have resorted to abusing the setuptools function 
> find_packages()
>     again and again with different include/exclude args.  A solution is to
>     expand the set of entrypoints into finer grained categories.  They also
>     want a way to expand the set of categories rather than a fixed set, which
>     can be easily done with entrypoint groups and names.
>
>     People also want a greater variety of file_finders to be included with
>     setuptools.  Instead of just CVS and SVN, they want it to comprehend
>     Mercurial, Bazaar, Git and so forth.

Did you point them to the Cheeseshop?  There are plugins already 
available for all the systems you mentioned, plus Darcs and 
Monotone.  If you mean "included" as in "bundled", this doesn't make 
a whole lot of sense to me.  I'd think that if you're using 
setuptools as a developer (the only reason you need the file finders, 
since source distributions include a prebuilt manifest), you'd not 
have a problem saying "easy_install setuptools-git" or adding a 
"setup_requires='setuptools-git'" line to your setup.py.  (Although 
the latter would only be needed for *development*, not deployment.)

If you mean support for *installing* from these tools, I really 
wanted to add a pluggable download/retrieval mechanism for 
easy_install in 0.7, and would still love to see it happen.


>4. They want an uninstall setuptools command.  Adding one to remove a specific
>     egg isn't difficult but correctly removing those dependencies 
> that came in
>     with that egg, without breaking later installs can be tricky.

Patches, once again, are welcome.  :)  (Btw, "setup.py develop" 
supports uninstallation, although it doesn't do a blessed thing about 
dependencies.)

By the way, there are also third-party tools on the Cheeseshop that 
show egg dependency graphs (e.g. tl.eggdeps) or dump out information 
about installed eggs (e.g. "yolk").


>     This is complicated because there isn't a single global package namespace
>     to manage, when you factor in virtualenv and buildout sandboxes and
>     per-user package areas.  This differs from how RPMs and .debs are viewed.

Yep.  I really wanted, for 0.7, to study virtualenv and buildout and 
try to design a more general mechanism for managing things with the 
vaporware I've been calling "nest".  Unfortunately, I've lost both 
the will and the budget for working on that any time soon.


>5. There was concern over the .pth mechanism used by setuptools re activation.
>     First, there is a (perceived) performance issue with increasingly adding
>     every ZIP file explicitly onto sys.path.  This may or may not be a red
>     herring.

It is.  My tests a few years back, when MAL first brought this up on 
the distutils-sig, showed the startup cost to be positively 
miniscule, if actual zipfiles are used.  At the same time, I myself 
have come to the conclusion that if I had it to do over, I would use 
something more like the .egg-info egg style for general package 
installation, and added an installation manifest to it.  If I ever 
write "nest", it will use such, with the ability to also support .egg 
files and directories.

.egg files were created for extensible application platforms like 
Chandler and Zope and Trac and so on.  Plugins usually need 
libraries, though, so the rest got added on because it was useful, 
and then the whole thing escaped its niche like a foreign organism 
added to an ecosystem with no natural predators.  :)


>     The other is the use of a single .pth file to control the list 
> of activated
>     packages.  Those who produce distributions would prefer a magic directory
>     into which links to distributions could be dropped, similar to 
> the current
>     best practices for Linux, with /etc/conf.d/, /etc/profile.d/,
>     /etc/xinetd.d/ and so forth.

site-packages is that directory, and has been since long before 
setuptools.  Just drop uniquely-named .pth files there, and you're good to go.


>6. There is a need for more extensibility hooks.  People want places to plug
>     in special handling.  For example:
>
>     a) setuptools has a --record option to capture the list of 
> files installed
>        for use by subsequent packaging tools.  Some want that list to be
>        available to a setuptools plugin.
>
>     b) some want hooks for post-build/post-install actions, instead of the
>        current approach of writing a custom build class that handles it all.

Patches welcome!


>7. Many wanted to ability to install files anywhere in the install tree and
>     not just under the Python package.  Under distutils this was possible but
>     it was removed in setuptools for security reasons.

It wasn't security, it was manageability.  Egg-based installation 
means containment, (analagous to GNU stow) and therefore portability 
and disposability of plugins.  (Which again is what eggs were really 
developed for in the first place.)


>   Custom code can still
>     be written to do this explicitly but this is not popular.

No kidding.  :)  Current best practice is to include a script or 
module in the package that can install other files to a designated 
location.  Personally, though, I tend to view applications and 
libraries that target specific install locations to be overreaching 
their bounds, and stepping into sysadmin territory.  Give me the 
tools to install the data, don't just dump it somewhere on my system 
where *you* think it should go, in other words.

On the other hand, I've been puzzling over how to handle legitimate 
post-install features.  On Windows, both wx and pywin32 have a real 
need to do some actuall "install" operations.  Some is just copying 
files, but pywin32 also has to do some registry stuff.  I don't know 
how to allow just what's sensible, without opening up a huge can of 
worms, though.

Proposals welcome.


From guido at python.org  Tue Mar 18 06:31:55 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Mar 2008 00:31:55 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
Message-ID: <ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>

After reading all this, I really don't  believe that adding egg
support to the stdlib at this time is the right thing to do. I am
therefore rejecting the PEP.

I am hoping that someone will create a simpler bootstrap module that
is able to download a file of pure Python code and install it, perhaps
by running its setup.py, assuming that it only depends on distutils
(or other things previously installed). I will welcome such a module
into the stdlib. I'm not sure a PEP is even needed, though interested
parties are certainly welcome to write a PEP specifying the behavior
first. With 2.6 and 3.0 slated for release in September, there should
be enough time to get this done before then.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From terry at jon.es  Tue Mar 18 06:27:50 2008
From: terry at jon.es (Terry Jones)
Date: Tue, 18 Mar 2008 06:27:50 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
Message-ID: <18399.21206.481836.992152@jon.es>

[Apologies if this is in the mailing list archives. I looked but didn't
see anything, based on a search for run_setup]

I'm trying to programmatically install something built using distutils. I
found distutils.core.run_setup and can use it via

  >>> dist = run_setup('setup.py', ['-q', 'install'])

Is that the recommended way to do an install from inside Python (as opposed
to doing it on the command line)?

If so, how can I find where the thing(s) I installed now resides?  I saw
dist.packages but that just has top-level package names. I could __import__
these (and then use module.__file__), but that's not a good solution as it
may run code I don't want run. On my machine, I can see the packages have
been installed under the system's python2.5/site-packages directory. But
how can I determine that programmatically? I don't see anything useful on
the distutils.dist.Distribution instance I'm getting back from run_setup.

Thanks!

Terry

From ziade.tarek at gmail.com  Tue Mar 18 11:05:13 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 18 Mar 2008 11:05:13 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: <18399.21206.481836.992152@jon.es>
References: <18399.21206.481836.992152@jon.es>
Message-ID: <94bdd2610803180305o6424ead8y8a55e4d88321f399@mail.gmail.com>

On Tue, Mar 18, 2008 at 6:27 AM, Terry Jones <terry at jon.es> wrote:

> [Apologies if this is in the mailing list archives. I looked but didn't
> see anything, based on a search for run_setup]
>
> I'm trying to programmatically install something built using distutils. I
> found distutils.core.run_setup and can use it via
>
>  >>> dist = run_setup('setup.py', ['-q', 'install'])
>
> Is that the recommended way to do an install from inside Python (as
> opposed
> to doing it on the command line)?


I don't use this command because it does an execfile, which will lead to
errors
if the setup script uses some globals like __file__, iirc

You can import the module that contains the setup() call  from you code as
long as you
set the sys.argv values as you would do through the command line *before*
the import statement.

I would do something like this::

import sys
import os

def run_setup(script='setup.py', args=('install',)):
    base_name = os.path.basename(script)
    module_name = script.split('.')[0]
    folder_name = os.path.realpath(os.path.dirname(script))
    old = sys.argv
    old_location = os.curdir
    try:
        os.chdir(folder_name)
        sys.argv = [base_name] + list(args)
        __import__(module_name)
    finally:
        os.chdir(old_location)
        sys.argv = old

if __name__ == '__main__':
    if len(sys.argv) > 1:
        setup_py = sys.argv[1]
        run_setup(setup_py, sys.argv[2:])
    else:
        run_setup()


>
> If so, how can I find where the thing(s) I installed now resides?  I saw
> dist.packages but that just has top-level package names. I could
> __import__
> these (and then use module.__file__), but that's not a good solution as it
> may run code I don't want run. On my machine, I can see the packages have
> been installed under the system's python2.5/site-packages directory. But
> how can I determine that programmatically? I don't see anything useful on
> the distutils.dist.Distribution instance I'm getting back from run_setup.


I don't know about run_setup but you can use get_python_lib when you are
using
default locations on your setup() call. It will give you the default
location:

>>> from distutils.sysconfig import get_python_lib
>>> get_python_lib()
'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages'



>
> Thanks!
>
> Terry
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080318/097c24bf/attachment.htm 

From pje at telecommunity.com  Tue Mar 18 17:23:29 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 12:23:29 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
Message-ID: <20080318162331.C820D3A4074@sparrow.telecommunity.com>

At 12:31 AM 3/18/2008 -0500, Guido van Rossum wrote:
>I am hoping that someone will create a simpler bootstrap module that
>is able to download a file of pure Python code and install it, perhaps
>by running its setup.py, assuming that it only depends on distutils
>(or other things previously installed). I will welcome such a module
>into the stdlib. I'm not sure a PEP is even needed, though interested
>parties are certainly welcome to write a PEP specifying the behavior
>first. With 2.6 and 3.0 slated for release in September, there should
>be enough time to get this done before then.

Unfortunately, as I've already tried to explain, "download a file ... 
and install it" is not a sufficiently well-specified requirement to 
implement a robust tool.

Even if it is not to support arbitrary existing distutils sources, 
there still needs to be a way to document precisely what the tool 
does and does not support installing, so that users can produce 
correct files for it to consume, register them properly with PyPI, etc.

And as I said before (perhaps not very well) the distutils 
documentation has already proven to be inadequate as such a 
specification, both for users to create these files -- and even more 
important -- for programs to consume them.  (For example, distutils 
source distribution tarball filenames are not unambiguously machine-parseable.)

That's why I think some specific "format" (i.e. conventions) have to 
be defined for this to work, even if it's merely a set of documented 
restrictions on a distutils-based layout, file naming conventions, 
versioning, etc.

In other words, you can't have your cake and eat it, too.  If there's 
to be a bootstrap tool, you must bless *some* set of packaging 
conventions, including file naming, version parsing, and so on.

Can we use setuptools' version parsing scheme to identify the "latest 
stable version", for example?  What about setuptools' filename 
component canonicalization and escaping rules?

Frankly, I don't care what the conventions are, only that they be 
unambiguously defined and reasonably implementable for producers and 
consumers alike.

I just want there to be *some* sort of robust, documented, standard 
installation bootstrap vector in the stdlib, so that setuptools, 
zc.buildout, and virtualenv don't have to maintain their own (or 
depend on setuptools maintaining its own).

But not only have you rejected the *only* existing robust and 
well-documented conventions for automated processing of Python 
libraries, you say you "have no time for this part of the thread" 
when I ask what conventions you want to bless *instead*.

So I'm at a bit of a loss for what we're supposed to do now.


From pje at telecommunity.com  Tue Mar 18 17:34:26 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 12:34:26 -0400
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: <18399.21206.481836.992152@jon.es>
References: <18399.21206.481836.992152@jon.es>
Message-ID: <20080318163429.ABE703A4074@sparrow.telecommunity.com>

At 06:27 AM 3/18/2008 +0100, Terry Jones wrote:
>[Apologies if this is in the mailing list archives. I looked but didn't
>see anything, based on a search for run_setup]
>
>I'm trying to programmatically install something built using distutils. I
>found distutils.core.run_setup and can use it via
>
>   >>> dist = run_setup('setup.py', ['-q', 'install'])
>
>Is that the recommended way to do an install from inside Python (as opposed
>to doing it on the command line)?

I never could get run_setup() to work robustly.

Here's how setuptools does it, and it works well with quite a lot of packages:

         os.chdir(setup_dir)
         try:
             sys.argv[:] = [setup_script]+list(args)
             sys.path.insert(0, setup_dir)
             DirectorySandbox(setup_dir).run(
                 lambda: execfile(
                     "setup.py",
                     {'__file__':setup_script, '__name__':'__main__'}
                 )
             )
         except SystemExit, v:
             if v.args and v.args[0]:
                 raise
             # Normal exit, just return

This is actually wrapped in a try-finally that saves and restores 
sys.argv, sys.path, os.chdir(), etc.  Have a look at 
setuptools.sandbox.run_setup() for the full source.

Of course, this won't get you access to the distribution object; 
setuptools uses this mainly to run "bdist_egg" on the setup script, 
rather than having it do the installation directly.

(Note, by the way, that distutils setup scripts can get very confused 
if you don't os.chdir() to their directory before running them.)


>If so, how can I find where the thing(s) I installed now resides?

The simplest way would be to use the --record option to "install", to 
write a file listing all the installed files.


>I saw
>dist.packages but that just has top-level package names. I could __import__
>these (and then use module.__file__), but that's not a good solution as it
>may run code I don't want run. On my machine, I can see the packages have
>been installed under the system's python2.5/site-packages directory. But
>how can I determine that programmatically? I don't see anything useful on
>the distutils.dist.Distribution instance I'm getting back from run_setup.

It'd be on the 'install_lib' command instance, not the 
distribution.  Try 
dist.get_finalized_command('install_lib').install_dir instead.



From ziade.tarek at gmail.com  Tue Mar 18 17:55:13 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 18 Mar 2008 17:55:13 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: <20080318163429.ABE703A4074@sparrow.telecommunity.com>
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
Message-ID: <94bdd2610803180955k992d3a2i642b8074ffefbc4d@mail.gmail.com>

On Tue, Mar 18, 2008 at 5:34 PM, Phillip J. Eby <pje at telecommunity.com>
wrote:

> [cut]



> I never could get run_setup() to work robustly.
>


>
> Here's how setuptools does it, and it works well with quite a lot of
> packages:
>
>         os.chdir(setup_dir)
>         try:
>             sys.argv[:] = [setup_script]+list(args)
>             sys.path.insert(0, setup_dir)
>             DirectorySandbox(setup_dir).run(
>                 lambda: execfile(
>                     "setup.py",
>                     {'__file__':setup_script, '__name__':'__main__'}
>                 )
>             )
>         except SystemExit, v:
>             if v.args and v.args[0]:
>                 raise
>             # Normal exit, just return
>
> This is actually wrapped in a try-finally that saves and restores
> sys.argv, sys.path, os.chdir(), etc.  Have a look at
> setuptools.sandbox.run_setup() for the full source.
>

Cool ! Maybe this could be pushed into distutils ? It looks more like a fix
to me

Tarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080318/de2615f1/attachment.htm 

From mhammond at keypoint.com.au  Tue Mar 18 18:05:37 2008
From: mhammond at keypoint.com.au (mhammond at keypoint.com.au)
Date: Wed, 19 Mar 2008 02:05:37 +0900 (WST)
Subject: [Distutils] Consistent platform name for 64bit windows (was:
 distutils.util.get_platform() for Windows)
Message-ID: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>

I'm reviving a very old thread based on discussions with Martin at pycon.

> Sent: Monday, 23 July 2007 5:12 PM
> Subject: Re: [Distutils] distutils.util.get_platform() for Windows

Rather than forcing everyone to read the context, allow me to summarize:
On 64bit Windows versions, we need a "string" that identifies the
platform, and this string should ideally be used consistently.  This
original thread related to the files created by distutils (eg,
pywin32-210.win???64??-py2.6.exe) but it seems obvious that we should be
consistent wherever Python wants to display the platform (eg, in the
startup banner, in platform.py, etc).

In the old thread, there was a semi-consensus that 'x86_64' be used by
distutils (and indeed, Lib/distutils/util.py in get_platform() has been
changed, by me, to use this string), but the Python 'banner', for example,
reports AMD64.  Platform.py doesn't report much at all in this area, at
least when pywin32 isn't installed, but it arguably should.

Both Martin and I prefer AMD64 as the string, for various reasons. 
Firstly, it is less ugly than 'x86_64', and doesn't include an '_'/'-'
which might tend to confuse parsing by humans or computers.  Martin also
made the point that AMD invented the architecture and AMD64 is their
preferred name, so we should respect that.

So, at the risk of painting a bike-shed, I'd like to propose that we adopt
'AMD64' in distutils (needs a change), platform.py (needs a change to use
sys.getwindowsversion() in preference to pywin32, if possible, anyway),
and the Python banner (which already uses AMD64).

Any objections?  Any strong feelings that using 'AMD' will confuse people
with Intel processors?  Strong feelings about the parsability of the name
(PJE? <wink>)?  Strong feelings about the color <wink>?

Thanks,

Mark



From lists at cheimes.de  Tue Mar 18 18:54:08 2008
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 18 Mar 2008 18:54:08 +0100
Subject: [Distutils] Consistent platform name for 64bit windows (was:
 distutils.util.get_platform() for Windows)
In-Reply-To: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>
References: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>
Message-ID: <47E001C0.4000101@cheimes.de>

mhammond at keypoint.com.au schrieb:
> So, at the risk of painting a bike-shed, I'd like to propose that we adopt
> 'AMD64' in distutils (needs a change), platform.py (needs a change to use
> sys.getwindowsversion() in preference to pywin32, if possible, anyway),
> and the Python banner (which already uses AMD64).

+1 for AMD64

If we ever need names for Itanium and i386 compatible arch I propose 
IA64 and X86.

Christian

From terry at jon.es  Tue Mar 18 18:48:34 2008
From: terry at jon.es (Terry Jones)
Date: Tue, 18 Mar 2008 18:48:34 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: Your message at 12:34:26 on Tuesday, 18 March 2008
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
Message-ID: <18400.114.514905.239740@jon.es>

Hi Phillip & Tarek

Thanks for the replies, and the code!

Terry

From tnelson at onresolve.com  Tue Mar 18 19:02:24 2008
From: tnelson at onresolve.com (Trent Nelson)
Date: Tue, 18 Mar 2008 11:02:24 -0700
Subject: [Distutils] [Python-Dev] Consistent platform name for 64bit
 windows (was: distutils.util.get_platform() for Windows)
In-Reply-To: <47E001C0.4000101@cheimes.de>
References: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>,
	<47E001C0.4000101@cheimes.de>
Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E160CEE20@EXMBX04.exchhosting.com>

+1 for avoiding a bikeshed, so +1 to AMD64.

________________________________________
From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Christian Heimes [lists at cheimes.de]
Sent: 18 March 2008 13:54
To: mhammond at keypoint.com.au
Cc: distutils-sig at python.org; python-dev at python.org
Subject: Re: [Python-Dev] Consistent platform name for 64bit windows (was: distutils.util.get_platform() for Windows)

mhammond at keypoint.com.au schrieb:
> So, at the risk of painting a bike-shed, I'd like to propose that we adopt
> 'AMD64' in distutils (needs a change), platform.py (needs a change to use
> sys.getwindowsversion() in preference to pywin32, if possible, anyway),
> and the Python banner (which already uses AMD64).

+1 for AMD64

If we ever need names for Itanium and i386 compatible arch I propose
IA64 and X86.

Christian
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com

From pje at telecommunity.com  Tue Mar 18 19:27:51 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 14:27:51 -0400
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: <94bdd2610803180955k992d3a2i642b8074ffefbc4d@mail.gmail.com
 >
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
	<94bdd2610803180955k992d3a2i642b8074ffefbc4d@mail.gmail.com>
Message-ID: <20080318182754.B48C63A4074@sparrow.telecommunity.com>

At 05:55 PM 3/18/2008 +0100, Tarek Ziad? wrote:
>Cool ! Maybe this could be pushed into distutils ? It looks more 
>like a fix to me

Well, it's not really even a substitute for run_setup(), let alone a 
fix.  It just lets you run the script, not get access to its 
distribution object.

I'm not sure that run_setup() is fixable, though, even in 
principle.  But it's been too many years since I struggled with it, 
so I don't remember what the problem was.  In fact, I know for sure 
that I've gone through more than one cycle of forgetting what the 
problem was after not touching it for a few years.  :)


From marius at pov.lt  Tue Mar 18 19:51:54 2008
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 18 Mar 2008 20:51:54 +0200
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
Message-ID: <20080318185154.GA31013@fridge.pov.lt>

On Mon, Mar 17, 2008 at 08:19:11PM +0000, Paul Moore wrote:
> On 17/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> >  That leaves MAL, whose objections to PEP 365 centered on the API (he
> >  said he was "+1 on the concepts being added to the stdlib, -1 on
> >  adding the module in its current state").  Among other concerns, he
> >  wanted pkg_resources to be split into pkgutil and a new "egglib"
> >  module.  I don't have a problem with this in principle, if there were
> >  a pkg_resources module that reconstituted the merged API.  (But there
> >  are some practical problems with that approach, such as trying to
> >  split namespace package support between two theoretically-unrelated modules.)
> 
> I would personally like to see such a separation. As one of the
> authors of PEP 302 (well, the documentation - Just did all of the
> implementation) I have an interest in strengthening the standard
> library's support for transparent use of PEP 302 importers. To that
> end, I would like to see such functions as
> pkg_resources.resource_string() standardised.

+1 for something like this in the stdlib.

os.path.join(os.path.dirname(__file__), 'foo') just has too many
problems.

Could namespace package support be split into a set of hooks in the
stdlib and a concrete implementation in setuptools?  Isn't that the way
zip importers were originally done?

Marius Gedminas
-- 
As an aside, UPnP's implementation (which features SOAP, HTTP over
multicast/broadcast UDP, and extremely odd XML) is a must-read for fans of
unnatural and baroque network protocols.
        -- Anthony Baxter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080318/173cfb8c/attachment-0001.pgp 

From dpeterson at enthought.com  Tue Mar 18 20:20:49 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Tue, 18 Mar 2008 14:20:49 -0500
Subject: [Distutils] Consistent platform name for 64bit windows
In-Reply-To: <47E001C0.4000101@cheimes.de>
References: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>
	<47E001C0.4000101@cheimes.de>
Message-ID: <47E01611.2040600@enthought.com>

Christian Heimes wrote:
> mhammond at keypoint.com.au schrieb:
>   
>> So, at the risk of painting a bike-shed, I'd like to propose that we adopt
>> 'AMD64' in distutils (needs a change), platform.py (needs a change to use
>> sys.getwindowsversion() in preference to pywin32, if possible, anyway),
>> and the Python banner (which already uses AMD64).
>>     
>
> +1 for AMD64
>   

I'm also +1 for AMD64 (or amd64).

BTW, the wikipedia entry for AMD64 points out that FreeBSD, NetBSD, 
OpenBSD, Debian, Ubuntu, Solaris, and the Java Development Kit all use 
"amd64".   Though it claims that Microsoft and Sun both use 'x64' in 
their marketing material.

> If we ever need names for Itanium and i386 compatible arch I propose 
> IA64 and X86.
>   

IA64 seems pretty standard so +1 on that.

But I'm -1 on the X86.  It's too easy to get confused with what that 
means versus terms like x86-16, x86-32, and x86-64.  Admittedly the 
first two are little used, but I have seen them once or twice.  What's 
wrong with using i386 itself?


-- Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080318/21480340/attachment.htm 

From marius at pov.lt  Tue Mar 18 20:24:46 2008
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 18 Mar 2008 21:24:46 +0200
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
	Packaging/Deployment Technology Concerns
In-Reply-To: <20080318004718.56E173A40FF@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
Message-ID: <20080318192446.GB31013@fridge.pov.lt>

On Mon, Mar 17, 2008 at 08:37:30PM -0400, Phillip J. Eby wrote:
> At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote:
> >     People also want a greater variety of file_finders to be included with
> >     setuptools.  Instead of just CVS and SVN, they want it to comprehend
> >     Mercurial, Bazaar, Git and so forth.
> 
> Did you point them to the Cheeseshop?  There are plugins already 
> available for all the systems you mentioned, plus Darcs and 
> Monotone.  If you mean "included" as in "bundled", this doesn't make 
> a whole lot of sense to me.

It does to me.  Think "batteries included".

Running 'setup.py sdist/bdist_foo' should not silently produce an
incorrect package with some files missing just because the person
running it did not have some additional plugin installed.

> I'd think that if you're using 
> setuptools as a developer (the only reason you need the file finders, 
> since source distributions include a prebuilt manifest), you'd not 
> have a problem saying "easy_install setuptools-git" or adding a 
> "setup_requires='setuptools-git'" line to your setup.py.  (Although 
> the latter would only be needed for *development*, not deployment.)

setup_requires looks like a solution, but it requires extra attention
from the developers who write the setup.py.  Writing a setup.py is
already quite complicated -- I usually end up copying an existing one
and modifying it.

Marius Gedminas
-- 
Most security experts REALLY believe in firewalls. The expect that, when they
die, arrive at the great firewall in the sky where Saint Peter is running a
default policy of REJECT.
                --- Sander Plomp
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080318/ff07c663/attachment.pgp 

From jeff at taupro.com  Tue Mar 18 21:20:19 2008
From: jeff at taupro.com (Jeff Rush)
Date: Tue, 18 Mar 2008 15:20:19 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080318192446.GB31013@fridge.pov.lt>
References: <47DEEC6B.5040602@taupro.com>	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<20080318192446.GB31013@fridge.pov.lt>
Message-ID: <47E02403.4040909@taupro.com>

Marius Gedminas wrote:
> On Mon, Mar 17, 2008 at 08:37:30PM -0400, Phillip J. Eby wrote:
>> At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote:
>>>     People also want a greater variety of file_finders to be included with
>>>     setuptools.  Instead of just CVS and SVN, they want it to comprehend
>>>     Mercurial, Bazaar, Git and so forth.
>> Did you point them to the Cheeseshop?  There are plugins already 
>> available for all the systems you mentioned, plus Darcs and 
>> Monotone.  If you mean "included" as in "bundled", this doesn't make 
>> a whole lot of sense to me.

They knew there were plugins out there, of various quality and availability 
but wanted them bundled. ;-)  It's a pain to track them down.  Perhaps if the 
RPM format were broken out from setuptools, as the inclusion of some formats 
leads them to believe the set is just incomplete, not intentionally sparse.


>> I'd think that if you're using 
>> setuptools as a developer (the only reason you need the file finders, 
>> since source distributions include a prebuilt manifest), you'd not 
>> have a problem saying "easy_install setuptools-git" or adding a 
>> "setup_requires='setuptools-git'" line to your setup.py.  (Although 
>> the latter would only be needed for *development*, not deployment.)
> 
> setup_requires looks like a solution, but it requires extra attention
> from the developers who write the setup.py.  Writing a setup.py is
> already quite complicated -- I usually end up copying an existing one
> and modifying it.

As a compromise, of making new formats easily available but not bundled, and 
not requiring special action within setup.py, setuptools could treat 
--formats=dpkg as an implicit setup_requires= and pull it from PyPI.  And the 
--list-formats option could query PyPI for the possibilities, just as 
--list-classifiers does today.  If would require a few standards in 
keywording/classifying those format eggs but we already need those standards 
for other projects, such as locating recipes for buildout and plugins for trac.

-Jeff


From stephen at xemacs.org  Tue Mar 18 21:45:53 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 19 Mar 2008 05:45:53 +0900
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
Message-ID: <87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > I am hoping that someone will create a simpler bootstrap module

FWIW (I've never tried to implement one of these things) I agree with
Phillip.  This is not possible in the sense you are advocating.
Anything "simpler" is simply an invitation to an unending stream of
issues, far more so than adopting eggs as a best current practice
would.  Better to Just Say No to an installer in the stdlib.

 > I'm not sure a PEP is even needed

This I simply don't understand.  I *have* participated in bolting on
features to such systems, and it's a mess.  As features are added,
existing users will demand that old packages install exactly as they
ever did, except that sometimes (but not always!) they want them
upgraded to put things in newly blessed places.  Features are easy to
add, since on the main thread of control installers are basically just
a sequence of single commands, sometimes optional.

python-dev has some pretty good controls that will help slow such
trends.  Nonetheless, PEP-less development of an installer system is
scary.  Installer complexity is a creeping horror, worse than
kudzu.[1]


Footnotes: 
[1]  http://en.wikipedia.org/wiki/Kudzu#Invasive_species


From guido at python.org  Tue Mar 18 21:40:03 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Mar 2008 15:40:03 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080318162331.C820D3A4074@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<20080318162331.C820D3A4074@sparrow.telecommunity.com>
Message-ID: <ca471dc20803181340v64d2c594v49decb263d5d44c@mail.gmail.com>

On Tue, Mar 18, 2008 at 11:23 AM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 12:31 AM 3/18/2008 -0500, Guido van Rossum wrote:
>  >I am hoping that someone will create a simpler bootstrap module that
>  >is able to download a file of pure Python code and install it, perhaps
>  >by running its setup.py, assuming that it only depends on distutils
>  >(or other things previously installed). I will welcome such a module
>  >into the stdlib. I'm not sure a PEP is even needed, though interested
>  >parties are certainly welcome to write a PEP specifying the behavior
>  >first. With 2.6 and 3.0 slated for release in September, there should
>  >be enough time to get this done before then.
>
>  Unfortunately, as I've already tried to explain, "download a file ...
>  and install it" is not a sufficiently well-specified requirement to
>  implement a robust tool.
>
>  Even if it is not to support arbitrary existing distutils sources,
>  there still needs to be a way to document precisely what the tool
>  does and does not support installing, so that users can produce
>  correct files for it to consume, register them properly with PyPI, etc.
>
>  And as I said before (perhaps not very well) the distutils
>  documentation has already proven to be inadequate as such a
>  specification, both for users to create these files -- and even more
>  important -- for programs to consume them.  (For example, distutils
>  source distribution tarball filenames are not unambiguously machine-parseable.)
>
>  That's why I think some specific "format" (i.e. conventions) have to
>  be defined for this to work, even if it's merely a set of documented
>  restrictions on a distutils-based layout, file naming conventions,
>  versioning, etc.
>
>  In other words, you can't have your cake and eat it, too.  If there's
>  to be a bootstrap tool, you must bless *some* set of packaging
>  conventions, including file naming, version parsing, and so on.
>
>  Can we use setuptools' version parsing scheme to identify the "latest
>  stable version", for example?  What about setuptools' filename
>  component canonicalization and escaping rules?
>
>  Frankly, I don't care what the conventions are, only that they be
>  unambiguously defined and reasonably implementable for producers and
>  consumers alike.
>
>  I just want there to be *some* sort of robust, documented, standard
>  installation bootstrap vector in the stdlib, so that setuptools,
>  zc.buildout, and virtualenv don't have to maintain their own (or
>  depend on setuptools maintaining its own).
>
>  But not only have you rejected the *only* existing robust and
>  well-documented conventions for automated processing of Python
>  libraries, you say you "have no time for this part of the thread"
>  when I ask what conventions you want to bless *instead*.
>
>  So I'm at a bit of a loss for what we're supposed to do now.

You're welcome to write a PEP as long as it is self-contained (at best
referencing other accepted PEPs like the PyPI metadata PEPs). And the
PEP better not be 2500 lines long.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From dpeterson at enthought.com  Tue Mar 18 21:40:10 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Tue, 18 Mar 2008 15:40:10 -0500
Subject: [Distutils] Capsule Summary of Some Packaging/Deployment
 Technology Concerns
In-Reply-To: <47DEEC6B.5040602@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
Message-ID: <47E028AA.3070300@enthought.com>

I've added your comments to a wiki page 
(http://wiki.python.org/moin/PackagingBOF) I was working on to summarize 
some of what went on during these BoF meeting, at least from the 
Enthought point-of-view.  Unfortunately, I wasn't at the first night's 
event and don't yet have Travis Oliphant's notes on it here in front of 
me (he's still sprinting) so I only added some more detail to your 
comments, and also noted some previous issues we'd briefly discussed via 
e-mail with Phillip.

It was great to see so many people interested in sharing their 
experiences and wanting to help things get better!  As you can probably 
guess as a result of this being a two-night meeting, there wasn't enough 
time to discuss everything that needed to be brought up.  It was 
suggested that a wiki page be created (see above) and that a new mailing 
list be setup for those who wanted to discuss further.   (Some didn't 
feel the existing distutils-sig was appropriate.)   I'll try to get the 
latter done shortly.

-- Dave


Jeff Rush wrote:
> I was in a Packaging BoF yesterday and, although not very relevant to the 
> packager bootstrap thread, Guido has asked me to post some of the concerns.
>
> The BoF drew about 15 people...
>   

<snipped>


From guido at python.org  Tue Mar 18 21:43:41 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Mar 2008 15:43:41 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>

There seems to be a misunderstanding about what I am proposing we do
instead. The boostrap installer should only be powerful enough to
allow it to be used to install a real package manager like setuptools.
Maybe my use of Django as an example was confusing; I didn't actually
mean that it should be possible to support installing Django directly
(although I don't understand all the issure that would make it
impossible). Only very few people would care about writing a setup
script that works with this bootstrap module; basically only package
manager implementers.

--Guido

On Tue, Mar 18, 2008 at 3:45 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Guido van Rossum writes:
>
>   > I am hoping that someone will create a simpler bootstrap module
>
>  FWIW (I've never tried to implement one of these things) I agree with
>  Phillip.  This is not possible in the sense you are advocating.
>  Anything "simpler" is simply an invitation to an unending stream of
>  issues, far more so than adopting eggs as a best current practice
>  would.  Better to Just Say No to an installer in the stdlib.
>
>
>   > I'm not sure a PEP is even needed
>
>  This I simply don't understand.  I *have* participated in bolting on
>  features to such systems, and it's a mess.  As features are added,
>  existing users will demand that old packages install exactly as they
>  ever did, except that sometimes (but not always!) they want them
>  upgraded to put things in newly blessed places.  Features are easy to
>  add, since on the main thread of control installers are basically just
>  a sequence of single commands, sometimes optional.
>
>  python-dev has some pretty good controls that will help slow such
>  trends.  Nonetheless, PEP-less development of an installer system is
>  scary.  Installer complexity is a creeping horror, worse than
>  kudzu.[1]
>
>
>  Footnotes:
>  [1]  http://en.wikipedia.org/wiki/Kudzu#Invasive_species

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From p.f.moore at gmail.com  Tue Mar 18 22:01:14 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 18 Mar 2008 21:01:14 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080318185154.GA31013@fridge.pov.lt>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
Message-ID: <79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>

On 18/03/2008, Marius Gedminas <marius at pov.lt> wrote:
> +1 for something like this in the stdlib.
>
>  os.path.join(os.path.dirname(__file__), 'foo') just has too many
>  problems.

Hmm, maybe I should put some of my money where my mouth is. I'd be
willing to work up a patch to add some key items into the pkgutils
module. However, I don't like the thought of reinventing the wheel -
Phillip, would you be OK with me stealing code where I can from
pkg_resources?

Paul.

From pje at telecommunity.com  Tue Mar 18 23:23:10 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 18:23:10 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
 module)
In-Reply-To: <79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.co
 m>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
Message-ID: <20080318222317.8C4B83A409D@sparrow.telecommunity.com>

At 09:01 PM 3/18/2008 +0000, Paul Moore wrote:
>On 18/03/2008, Marius Gedminas <marius at pov.lt> wrote:
> > +1 for something like this in the stdlib.
> >
> >  os.path.join(os.path.dirname(__file__), 'foo') just has too many
> >  problems.
>
>Hmm, maybe I should put some of my money where my mouth is. I'd be
>willing to work up a patch to add some key items into the pkgutils
>module. However, I don't like the thought of reinventing the wheel -
>Phillip, would you be OK with me stealing code where I can from
>pkg_resources?

Sure...  though I doubt you'll have much luck, unless you drop 
support for resource_filename, and stick with 
resource_string()/resource_stream().  Supporting resource_filename() 
for zipfiles inherently requires that you extract the file somewhere, 
which is where all the jillions of lines of code for import providers 
and importer registrations and egg caches and all that other stuff come in.

By contrast, returning a resource string or stream is easy, since you 
can just delegate to a __loader__ object's get_data, or fall back to 
filesystem access if the importer is None.

In Python 2.5, the pkgutil module grew a bunch of helpful APIs that 
originally came from pkg_resources.  (If you look at pkg_resources in 
the setuptools 0.7 trunk, it actually uses these new pkgutil APIs, 
instead of the bundled versions that live in the 0.6 branch.)  You 
should be able to use pkgutil.get_loader(modulename) to get either a 
loader or None, and then look for a 'get_data' method from there.


From pje at telecommunity.com  Tue Mar 18 23:36:57 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 18:36:57 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
Message-ID: <20080318223700.C64123A4074@sparrow.telecommunity.com>

At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote:
>Only very few people would care about writing a setup
>script that works with this bootstrap module; basically only package
>manager implementers.

That's true today, sure, but as soon as it is widely available, 
others are sure to want to use it too.  I just want a bright-line 
distinction between what is and isn't bootstrappable, rather than a 
murky region of "maybe, if you're not doing anything too complicated".


>There seems to be a misunderstanding about what I am proposing we do
>instead. The boostrap installer should only be powerful enough to
>allow it to be used to install a real package manager like setuptools.

Which is why PEP 365 proposed only downloading an archive to a cache 
directory, and optionally running something from it.  It explicitly 
disavows "installation" of anything, since the downloaded archive 
wouldn't have been added to sys.path except for the duration of the 
bootstrap process, and no scripts were to be installed.  (Indeed, 
apart from the methods it would have used to locate the archive on 
PyPI, and to determine what to run from inside it, there was nothing 
particularly egg-specific about the proposed bootstrapping process.)

So, to fully egg-neutralize the bootstrapping approach, we need only 
know how to locate an appropriate archive, and how to determine what 
to run from it.

For the latter, we could use the already-in-2.6 convention of running 
__main__ from a zipfile or directory.  (Too bad distutils source 
distributions have an extra directory name embedded in them, so one 
can't just execute them directly.  Otherwise, we could've just let 
people drop in a __main__.py next to setup.py.  OTOH, maybe it would 
be enough to use setuptools' algorithm for finding setup.py to locate 
__main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.)

The other open question is a naming convention and version detection, 
so that the bootstrap tool can identify which of the files listed on 
PyPI is suitable for its use.  (Both with regard to the version 
selection, and file type.)  However, if PyPI were to grow support for 
designating the appropriate files and/or versions in some other way, 
we wouldn't need a naming convention as such.

Without one or the other, the bootstrap tool would have to grow a 
version parsing scheme of some type, and play guessing games with 
file extensions.  (Which is one reason I limited PEP 365's scope to 
downloading eggs actually *uploaded* to PyPI, rather than arbitrary 
packages *linked* from PyPI.)

So, if I had to propose something right now, I would be inclined to propose:

* using setuptools' version parsing semantics for interpretation of 
alpha/beta/dev/etc. releases

* having a bdist_bootstrap format that's essentially a bdist_dumb 
.zip file with the internal path prefixes stripped off, making it an 
importable .zip with a different file extension.  (Or maybe just 
.pyboot.zip?)  The filename convention would use setuptools' 
canonicalization and escaping of names and version numbers, to allow 
unambiguous machine parsing of the filename.  A __main__ module would 
have to be present for the archive to be run, as opposed to just 
being downloaded to a temporary directory.

* calling the bootstrap module 'bootstrap', as in 'python -m 
bootstrap projectname optionalversion'.  The module would expose an 
API to allow it to be used programmatically as well as the command 
line, so that bootstrapped packages can use the bootstrap process to 
locate dependencies if they so desire.  (Today's package management 
tools, at least, are all based on setuptools, so if it's not present 
they'll need to download that before beginning their own 
bootstrapping process.)

Apart from keeping the PEP self-contained and short, is there 
anything in this that you think you would object to?  (You may 
reserve the right, of course, to later not like something in the 
details of setuptools' version/filename rules, after I've put them 
into the PEP, or really, anything else.  I'm just asking if there's 
anything that's obviously offensive at this point, before I spend 
time on a new PEP.)


From p.f.moore at gmail.com  Tue Mar 18 23:44:41 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 18 Mar 2008 22:44:41 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080318222317.8C4B83A409D@sparrow.telecommunity.com>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
Message-ID: <79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>

On 18/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> Sure...  though I doubt you'll have much luck, unless you drop
>  support for resource_filename, and stick with
>  resource_string()/resource_stream().  Supporting resource_filename()
>  for zipfiles inherently requires that you extract the file somewhere,
>  which is where all the jillions of lines of code for import providers
>  and importer registrations and egg caches and all that other stuff come in.

Yes, I intended to skip resource_filename. It's controversial at best.

>  By contrast, returning a resource string or stream is easy, since you
>  can just delegate to a __loader__ object's get_data, or fall back to
>  filesystem access if the importer is None.

Agreed. I'm not suggesting that anything hard be added, rather that
(relatively) simple things get standardised.

>  In Python 2.5, the pkgutil module grew a bunch of helpful APIs that
>  originally came from pkg_resources.

In which case, the documentation wasn't updated. The 2.5 docs only
mention extend_path.

>  (If you look at pkg_resources in
>  the setuptools 0.7 trunk, it actually uses these new pkgutil APIs,
>  instead of the bundled versions that live in the 0.6 branch.)  You
>  should be able to use pkgutil.get_loader(modulename) to get either a
>  loader or None, and then look for a 'get_data' method from there.

Someone should document these new functions :-( Was it you that added
them? Assuming it was, what's the "simplegeneric" function doing in
there? Can it be moved somewhere more sensible?

I don't particularly want to use undocumented functions - even within
the module that defines them.

Paul.

From janssen at parc.com  Wed Mar 19 00:59:45 2008
From: janssen at parc.com (Bill Janssen)
Date: Tue, 18 Mar 2008 16:59:45 PDT
Subject: [Distutils] platform management
In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E160CEE20@EXMBX04.exchhosting.com>
References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E160CEE20@EXMBX04.exchhosting.com>
Message-ID: <08Mar18.165949pdt."58696"@synergy1.parc.xerox.com>

I don't think this is bike-shedding.

The debate about "AMD64" vs. "amd64" vs. "x86_64" reminded me that
I've been bit more and more frequently by bits of platform-specific
knowledge scattered around the standard library.  The latest is the
code in distutils.unixccompiler that tries to figure out what flags to
send to the linker in order to add a dynamic library path lookup to a
shared library.

There are lots of different ways of figuring out which platform is
being used, and they're all over the place.  The code in
distutils.unixccompiler uses "sys.platform[:6]", and looks for
"darwin"; the code in urllib.py uses "os.name", and expects it to be
"mac", but later looks for "sys.platform == 'darwin'; posixfile
believes that platforms come with version numbers ("linux2", "aix4");
pydoc and tarfile have tests for "sys.platform == 'mac'".  tempfile
looks at os.sys.platform *and* os.name.

Could well be that all of these are correct (I believe that "mac", for
instance, refers to the generations of the Mac OS before OS X).  But
it means that when someone tries to update "Python" to a new major
version release for, say, OS X, it's really easy to miss things.  And
the fact that the platform version is sometimes included and sometimes
not is also problematic; darwin9 is different from darwin8 in some
important aspects.

It would be nice to

  (a) come up with a list of standard platform symbols,
  (b) put those symbols somewhere in the documentation,
  (c) have one way of discovering them, via sys or os or platform or
      whichever module,
  (d) add a standard way of discovering the platform version, and
  (e) stamp out all the other ways that have crept in over the years.

Bill

From pje at telecommunity.com  Wed Mar 19 01:05:27 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 20:05:27 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
 module)
In-Reply-To: <79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.co
 m>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
Message-ID: <20080319000559.C80223A4074@sparrow.telecommunity.com>

At 10:44 PM 3/18/2008 +0000, Paul Moore wrote:
>Someone should document these new functions :-( Was it you that added
>them?

Yes, and it was specifically to start pulling some of that stuff out 
of pkg_resources so it'd be more widely available.

>  Assuming it was, what's the "simplegeneric" function doing in
>there?

It's a substitute for the adapter machinery that the corresponding 
code used when it lived in pkg_resources.  Guido had just shot down 
adapting in favor of generic functions, so I used what I thought 
would be a more palatable name.  :)


>Can it be moved somewhere more sensible?

Er, sure, if you can get that past Python-Dev.  It's not a 
particularly full-featured generic function implementation, just a 
crude subset of the simplegeneric that's on PyPI.  If I were doing it 
today, I'd have called it 'trivialgeneric' instead.  ;)


>I don't particularly want to use undocumented functions - even within
>the module that defines them.

Er, you could always document them, you know.  I just didn't get 
around to it before the whole "setuptools flap of 2005" wiped out my 
motivation to do any further work on Python 2.5.  For the most part, 
they do have docstrings.  I just never did the LaTeX work on 
them.  And now, you can use reST instead of LaTeX.  :)


From dpeterson at enthought.com  Wed Mar 19 01:27:01 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Tue, 18 Mar 2008 19:27:01 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080318004718.56E173A40FF@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
Message-ID: <47E05DD5.1090707@enthought.com>

Phillip J. Eby wrote:
> At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote:
>   
>
>> 1. Many felt the existing dependency resolver was not correct.  They wanted a
>>     full tree traversal resulting in an intersection of all restrictions,
>>     instead of a first-acceptable-solution approach taking now, which can
>>     result in top-level dependencies not being enforced upon lower 
>> levels.  The
>>     latter is faster however.  One solution would be to make the resolver
>>     pluggable.
>>     
>
> Patches welcome, on both counts.  Personally, Bob and I originally 
> wanted a full-tree intersection, too, but it turned out to be hairier 
> to implement than it seems at first.  My guess is that none of the 
> people who want it, have actually tried to implement it without a 
> factorial or exponential O().  But that doesn't mean I'll be unhappy 
> if somebody succeeds.  :)
>   

I think we'd make significant progress by just intersecting the 
dependencies we know about as we progress through the dependency tree.  
For example, if A requires B==2 and C==3, and if B requires C>=2,<=4, 
then at the time we install A we'd pick C==3 and also at the time we 
install B we'd pick C==3.   As opposed to the current scheme that would 
choose C==4 for the latter case.   This would allow dependent projects 
(think applications here) to better control the versions of the full set 
of libraries they use.   Things would still fail (like they do now) if 
you ran across dependencies that had no intersection or if you 
encountered a new requirement after the target projected was already 
installed.


If you really wanted to do a full-tree intersection, it seems to me that 
the problem is detecting all the dependencies without having to spend 
significant time downloading/building in order to find them out.   This 
could be solved by simply extending the cheeseshop interface to export 
the set of requirements outside of the egg / tarball / etc.  We've done 
this for our own egg repository by extracting the appropriate meta-data 
files out of EGG-INFO and putting it into a separate file.  This info is 
also useful for users as it gives them an idea of how much *new* stuff 
is going to be installed (a la yum, apt-get, etc.)


> In other words, we attempt to achieve heuristically what's being 
> proposed to do algorithmically.  And my guess is that whatever cases 
> the heuristic is failing at, would probably not be helped by an 
> algorithmic approach either.  But I would welcome some actual data, either way.
>   

With our ETS projects, we've run into problems with the current 
heuristic.  Perhaps we just don't know how to make it work like we want? 

We have a set of projects that we want to be individually installable 
(to the extent that we limit cross-project dependencies) but we also 
want to make it easy to install the complete set.  We use a meta-egg for 
the latter.  It's purpose is only to specify the exact versions of each 
project that have been explicitly tested to work together -- you could 
almost think of it as a source control system tag.  Whereas on the 
individual projects, we explicitly want to ensure that people get the 
latest possible release of each required API so the version requirements 
are wider here.   This setup causes problems whenever we release new 
versions of projects because it seems easy_install ignores the meta-egg 
exact versions when it gets down into a project and comes across a wider 
cross-project dependency.   We ended up having to give up on the ranges 
in the cross-project dependencies and synchronize them to the same 
values in the meta-egg dependencies.   There are numerous side-effects 
of this that we don't like but we haven't found a way around it.

> Again, though, patches are welcome.  :)  (Specifically, for the 
> trunk; I don't see a resolver overhaul as being suitable for the 0.6 
> stable branch.)
>   

We're planning to pursue this (for the above mentioned strategy) as soon 
as we work ourselves out of a bit of a backlog of other things to do.



>> 2. People want a solution for the handling of documentation.  The distutils
>>     module has had commented out sections related to this for several years.
>>     
>
> As with so many other things, this gets tossed around the 
> distutils-sig every now and then.  A couple of times I've thrown out 
> some options for how this might be done, but then the conversation 
> peters out around the time anybody would have to actually do some 
> work on it.  (Me included, since I don't have an itch that needs 
> scratching in this area.)
>
> In particular, if somebody wants to come up with a metadata standard 
> for including documentation in eggs, we've got a boatload of hooks by 
> which it could be done.  Nothing's stopping anybody from proposing a 
> standard and building a tool, here.  (e.g. using the setuptools 
> command hook, .egg-info writer hook, etc.)

Enthought has started an effort (it's currently one of two things in our 
ETSProjectTools project at 
https://svn.enthought.com/svn/enthought/ETSProjectTools/trunk) and we're 
experimenting with our solution before proposing it as a patch.  We'd 
love some more help if anyone wants to participate.


>> 3. A more flexible internal handing of the different types of files is needed.
>>     Currently the code, data, lib, etc. files are aggregated at 
>> build time and
>>     people would like them to be kept separate until install/packaging time.
>>     
>
> I don't know what this means, exactly.
>   

A number of projects want to provide various types of files besides code 
in their distributable, and they'd like these to end up in standard 
locations for that type of file.  Think documentation, sample data, web 
templates, configuration settings, etc.   Each of these should be 
treated differently at installation time depending on platform.  On 
*nix, docs should go in /usr/share/doc whereas we might need to create a 
C:\Python2.5\docs on Windows.   With sample data and templates, you 
probably just want it accessible outside of the zipped egg so users can 
easily look at it, add to it, edit it, etc.  Configuration settings 
should be installed with some defaults into a standard configuration 
directory like /etc on *nix, etc.

Basically the issue is that it needs to be easier to include different 
sets of files into an egg for different actions to be taken during 
installation or packaging into an OS-specific distribution format.


>>     The other is the use of a single .pth file to control the list 
>> of activated
>>     packages.  Those who produce distributions would prefer a magic directory
>>     into which links to distributions could be dropped, similar to 
>> the current
>>     best practices for Linux, with /etc/conf.d/, /etc/profile.d/,
>>     /etc/xinetd.d/ and so forth.
>>     
>
> site-packages is that directory, and has been since long before 
> setuptools.  Just drop uniquely-named .pth files there, and you're good to go.
>   

But the docs for easy_install claim that the list of active eggs is 
maintained in easy-install.pth.  Also, if I create my own .pth file, and 
the user tries to update my version to a new one, will the easy_install 
tool modify my .pth file to remove the mention of the old version from 
my sys.path and put the new version in the same .pth file?  Or will it 
now be listed in both places?  Or will it only in easy-install.pth?  


>> 7. Many wanted to ability to install files anywhere in the install tree and
>>     not just under the Python package.  Under distutils this was possible but
>>     it was removed in setuptools for security reasons.
>>     
>
> It wasn't security, it was manageability.  Egg-based installation 
> means containment, (analagous to GNU stow) and therefore portability 
> and disposability of plugins.  (Which again is what eggs were really 
> developed for in the first place.)
>   

Yes, but as you've already pointed out, they've escaped into a larger 
ecosystem and this restriction is a severe limitation -- leading to 
significant frustration.  Especially as projects evolve and want to do 
something more complex than simply install pure Python code.  Here at 
Enthought, we use and ship a number of projects that have extensions and 
thus dynamic libraries that need to either be modified during 
installation to work from the user's installed location, or copied 
elsewhere on the system to avoid the need to modify (which we also can't 
do via an egg install) env variables, registries, etc.   We'd also love 
to be able to ship end-user enterprise-scale applications via eggs so 
that bug fixes and updates don't require downloading a monolithic 100MB+ 
installer.  But doing that requires the ability to update desktop icons, 
menus, etc. which we also can't do automatically via an egg.

If you don't want the burden on setuptools to support, much less track, 
all these options, then perhaps it could just support automatic 
execution of a post-install script (and pre-uninstall script if 
uninstallation ever happens) that allows individual project developers 
to do what they need to do?  Let the burden of describing how those 
things happen and how to uninstall/relocate/update them fall to the 
provider of the projects that do them.

Also, IIUC, stow only tries to "contain" the hard files.  It puts links 
in multiple standard locations (for man pages, executables, libraries, 
etc.)   If setuptools supported these options, I don't think there'd be 
any discussion here except for things like "how do I extend the set of 
things the tool supports so that my foo-type files get linked into the 
standard /os/path/to/foo for the X os?"


>>   Custom code can still
>>     be written to do this explicitly but this is not popular.
>>     
>
> No kidding.  :)  Current best practice is to include a script or 
> module in the package that can install other files to a designated 
> location.  Personally, though, I tend to view applications and 
> libraries that target specific install locations to be overreaching 
> their bounds, and stepping into sysadmin territory.  Give me the 
> tools to install the data, don't just dump it somewhere on my system 
> where *you* think it should go, in other words.
>   

I should have read ahead.  This sounds close to what I've been 
describing except that this leads me to picture a script that prompts 
for install locations and allows the user to customize the destinations 
rather than one that assumes everything goes in a standard place.  I'm 
all for this, and the continuation of the ability to install an egg into 
a user-environment vs. a system-environment.  

The only thing missing here is the ability for the installer to 
automatically run that script so that installation isn't a disjointed, 
two-step manual process that a user is prone to forgot to complete. 

One of the features of Enthought's Enstaller extension to easy_install 
was that it looks for a post_install.py script in EGG-INFO and if one is 
found, runs it.  I would think that getting this into setuptools would 
be a significant step forward but I believe you previously rejected that 
idea.   We'll take a stab at creating a patch for you if you're more 
receptive to that idea now.  Just let me know.


> On the other hand, I've been puzzling over how to handle legitimate 
> post-install features.  On Windows, both wx and pywin32 have a real 
> need to do some actuall "install" operations.  Some is just copying 
> files, but pywin32 also has to do some registry stuff.  I don't know 
> how to allow just what's sensible, without opening up a huge can of 
> worms, though.
>   

I think there are lots of situations that are legitimate (projects with 
extensions, projects that want to put icons on the desktop or in menus, 
projects that need to interact with a registry, projects that want to 
put configuration information somewhere other than in a zip file in a 
site-packages dir, etc.)   I think we should worry less about preventing 
developers from shooting themselves in the foot and more about ensuring 
that they can hunt for food for their survival.   We can always tighten 
things down after seeing the usecases that develop, right?


-- Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080318/ac9b10a3/attachment-0001.htm 

From pje at telecommunity.com  Wed Mar 19 02:20:44 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 18 Mar 2008 21:20:44 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47E05DD5.1090707@enthought.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E05DD5.1090707@enthought.com>
Message-ID: <20080319012046.7B1233A4074@sparrow.telecommunity.com>

We should probably move this off of Python-Dev, as we're getting into 
deep details now...

At 07:27 PM 3/18/2008 -0500, Dave Peterson wrote:
>If you really wanted to do a full-tree intersection, it seems to me 
>that the problem is detecting all the dependencies without having to 
>spend significant time downloading/building in order to find them 
>out.   This could be solved by simply extending the cheeseshop 
>interface to export the set of requirements outside of the egg / 
>tarball / etc.  We've done this for our own egg repository by 
>extracting the appropriate meta-data files out of EGG-INFO and 
>putting it into a separate file.  This info is also useful for users 
>as it gives them an idea of how much *new* stuff is going to be 
>installed (a la yum, apt-get, etc.)

...and now we're more directly competing with them, too.  The 
original idea Bob and I had was to do XML files ala Eclipse feature 
repositories, but then later I realized that for what we were doing, 
HTML was both adequate and already available.  However, I don't see a 
problem in principle with having "header" files available for this 
sort of thing.


>With our ETS projects, we've run into problems with the current 
>heuristic.  Perhaps we just don't know how to make it work like we want?
>
>We have a set of projects that we want to be individually 
>installable (to the extent that we limit cross-project dependencies) 
>but we also want to make it easy to install the complete set.  We 
>use a meta-egg for the latter.  It's purpose is only to specify the 
>exact versions of each project that have been explicitly tested to 
>work together -- you could almost think of it as a source control system tag.

I would think that as long as that meta-egg specifies *all* the 
required versions (right down to recursive dependencies), then there 
shouldn't be any problem.  Maybe it's me who's not understanding something?

I would think that you could get the appropriate data by running the 
tl.eggdeps tool.


>A number of projects want to provide various types of files besides 
>code in their distributable, and they'd like these to end up in 
>standard locations for that type of file.  Think documentation, 
>sample data, web templates, configuration settings, etc.   Each of 
>these should be treated differently at installation time depending 
>on platform.  On *nix, docs should go in /usr/share/doc whereas we 
>might need to create a C:\Python2.5\docs on Windows.   With sample 
>data and templates, you probably just want it accessible outside of 
>the zipped egg so users can easily look at it, add to it, edit it, 
>etc.  Configuration settings should be installed with some defaults 
>into a standard configuration directory like /etc on *nix, etc.
>
>Basically the issue is that it needs to be easier to include 
>different sets of files into an egg for different actions to be 
>taken during installation or packaging into an OS-specific distribution format.

Yes, it would be nice to define a metadata standard for including 
installable "datasets" either through copying or symlinking, 
optionally with entry points for running some code, too.  When you 
install an egg, these things could get added to a "post-install 
to-do" list, that you could then read to find out what steps to do, 
or invoke a tool on to actually do some of those steps.


>But the docs for easy_install claim that the list of active eggs is 
>maintained in easy-install.pth.  Also, if I create my own .pth file, 
>and the user tries to update my version to a new one, will the 
>easy_install tool modify my .pth file to remove the mention of the 
>old version from my sys.path and put the new version in the same 
>.pth file?  Or will it now be listed in both places?  Or will it 
>only in easy-install.pth?

My understanding of the context of the question was that it applied 
to *system* packaging tools, which would be exclusively maintaining 
the .pth entries for the packages they installed.  i.e., a scenario 
with *no* easy-install.pth.  Setuptools will still detect the 
presence of their eggs, regardless of the means by which they're 
added to sys.path.  But it would not *maintain* those .pth files.


>Yes, but as you've already pointed out, they've escaped into a 
>larger ecosystem and this restriction is a severe limitation -- 
>leading to significant frustration.  Especially as projects evolve 
>and want to do something more complex than simply install pure 
>Python code.  Here at Enthought, we use and ship a number of 
>projects that have extensions and thus dynamic libraries that need 
>to either be modified during installation to work from the user's 
>installed location, or copied elsewhere on the system to avoid the 
>need to modify (which we also can't do via an egg install) env 
>variables, registries, etc.

By the way, there *is* experimental shared library building support 
in setuptools, and I recently heard from Andi Vajda that he was 
successful in using it in his JCC project to make available a C++ 
library for linkage from JCC-built projects.  (I'm also sitting on 
his patch that makes it work...)  I'm not sure that it actually fixes 
the larger problem, in that e.g., if the main project is installed by 
the system, and then you build or install an egg yourself.  But I 
think those problems are solvable.


>    We'd also love to be able to ship end-user enterprise-scale 
> applications via eggs so that bug fixes and updates don't require 
> downloading a monolithic 100MB+ installer.  But doing that requires 
> the ability to update desktop icons, menus, etc. which we also 
> can't do automatically via an egg.

Yep...  a good post-install mechanism would be handy for wx and 
pywin32 as well.


>If you don't want the burden on setuptools to support, much less 
>track, all these options, then perhaps it could just support 
>automatic execution of a post-install script (and pre-uninstall 
>script if uninstallation ever happens) that allows individual 
>project developers to do what they need to do?  Let the burden of 
>describing how those things happen and how to 
>uninstall/relocate/update them fall to the provider of the projects 
>that do them.

Yeah, that's what I really *don't* want.  I'd like to enable a more 
trustable mechanism than a blindly-executed script.  I'd rather see a 
standard that makes a developer document more, and have to at least 
*convince* the user that their post-install is worthwhile, even if 
the tool then makes it easy to run.

Better still, I'd rather have those post-install parts done in such a 
way that things like icons, menus, manifests, registry stuff, etc., 
have to get explicitly listed instead of being done programatically.


>Also, IIUC, stow only tries to "contain" the hard files.  It puts 
>links in multiple standard locations (for man pages, executables, 
>libraries, etc.)   If setuptools supported these options, I don't 
>think there'd be any discussion here except for things like "how do 
>I extend the set of things the tool supports so that my foo-type 
>files get linked into the standard /os/path/to/foo for the X os?"

Yep.  Having that would be a worthwhile thing, I think.  Discussion 
leading to specs is most welcome.



>I should have read ahead.  This sounds close to what I've been 
>describing except that this leads me to picture a script that 
>prompts for install locations and allows the user to customize the 
>destinations rather than one that assumes everything goes in a 
>standard place.  I'm all for this, and the continuation of the 
>ability to install an egg into a user-environment vs. a system-environment.

+1.


>The only thing missing here is the ability for the installer to 
>automatically run that script so that installation isn't a 
>disjointed, two-step manual process that a user is prone to forgot 
>to complete.

I don't see a problem with a prompting process, backed by a log file 
that records what post-install steps are pending, finished, or 
explicitly rejected by the user.

One possibility, by the way, is that we could overload "extras" for 
this purpose.  Entry points (such as those for scripts) can require 
extras; if extras could mean post-install components like docs or 
icons or what-have-you, then trying to run the script could result in 
an error message telling you you need to "easy_install 
foo_package[icons]" or whatever.


>One of the features of Enthought's Enstaller extension to 
>easy_install was that it looks for a post_install.py script in 
>EGG-INFO and if one is found, runs it.  I would think that getting 
>this into setuptools would be a significant step forward but I 
>believe you previously rejected that idea.   We'll take a stab at 
>creating a patch for you if you're more receptive to that idea 
>now.  Just let me know.

No -- I'm not happy with a straight-up executable hook for 
post-install steps.  My evaluation of the state of PyPI is that I 
don't trust the community to write non-hazardous setup.py files, let 
alone post-install scripts.  There should be a high technical and 
social barrier to including post-install hooks with arbitrary code.

For example, if there was a required separation between installer 
tools and the things they install, such that any post-install 
operation had to be performed strictly by providing some 
human-readable data that will be passed to a separately-installed 
tool, and there was a high social stigma associated with writing your 
own post-install tool, then that might work.

So, for example, if the community creates an icons and menus 
installer tool for the various platforms, and then anybody can use it 
in their project by adding the right data, then the user doesn't have 
to fully trust arbitrary package authors, only the authors of the 
post-install tools.

I'm not saying that model is perfect; in fact I can see some 
potential pitfalls.  But once an automatic post-install hole is 
opened it will be *very* hard to close, because it will always be 
*easier* to roll your own crappy post-installer instead of 
contributing to a set of robust cross-project/cross-platform 
tools.  So I'd rather keep this particular "itch" in play and try to 
build up the scratching pressure until some people get together and 
pay attention long enough to solve the problem in a less hacky way.  :)


>>On the other hand, I've been puzzling over how to handle legitimate
>>post-install features.  On Windows, both wx and pywin32 have a real
>>need to do some actuall "install" operations.  Some is just copying
>>files, but pywin32 also has to do some registry stuff.  I don't know
>>how to allow just what's sensible, without opening up a huge can of
>>worms, though.
>>
>
>I think there are lots of situations that are legitimate (projects 
>with extensions, projects that want to put icons on the desktop or 
>in menus, projects that need to interact with a registry, projects 
>that want to put configuration information somewhere other than in a 
>zip file in a site-packages dir, etc.)   I think we should worry 
>less about preventing developers from shooting themselves in the foot

It's the users' feet that I'm concerned with.  Some people are 
already paranoid about the fact that PyPI doesn't use SSL and code 
signing, or that easy_install uses the intarwebs at all.  I can just 
see the witch hunt when we start executing arbitrary code.  Unh 
unh.  No way am I letting that happen.  Nope.


>  and more about ensuring that they can hunt for food for their survival.

Right now, if you have a post-install script that's essential, you'll 
just have to convince your users to run it.  Which nicely keeps 
easy_install out of what should be a conversation between developer and user.

Enstaller is a different case - you are presumably installing an 
application, and the user is trusting your installer.  easy_install 
is something else altogether, and is used by other programs such as buildout.

Actually, I wonder if instead of trying to enhance setuptools for 
post-install, if maybe we should be looking at buildout recipes and 
maybe having a way for setuptools dependencies to point to buildout 
specs.  IIRC, buildout specs can be remotely retrieved from a single URL, too.


>    We can always tighten things down after seeing the usecases that 
> develop, right?

Actually, no, we can't, since backward compatibility would keep us 
from removing the hook, once people rely on it.

I really feel yours (and others) pain on this issue, but it's one 
place where the users have to come first, and they need protection 
from the wilds of PyPI.  Distribution and installation issues are not 
first on most developers' minds, so the fact that someone writes a 
great library on PyPI doesn't mean they can write installers worth a 
crap.  Frankly, I wouldn't trust myself to write a correct 
post-installer on the first go -- perhaps *because* I have seen so 
many "simple" things go wrong.


From terry at jon.es  Wed Mar 19 04:03:26 2008
From: terry at jon.es (Terry Jones)
Date: Wed, 19 Mar 2008 04:03:26 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: Your message at 12:34:26 on Tuesday, 18 March 2008
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
Message-ID: <18400.33406.254201.374962@jon.es>

>>>>> "Phillip" == Phillip J Eby <pje at telecommunity.com> writes:
Phillip> At 06:27 AM 3/18/2008 +0100, Terry Jones wrote:
[snip]
>> If so, how can I find where the thing(s) I installed now resides?
[snip]

Phillip> It'd be on the 'install_lib' command instance, not the
Phillip> distribution.  Try
Phillip> dist.get_finalized_command('install_lib').install_dir instead.

For the record / archives, I think this should be

  dist.get_command_obj('install_lib').install_dir

Thanks again,
Terry

From pje at telecommunity.com  Wed Mar 19 05:55:39 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 00:55:39 -0400
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: <18400.33406.254201.374962@jon.es>
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
	<18400.33406.254201.374962@jon.es>
Message-ID: <20080319045541.7E9263A4074@sparrow.telecommunity.com>

At 04:03 AM 3/19/2008 +0100, Terry Jones wrote:
> >>>>> "Phillip" == Phillip J Eby <pje at telecommunity.com> writes:
>Phillip> At 06:27 AM 3/18/2008 +0100, Terry Jones wrote:
>[snip]
> >> If so, how can I find where the thing(s) I installed now resides?
>[snip]
>
>Phillip> It'd be on the 'install_lib' command instance, not the
>Phillip> distribution.  Try
>Phillip> dist.get_finalized_command('install_lib').install_dir instead.
>
>For the record / archives, I think this should be
>
>   dist.get_command_obj('install_lib').install_dir

No, it shouldn't.  Check again.  :)


From terry at jon.es  Wed Mar 19 07:00:58 2008
From: terry at jon.es (Terry Jones)
Date: Wed, 19 Mar 2008 07:00:58 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: Your message at 00:55:39 on Wednesday, 19 March 2008
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
	<18400.33406.254201.374962@jon.es>
	<20080319045541.7E9263A4074@sparrow.telecommunity.com>
Message-ID: <18400.44058.555396.751733@jon.es>

>>>>> "Phillip" == Phillip J Eby <pje at telecommunity.com> writes:

Phillip> At 04:03 AM 3/19/2008 +0100, Terry Jones wrote:
>> >>>>> "Phillip" == Phillip J Eby <pje at telecommunity.com> writes:
Phillip> At 06:27 AM 3/18/2008 +0100, Terry Jones wrote:
>> >> If so, how can I find where the thing(s) I installed now resides?
>> 
Phillip> It'd be on the 'install_lib' command instance, not the
Phillip> distribution.  Try
Phillip> dist.get_finalized_command('install_lib').install_dir instead.
>> 
>> For the record / archives, I think this should be
>> dist.get_command_obj('install_lib').install_dir

Phillip> No, it shouldn't.  Check again.  :)

I get this (using the simplistic direct invocation of run_setup):

    $ python
    Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) 
    [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from distutils.core import run_setup
    >>> d = run_setup('setup.py', ['-q', 'install'])
    >>> d.get_finalized_command('install_lib').install_dir
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: Distribution instance has no attribute 'get_finalized_command'
    >>> d.get_command_obj('install_lib').install_dir
    '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/'

I guess I'm missing something here?

Terry

From jeff at taupro.com  Wed Mar 19 09:57:21 2008
From: jeff at taupro.com (Jeff Rush)
Date: Wed, 19 Mar 2008 03:57:21 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080318004718.56E173A40FF@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
Message-ID: <47E0D571.2070004@taupro.com>

Phillip J. Eby wrote:
> 
> I'm actually happy to hear that there's this much energy available -- 
> hopefully some of it can be harnessed towards positive solutions.
> 
> When I began developing setuptools, I often asked for the input of 
> packagers, developers, etc., through the distutils-sig...  and was met 
> with overwhelming silence.  So the fact that there is now a group of 
> people who are ready to work for some solutions seems like a positive 
> change, to me.

I can appreciate how frustrating silence is when you call for input.  Let's 
see if we can keep the volunteer energy going this time around.


> It's hard to make design decisions regarding itches you don't personally 
> have, and which other people won't help scratch.  Unfortunately, a lot 
> of the proposals from packaging system people have been of the form of, 
> "fix this for us by breaking things for other people".  Not all of them, 
> though.  Many have been very helpful, contributing troubleshooting help 
> and good patches.
> 
> That some of those good patches took nearly a year to get into 
> setuptools (some from Fedora just got into 0.6c8 that were sent to me 
> almost a year ago) is because I'm the only person reviewing setuptools 
> patches, and I've spent only a few days in the last year doing focused 
> development work on setuptools (as opposed to answering questions about 
> it  on the SIG).
> 
> It's never a good thing when people's patches sit around, regardless of 
> where they come from.  But that's not the same thing as *rejecting* the 
> patches.

I and others appreciate your call for more patches on various topics.  However 
a long delay in applying them will discourage contribution.  Are you open to 
giving certain others patch view/commit privileges to setuptools?  I'd be 
willing to help out, and keep a carefully balanced hand in what is accepted.

-Jeff

From jeff at taupro.com  Wed Mar 19 10:10:27 2008
From: jeff at taupro.com (Jeff Rush)
Date: Wed, 19 Mar 2008 04:10:27 -0500
Subject: [Distutils] The Breaking of distutils and PyPI for Python 3000?
Message-ID: <47E0D883.9030402@taupro.com>

As I'm digging into packaging issues here at PyCon, a couple of Python 3000 
related matters occur to me.  As I'm new to the Python 3000 development, if 
these have already been addressed in prior discussions, I apologize for your time.

1. What is the plan for PyPI when Python 3.0 comes out and
    dependencies start getting satisfied from distribution
    across the great divide, e.g. a 3.0-specific package
    pulls from PyPI a 2.x-specific package to meet some
    need?  Are there plans to fork PyPI, apply special
    tags to uploads or what?  While binary distributions
    are tagged with the Python version, source distributions
    are not.  And of course a dependency expression as
    it stands today for "SomePackage > 2.4" may pull 3.0
    to satisfy it.

2. There have been attempts over the years to fix distutils,
    with the last one being in 2006 by Anthony Baxter. He
    stated that a major hurdle was the strong demand to
    respect backward compatibility and he finally gave up.
    One of the purposes of Python 3.0 was the freedom to
    break backward compatibility for the sake of "doing
    the right thing".  So is it now permissible to give
    distutils a good reworking and stop letting
    compatibility issues hold us back?

-Jeff

From p.f.moore at gmail.com  Wed Mar 19 10:43:16 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 19 Mar 2008 09:43:16 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080319000559.C80223A4074@sparrow.telecommunity.com>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
	<20080319000559.C80223A4074@sparrow.telecommunity.com>
Message-ID: <79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.com>

On 19/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
>  >I don't particularly want to use undocumented functions - even within
>  >the module that defines them.
>
> Er, you could always document them, you know.  I just didn't get
>  around to it before the whole "setuptools flap of 2005" wiped out my
>  motivation to do any further work on Python 2.5.  For the most part,
>  they do have docstrings.  I just never did the LaTeX work on
>  them.  And now, you can use reST instead of LaTeX.  :)

I could, but the problem is I don't really follow the code. My
motivation is to add useful functions, not document stuff that's
already there. You mentioned using get_loader to implement
resource_string. OK, but I'd have done something a lot simpler than
the code of get_loader, and I don't understand why the code in there
is necessary.

Never mind, I'll implement what I'm planning on using my own code, and
ignore trying to understand the (corner cases of the) undocumented
functions. Looking at the pkg_resources code, I'd have had the same
issues there trying to disentangle the "simple bits" from the
infrastructure, as you suggested, so that's probably a better approach
anyway.

Paul.

From p.f.moore at gmail.com  Wed Mar 19 10:52:19 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 19 Mar 2008 09:52:19 +0000
Subject: [Distutils] The Breaking of distutils and PyPI for Python 3000?
In-Reply-To: <47E0D883.9030402@taupro.com>
References: <47E0D883.9030402@taupro.com>
Message-ID: <79990c6b0803190252w566d81n971a81df616c8cae@mail.gmail.com>

On 19/03/2008, Jeff Rush <jeff at taupro.com> wrote:
>  1. What is the plan for PyPI when Python 3.0 comes out and
>     dependencies start getting satisfied from distribution
>     across the great divide, e.g. a 3.0-specific package
>     pulls from PyPI a 2.x-specific package to meet some
>     need?

As distutils (and core Python) doesn't do any automatic dependency
management, this is a setuptools issue. As such, it's up to setuptools
to deal with it. There may be infrastructure changes that would be
generally useful, but there's nothing *needed* for the core.

>  2. There have been attempts over the years to fix distutils,
>     with the last one being in 2006 by Anthony Baxter. He
>     stated that a major hurdle was the strong demand to
>     respect backward compatibility and he finally gave up.
>     One of the purposes of Python 3.0 was the freedom to
>     break backward compatibility for the sake of "doing
>     the right thing".  So is it now permissible to give
>     distutils a good reworking and stop letting
>     compatibility issues hold us back?

Sounds reasonable. I'm sure patches would be considered, but past
discussions around "including setuptools" have been controversial and
generally not reached consensus (for reasons other than pure backward
compatibility). Also, while compatibility isn't as important for 3.0,
smooth migration *is* - so any incompatible proposal must include some
consideration of how to assist people with huge, complex setup.py
files which use distutils internals in complex ways. So be prepared to
do some work :-)

(But I'd be happy to see distutils improved. I just don't have any
need for such improvement, personally).

Paul.

From barry at python.org  Wed Mar 19 15:00:10 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 19 Mar 2008 09:00:10 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
	Packaging/Deployment Technology Concerns
In-Reply-To: <47E0D571.2070004@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E0D571.2070004@taupro.com>
Message-ID: <CFF57303-0AD8-4B11-BB62-03A9D5A346FB@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mar 19, 2008, at 3:57 AM, Jeff Rush wrote:
>
> I and others appreciate your call for more patches on various  
> topics.  However
> a long delay in applying them will discourage contribution.  Are you  
> open to
> giving certain others patch view/commit privileges to setuptools?   
> I'd be
> willing to help out, and keep a carefully balanced hand in what is  
> accepted.

The Python sandbox has a setuptools directory.  Is this the canonical  
location for the code?  If so, then anybody who has Python commit  
privileges can commit to it and help further develop setuptools.

If not, why not and what is the sandbox setuptools used for?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iQCVAwUBR+Eca3EjvBPtnXfVAQLabwP9F8NtQX6YsDXJMHiByCGILPAQ2NgtaIzg
en6yYbhl5IAweTr0DtWzxRXjSGMifK/D4PmtRSWWUTy3VY+8cRUkYuBjIxPOHJRF
4TA4dYoW4f2+qM1IO/l59FIAJgUyrXKhv3aznpXBFl+PaRCW9qP9G1lur3lolipB
h4i8ya+I7zU=
=2/iq
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Wed Mar 19 16:21:00 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 11:21:00 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47E0D571.2070004@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E0D571.2070004@taupro.com>
Message-ID: <20080319152101.730BC3A40A5@sparrow.telecommunity.com>

At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote:
>Are you open to giving certain others patch view/commit privileges 
>to setuptools?

Jim Fulton has such already.  I'm open to extending that to others 
who have a good grasp of the subtleties involved.

Truthfully, if we can just get 0.6 put to bed, I could probably open 
up the trunk a lot wider.

One of the things that slows me down is that patches usually don't 
come with tests, so I usually have to manually smoke-test them for 
scenarios I think they'll effect.  There isn't really any automated procedure.

Probably the most frustrating thing (or "chief amongst the most 
frustrating things") about setuptools development is that it's a 
black hole.  By which I mean that backward compatibility and cruft 
accretion make it difficult to get out of.

In the beginning, there was the distutils.  Distutils begat 
setuptools, and setuptools begat virtualenv and zc.buildout and 
source control plugins.  Etc., etc.

What I think is really needed in the long run is to keep eggs, but 
get rid of setuptools and the distutils in their current 
form.  There's a lot of brokenness there, and also a lot of 
accumulated cruft.  We really need a distutils 3000, and it needs to 
be built on a better approach.

In truth, my *real* motivation for PEP 365's bootstrap tool isn't so 
much to support the package management tools we have today, as it is 
to support a new one tomorrow.  I have a few ideas for ways to shift 
the paradigm of how individual projects get built, to incorporate 
many scenarios that don't work well now.  But to implement those 
things in such a next-generation tool, I will not want to be 
restricted to just what's in the stdlib or what can be bundled in the tool.

(Btw, by "real" motivation, I don't mean I've been deceptive about my 
intentions, I mean that my strong intuition that such a bootstrap 
facility is needed, is probably being fueled by the long term desire 
to replace the entire distutils-based infrastructure with something better.)


>   I'd be willing to help out, and keep a carefully balanced hand in 
> what is accepted.

And I think it's probably getting close to time I stepped down from 
day-to-day management of the codebase (which is more like 
month-to-month or quarter-to-quarter for me lately).  It will 
probably be a lot easier for me to step back and critique stuff that 
goes in, after the fact, than to go over the stuff beforehand.  :)

I'm not sure exactly how to go about such a handoff though.  My guess 
is that we need a bug/patch tracker, and a few people to review, 
test, and apply.  Maybe a transitional period during which I just say 
yea or nay and let others do the test and apply, before opening it up 
entirely.  That way, we can perhaps solidify a few principles that 
I'd like to have stay in place.  (Like no arbitrary post-install code hooks.)

btw, offtopic question: are you by any chance the same Jeff Rush who 
invented EchoMail?


From pje at telecommunity.com  Wed Mar 19 16:53:12 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 11:53:12 -0400
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: <18400.44058.555396.751733@jon.es>
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
	<18400.33406.254201.374962@jon.es>
	<20080319045541.7E9263A4074@sparrow.telecommunity.com>
	<18400.44058.555396.751733@jon.es>
Message-ID: <20080319155315.D0E9F3A40A5@sparrow.telecommunity.com>

At 07:00 AM 3/19/2008 +0100, Terry Jones wrote:
> >>>>> "Phillip" == Phillip J Eby <pje at telecommunity.com> writes:
>Phillip> It'd be on the 'install_lib' command instance, not the
>Phillip> distribution.  Try
>Phillip> dist.get_finalized_command('install_lib').install_dir instead.
> >>
> >> For the record / archives, I think this should be
> >> dist.get_command_obj('install_lib').install_dir
>
>Phillip> No, it shouldn't.  Check again.  :)
>
>I get this (using the simplistic direct invocation of run_setup):
>
>     $ python
>     Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
>     [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> from distutils.core import run_setup
>     >>> d = run_setup('setup.py', ['-q', 'install'])
>     >>> d.get_finalized_command('install_lib').install_dir
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     AttributeError: Distribution instance has no attribute 
> 'get_finalized_command'
>     >>> d.get_command_obj('install_lib').install_dir
> 
>'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/'
>
>I guess I'm missing something here?

Whoops, my bad.  'get_finalized_command' is a distutils *command 
object* method, not a distribution method.  I usually work with code 
in command objects much more than with distribution objects.

The difference, though, between get_finalized_command() and 
get_command_obj() is that the latter is not guaranteed to return a 
*finalized* command, i.e., one that has processed its config file and 
command-line options.  I would personally call .ensure_finalized() on 
the command object before checking its install_dir.  (This is done 
automatically by cmd.get_finalized_command(), which is why I usually 
use it, hence the confusion.)


From pje at telecommunity.com  Wed Mar 19 17:00:02 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 12:00:02 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
 module)
In-Reply-To: <79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.co
 m>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
	<20080319000559.C80223A4074@sparrow.telecommunity.com>
	<79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.com>
Message-ID: <20080319160002.74BC03A40A5@sparrow.telecommunity.com>

At 09:43 AM 3/19/2008 +0000, Paul Moore wrote:
>On 19/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> >  >I don't particularly want to use undocumented functions - even within
> >  >the module that defines them.
> >
> > Er, you could always document them, you know.  I just didn't get
> >  around to it before the whole "setuptools flap of 2005" wiped out my
> >  motivation to do any further work on Python 2.5.  For the most part,
> >  they do have docstrings.  I just never did the LaTeX work on
> >  them.  And now, you can use reST instead of LaTeX.  :)
>
>I could, but the problem is I don't really follow the code. My
>motivation is to add useful functions, not document stuff that's
>already there. You mentioned using get_loader to implement
>resource_string. OK, but I'd have done something a lot simpler than
>the code of get_loader, and I don't understand why the code in there
>is necessary.

It's so that you can get loaders for modules that aren't imported yet 
-- and your code would need to handle this case too.  (You could 
handle it by actually going ahead and importing the module, as 
pkg_resources does, but there are other tools using pkgutil to e.g. 
inspect as-yet-unimported modules.)


>Never mind, I'll implement what I'm planning on using my own code, and
>ignore trying to understand the (corner cases of the) undocumented
>functions.

You don't really need to, because even though they're technically 
"undocumented", the intent was for them to be published APIs (apart 
from simplegeneric, which is an implementation detail).  The docs 
just never got copied to the official docs.  These APIs are also 
actively used by at least pydoc and pyrun in Python 2.5, so they're 
unlikely to go away or break.


From p.f.moore at gmail.com  Wed Mar 19 18:04:24 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 19 Mar 2008 17:04:24 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080319160002.74BC03A40A5@sparrow.telecommunity.com>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
	<20080319000559.C80223A4074@sparrow.telecommunity.com>
	<79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.com>
	<20080319160002.74BC03A40A5@sparrow.telecommunity.com>
Message-ID: <79990c6b0803191004v5df2f25r57194214d657cf3d@mail.gmail.com>

On 19/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> It's so that you can get loaders for modules that aren't imported yet
>  -- and your code would need to handle this case too.

Well, technically, it need not, as I don't *need* to implement the
exact functionality that pkg_resources does. My (personal) goal is to
standardise the functionality, not to cater for every possible use
case.

>  (You could handle it by actually going ahead and importing the module, as
>  pkg_resources does, but there are other tools using pkgutil to e.g.
>  inspect as-yet-unimported modules.)

I'm not sure what you mean by that. There are no tools using
pkgutil.resource_stream, as it doesn't exist yet. So whether it
imports the module or not cannot be relevant (although it should be
documented).

> You don't really need to, because even though they're technically
>  "undocumented", the intent was for them to be published APIs (apart
>  from simplegeneric, which is an implementation detail).  The docs
>  just never got copied to the official docs.

If it's just a matter of copying documentation from the PkgResources
page, that's no issue. However, only get_importer is documented on the
PkgResources page.

On reviewing the pkg_resources APIs, I only see the following that
look worth porting to pkgutil:

    * resource_exists(package, resource_name)
    * resource_stream(package, resource_name)
    * resource_string(package, resource_name)
    * resource_isdir(package, resource_name)
    * resource_listdir(package, resource_name)

For all of these, the first argument would be restricted to a
package/module name. The option of specifying a "requirement" is not
suitable. Also, automatic extraction and resource_filename is far
beyond what I see as sensible in the stdlib.

Hmm, these go way beyond the simple (and optional) loader.get_data
interface from PEP 302. And yet the pkg_resources machinery is far
more complex than I'd want to see in the stdlib. Maybe I'll just
reinvent the wheel and see how far I get :-)

Paul.

From doko at cs.tu-berlin.de  Wed Mar 19 18:05:42 2008
From: doko at cs.tu-berlin.de (Matthias Klose)
Date: Wed, 19 Mar 2008 18:05:42 +0100
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
	Packaging/Deployment	Technology Concerns
In-Reply-To: <47DEEC6B.5040602@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
Message-ID: <18401.18406.521359.387045@gargle.gargle.HOWL>

Jeff Rush writes:
> I was in a Packaging BoF yesterday and, although not very relevant to the 
> packager bootstrap thread, Guido has asked me to post some of the concerns.

We did address many topics on both days, I added the following topics
which were addressed on the Friday BoF only, see
http://wiki.python.org/moin/PackagingBOF

- Linux distributions try to ship only one version of a
  package/egg/module in one release, only shipping more than one
  version if necessary. eggs (as least as shipped with Debian, Fedora,
  Ubuntu) are all built using --single-version-externally-managed.

   - import foo should work wether installed as an egg or installed
     with distutils, and without using pkg_resources.require
   - pkg_resources should handle the situation of one egg version
     installed as --single-version-externally-managed (default version) and
     one or more eggs installed not using
     --single-version-externally-managed. Currently these additional
     versions cannot be imported. 

- It would be useful if setuptools could handle separate build and
  install steps like most configure/make/make install systems do. Access
  to external resources should optionally be disabled during a build.

- The idea was brought up to use a to-be-defined api-version to
  describe dependencies between eggs. Version numbers are generally used
  for more than api changes; the idea follows existing practice for
  shared object names, only changing when the API is changed.

From guido at python.org  Wed Mar 19 18:25:22 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 19 Mar 2008 10:25:22 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803170648j2847d10cxc3c2eaf48bc43e21@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEC0EB.3000404@palladion.com>
	<440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com>
Message-ID: <ca471dc20803191025i7ea5af15wc0bdcc4a47632720@mail.gmail.com>

On Mon, Mar 17, 2008 at 2:19 PM, zooko <zooko at zooko.com> wrote:
>  4.  The standard Python library includes a tool to find and read
>  resources (other than Python modules) that came bundled in a Python
>  package.
>
>  Consider, for example, this snippets of code in Nevow:
>
>  http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786#L10
>  http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786
>  http://divmod.org/trac/browser/trunk/Nevow/setup_egg.py?rev=2406
>
>  When Nevow uses pkg_resources to import its files such as
>  "default.css", then it is able to find at runtime, even if is being
>  imported from a py2exe or py2app zip, or on other platforms where its
>  homegrown setup script and homegrown "find my file" function fail.
>  So using pkg_resources (and setuptools to install it) makes
>  "test_nevow" pass on all of the allmydata.org buildslaves:
>
>  http://allmydata.org/buildbot/waterfall?show_events=false

I think we're pretty close to this already. PEP 302 defines a
getdata() method. Hopefully most PEP 302 implementations support it.
The only thing missing IMO is a little function that does what
getdata() does when there is no __loader__ object (i.e. when the
default "import-from-filesystem" import method is used).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From lxander.m at gmail.com  Wed Mar 19 18:26:17 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Wed, 19 Mar 2008 13:26:17 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
	Packaging/Deployment Technology Concerns
In-Reply-To: <18401.18406.521359.387045@gargle.gargle.HOWL>
References: <47DEEC6B.5040602@taupro.com>
	<18401.18406.521359.387045@gargle.gargle.HOWL>
Message-ID: <525f23e80803191026t41ce1961tf813a4022b262e32@mail.gmail.com>

On Wed, Mar 19, 2008 at 1:05 PM, Matthias Klose <doko at cs.tu-berlin.de> wrote:
>  - It would be useful if setuptools could handle separate build and
>   install steps like most configure/make/make install systems do. Access
>   to external resources should optionally be disabled during a build.

What's wrong with "python setup.py bdist_egg"?

From doko at cs.tu-berlin.de  Wed Mar 19 18:37:37 2008
From: doko at cs.tu-berlin.de (Matthias Klose)
Date: Wed, 19 Mar 2008 18:37:37 +0100
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080318004718.56E173A40FF@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
Message-ID: <18401.20321.441120.809380@gargle.gargle.HOWL>

Phillip J. Eby writes:
> >7. Many wanted to ability to install files anywhere in the install tree and
> >     not just under the Python package.  Under distutils this was possible but
> >     it was removed in setuptools for security reasons.
> 
> It wasn't security, it was manageability.  Egg-based installation 
> means containment, (analagous to GNU stow) and therefore portability 
> and disposability of plugins.  (Which again is what eggs were really 
> developed for in the first place.)

defining containment this way doesn't help when preparing eggs for
inclusion in a linux distribution.  E.g. users on these distributions
are used to find log files in /var/log (maybe in a subdir),
documentation in /usr/share/doc/<package name>.  You probably will get
different views about manageability depending on your background (used
to linux distribution standards or used to standards set by
setuptools/cheeseshop).  Packagers currently move these files manually
to the standard locations and often have to keep symlinks in the egg
dirs to these locations.  Installation on linux distributions is
handled by existing package tools which is unlikely to change.  So it
would be nice to find a common layer which can be used for both
distribution methods, optionally enabling this with some kind of
option like --install-files-in-places-not-handled-by-setuptools ;)

  Matthias

From guido at python.org  Wed Mar 19 18:48:02 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 19 Mar 2008 10:48:02 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080318223700.C64123A4074@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
Message-ID: <ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>

On Tue, Mar 18, 2008 at 3:36 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote:
>  >Only very few people would care about writing a setup
>  >script that works with this bootstrap module; basically only package
>  >manager implementers.
>
>  That's true today, sure, but as soon as it is widely available,
>  others are sure to want to use it too.  I just want a bright-line
>  distinction between what is and isn't bootstrappable, rather than a
>  murky region of "maybe, if you're not doing anything too complicated".

How about "anything that uses only distutils in its setup.py and
doesn't have external dependencies"? See a (horribly incomplete)
prototype I added as sandbox/bootstrap/bootstrap.py. I wrote this on
the plane last night and have only tested it with file:/// URLs; it
needs to add the ability to consult PyPI to find the download URL, and
probably more. (PS: just now I also managed to successfully install
setuptools from source by giving it the URL to the gar.gz file.)

>  >There seems to be a misunderstanding about what I am proposing we do
>  >instead. The boostrap installer should only be powerful enough to
>  >allow it to be used to install a real package manager like setuptools.
>
>  Which is why PEP 365 proposed only downloading an archive to a cache
>  directory, and optionally running something from it.  It explicitly
>  disavows "installation" of anything, since the downloaded archive
>  wouldn't have been added to sys.path except for the duration of the
>  bootstrap process, and no scripts were to be installed.  (Indeed,
>  apart from the methods it would have used to locate the archive on
>  PyPI, and to determine what to run from inside it, there was nothing
>  particularly egg-specific about the proposed bootstrapping process.)

My bootstrap.py does exactly that: it downloads and unzips/untars a
file and runs its setup.py with "install" as the only command line
argument. (It currently looks for setup.py at the toplevel and one
level deep in the unpacked archive.) Of course you will likely have to
be root or administrator to run it effectively.

>  So, to fully egg-neutralize the bootstrapping approach, we need only
>  know how to locate an appropriate archive, and how to determine what
>  to run from it.

Right.

>  For the latter, we could use the already-in-2.6 convention of running
>  __main__ from a zipfile or directory.  (Too bad distutils source
>  distributions have an extra directory name embedded in them, so one
>  can't just execute them directly.  Otherwise, we could've just let
>  people drop in a __main__.py next to setup.py.  OTOH, maybe it would
>  be enough to use setuptools' algorithm for finding setup.py to locate
>  __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.)

What's wrong with just running "setup.py install"? I'd rather continue
existing standards / conventions. Of course, it won't work when
setup.py requires setuptools; but "old style" setup.py files that use
only distutils work great (I managed to install Django from a file:///
URL).

>  The other open question is a naming convention and version detection,
>  so that the bootstrap tool can identify which of the files listed on
>  PyPI is suitable for its use.  (Both with regard to the version
>  selection, and file type.)  However, if PyPI were to grow support for
>  designating the appropriate files and/or versions in some other way,
>  we wouldn't need a naming convention as such.

I don't understand PyPI all that well; it seems poor design that the
browsing via keywords is emphasized but there is no easy way to
*search* for a keyword (the list of all packages is not emphasized
enough on the main page -- it occurs in the side bar but not in the
main text). I assume there's a programmatic API (XML-RPC?) but I
haven't found it yet.

>  Without one or the other, the bootstrap tool would have to grow a
>  version parsing scheme of some type, and play guessing games with
>  file extensions.  (Which is one reason I limited PEP 365's scope to
>  downloading eggs actually *uploaded* to PyPI, rather than arbitrary
>  packages *linked* from PyPI.)

There are two version parsers in distutils, referenced by PEP 345, the
PyPI 1.2 metadata standard.

>  So, if I had to propose something right now, I would be inclined to propose:
>
>  * using setuptools' version parsing semantics for interpretation of
>  alpha/beta/dev/etc. releases

Can you point me to the code for this? What is its advantage over
distutils.version?

>  * having a bdist_bootstrap format that's essentially a bdist_dumb
>  .zip file with the internal path prefixes stripped off, making it an
>  importable .zip with a different file extension.  (Or maybe just
>  .pyboot.zip?)  The filename convention would use setuptools'
>  canonicalization and escaping of names and version numbers, to allow
>  unambiguous machine parsing of the filename.  A __main__ module would
>  have to be present for the archive to be run, as opposed to just
>  being downloaded to a temporary directory.

Hm. Why not just use the existing convention for running setup.py
after unpacking? This works great in my experience, and has the
advantage of having an easy fallback if you end up having to do this
manually for whatever reason.

>  * calling the bootstrap module 'bootstrap', as in 'python -m
>  bootstrap projectname optionalversion'.  The module would expose an
>  API to allow it to be used programmatically as well as the command
>  line, so that bootstrapped packages can use the bootstrap process to
>  locate dependencies if they so desire.  (Today's package management
>  tools, at least, are all based on setuptools, so if it's not present
>  they'll need to download that before beginning their own
>  bootstrapping process.)

This sounds like going beyond bootstrapping. My vision is that you use
the bootstrap module (with the command line you suggest above) once to
install setuptools or the alternate package manager of your choice,
and then you can use easy_install (or whatever alternative) to install
the rest.

>  Apart from keeping the PEP self-contained and short, is there
>  anything in this that you think you would object to?  (You may
>  reserve the right, of course, to later not like something in the
>  details of setuptools' version/filename rules, after I've put them
>  into the PEP, or really, anything else.  I'm just asking if there's
>  anything that's obviously offensive at this point, before I spend
>  time on a new PEP.)

I'd love it if you could write or point me to code that takes a
package name and optional version and returns the URL for the source
archive, and the type (in case it can't be guessed from the filename
or the Content-type header).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From p.f.moore at gmail.com  Wed Mar 19 19:34:02 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 19 Mar 2008 18:34:02 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803191025i7ea5af15wc0bdcc4a47632720@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317142225.7C7D83A40B1@sparrow.telecommunity.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEC0EB.3000404@palladion.com>
	<440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com>
	<ca471dc20803191025i7ea5af15wc0bdcc4a47632720@mail.gmail.com>
Message-ID: <79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com>

On 19/03/2008, Guido van Rossum <guido at python.org> wrote:
> On Mon, Mar 17, 2008 at 2:19 PM, zooko <zooko at zooko.com> wrote:
>  >  4.  The standard Python library includes a tool to find and read
>  >  resources (other than Python modules) that came bundled in a Python
>  >  package.
>
> I think we're pretty close to this already. PEP 302 defines a
>  getdata() method. Hopefully most PEP 302 implementations support it.
>  The only thing missing IMO is a little function that does what
>  getdata() does when there is no __loader__ object (i.e. when the
>  default "import-from-filesystem" import method is used).

I'm currently working on an addition to pkgutil to provide this type
of function. I'm considering going a little further (adding functions
to get a file-like object, test for existence, and list available
resources, modelled on the pkg_resources functions - but these extra
ones are not supported by the loader protocol, so I'm undecided as to
whether it's worth it, I'll see how complex the code gets).

Once I have a patch, I'll post it to the tracker. What's the best
approach? Code a patch for 3.0 and backport, or code for 2.6 and let
the merging process do its stuff?

Paul.

From pje at telecommunity.com  Wed Mar 19 19:35:54 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 14:35:54 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
 module)
In-Reply-To: <79990c6b0803191004v5df2f25r57194214d657cf3d@mail.gmail.com
 >
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
	<20080319000559.C80223A4074@sparrow.telecommunity.com>
	<79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.com>
	<20080319160002.74BC03A40A5@sparrow.telecommunity.com>
	<79990c6b0803191004v5df2f25r57194214d657cf3d@mail.gmail.com>
Message-ID: <20080319183600.67B953A40A5@sparrow.telecommunity.com>

At 05:04 PM 3/19/2008 +0000, Paul Moore wrote:
>On 19/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> > It's so that you can get loaders for modules that aren't imported yet
> >  -- and your code would need to handle this case too.
>
>Well, technically, it need not, as I don't *need* to implement the
>exact functionality that pkg_resources does. My (personal) goal is to
>standardise the functionality, not to cater for every possible use
>case.
>
> >  (You could handle it by actually going ahead and importing the module, as
> >  pkg_resources does, but there are other tools using pkgutil to e.g.
> >  inspect as-yet-unimported modules.)
>
>I'm not sure what you mean by that. There are no tools using
>pkgutil.resource_stream, as it doesn't exist yet. So whether it
>imports the module or not cannot be relevant (although it should be
>documented).

You asked why pkgutil.get_loader() was more complex than you'd 
expect.  That's the question I answered, in both of the snippets you 
quoted above.  That is, when I said, "your code would need to handle 
this case", I meant "your code for get_loader()", not "your code for 
resource_foo() methods".

IOW, I was trying to encourage you to use get_loader() as-is rather 
than modifying or replacing it with something "simpler" that would 
break other uses of get_loader() in the stdlib.


> > You don't really need to, because even though they're technically
> >  "undocumented", the intent was for them to be published APIs (apart
> >  from simplegeneric, which is an implementation detail).  The docs
> >  just never got copied to the official docs.
>
>If it's just a matter of copying documentation from the PkgResources
>page, that's no issue. However, only get_importer is documented on the
>PkgResources page.

I meant, the documentation was never copied from the *docstrings* of 
these functions in pkgutil; as you noted, not all of them are direct 
ports from pkg_resources.  I refactored the framework there to 
include support for pyrun and pydoc's use cases as well.

It's true that some additional documentation is needed, too, for 
anyone who wants to implement a PEP 302 loader or importer that wants 
to support enhanced features like listing modules or getting resources.


>On reviewing the pkg_resources APIs, I only see the following that
>look worth porting to pkgutil:
>
>     * resource_exists(package, resource_name)
>     * resource_stream(package, resource_name)
>     * resource_string(package, resource_name)
>     * resource_isdir(package, resource_name)
>     * resource_listdir(package, resource_name)
>
>...
>
>Hmm, these go way beyond the simple (and optional) loader.get_data
>interface from PEP 302. And yet the pkg_resources machinery is far
>more complex than I'd want to see in the stdlib. Maybe I'll just
>reinvent the wheel and see how far I get :-)

So that future PEP 302 importers can define support for these 
features, please use
pkgutil.simplegeneric to implement these, following the pattern set 
out by other pkgutil functions.  E.g.:

     def resource_listdir(package, resource_name):
         return loader_listdir(get_loader(package), package, resource_name)

Where 'loader_listdir' is a simplegeneric with default handlers 
registered for the built-in/stdlib loader types (zipimporter, None, 
and ImpLoader).

If you do this, then new importer/loader types can use 
loader_listdir.register() to support the functionality, without 
monkeypatching or other hacks.


From p.f.moore at gmail.com  Wed Mar 19 19:41:44 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 19 Mar 2008 18:41:44 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080319183600.67B953A40A5@sparrow.telecommunity.com>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
	<20080319000559.C80223A4074@sparrow.telecommunity.com>
	<79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.com>
	<20080319160002.74BC03A40A5@sparrow.telecommunity.com>
	<79990c6b0803191004v5df2f25r57194214d657cf3d@mail.gmail.com>
	<20080319183600.67B953A40A5@sparrow.telecommunity.com>
Message-ID: <79990c6b0803191141h7381de22v3070bf35247371df@mail.gmail.com>

On 19/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> So that future PEP 302 importers can define support for these
>  features, please use
>  pkgutil.simplegeneric to implement these, following the pattern set
>  out by other pkgutil functions.  E.g.:
>
>      def resource_listdir(package, resource_name):
>          return loader_listdir(get_loader(package), package, resource_name)
>
>  Where 'loader_listdir' is a simplegeneric with default handlers
>  registered for the built-in/stdlib loader types (zipimporter, None,
>  and ImpLoader).
>
>  If you do this, then new importer/loader types can use
>  loader_listdir.register() to support the functionality, without
>  monkeypatching or other hacks.

If I did this, I'd certainly make sure it was possible for other
loaders to add their own implementations. But I'm not sure I want to
get involved in the generic-function politics involved in using
pkgutil.simplegeneric - particularly as it's undocumented. I may just
go for a simple registry of implementation classes. Sorry, but if you
couldn't get sufficient agreement to simplegeneric to get it in a
"proper" location, I doubt I'd fare better, (Of course, if Guido pipes
up with support for using simplegeneric, I'll go with that - and
document it, and propose that it gets moved to a sensible location.
But I don't want to be the one to try to persuade Guido...)

Paul

From dpeterson at enthought.com  Wed Mar 19 19:55:47 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Wed, 19 Mar 2008 13:55:47 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080319152101.730BC3A40A5@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>	<20080318004718.56E173A40FF@sparrow.telecommunity.com>	<47E0D571.2070004@taupro.com>
	<20080319152101.730BC3A40A5@sparrow.telecommunity.com>
Message-ID: <47E161B3.9070208@enthought.com>

Phillip J. Eby wrote:
> At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote:
>   
>>   I'd be willing to help out, and keep a carefully balanced hand in 
>> what is accepted.
>>     
>
>
> I'm not sure exactly how to go about such a handoff though.  My guess 
> is that we need a bug/patch tracker, and a few people to review, 
> test, and apply.  Maybe a transitional period during which I just say 
> yea or nay and let others do the test and apply, before opening it up 
> entirely.  That way, we can perhaps solidify a few principles that 
> I'd like to have stay in place.  (Like no arbitrary post-install code hooks.)
>   

+1 to blessing more people to commit.
+1 to the transition period idea.

These two ought to enable things to move a bit quicker than taking a 
year to accept a patch. :-)

In addition to a bug tracker and patch manager, seems like perhaps a 
wiki to help document some of these solidified principles and other 
notes would be a good thing.  (Like a patch should almost always include 
at least one test, possibly more.)  Given that the source for setuptools 
is in the python.org svn, couldn't we just use the python.org roundup 
and wiki for these facilities?  Though looking at the list of 
components, it seems that things in the sandbox generally aren't tracked 
in this infrastructure.  In which case, I'm sure we could use sf, 
launchpad, or some such external provider.  Enthought could even host 
this stuff.

Like Jeff Rush, I'm also willing to help out as both a writer and 
reviewer of patches.  As you can see from my earlier posts there are a 
number of things (besides running an arbitrary post-install script) that 
we'd like to be able to get into the codebase.


-- Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080319/797705e9/attachment-0001.htm 

From guido at python.org  Wed Mar 19 20:06:28 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 19 Mar 2008 12:06:28 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<47DE8405.2030201@v.loewis.de>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEC0EB.3000404@palladion.com>
	<440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com>
	<ca471dc20803191025i7ea5af15wc0bdcc4a47632720@mail.gmail.com>
	<79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com>
Message-ID: <ca471dc20803191206j2eae1be3rc5745f955d8e0dce@mail.gmail.com>

On Wed, Mar 19, 2008 at 11:34 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 19/03/2008, Guido van Rossum <guido at python.org> wrote:
>  > On Mon, Mar 17, 2008 at 2:19 PM, zooko <zooko at zooko.com> wrote:
>  >  >  4.  The standard Python library includes a tool to find and read
>  >  >  resources (other than Python modules) that came bundled in a Python
>  >  >  package.

> > I think we're pretty close to this already. PEP 302 defines a
>  >  getdata() method. Hopefully most PEP 302 implementations support it.
>  >  The only thing missing IMO is a little function that does what
>  >  getdata() does when there is no __loader__ object (i.e. when the
>  >  default "import-from-filesystem" import method is used).
>
>  I'm currently working on an addition to pkgutil to provide this type
>  of function. I'm considering going a little further (adding functions
>  to get a file-like object, test for existence, and list available
>  resources, modelled on the pkg_resources functions - but these extra
>  ones are not supported by the loader protocol, so I'm undecided as to
>  whether it's worth it, I'll see how complex the code gets).

I'd only do what __loader__ offers. People can always wrap a StringIO around it.

>  Once I have a patch, I'll post it to the tracker. What's the best
>  approach? Code a patch for 3.0 and backport, or code for 2.6 and let
>  the merging process do its stuff?

Code for 2.6, let the merge do its thing.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From marius at pov.lt  Wed Mar 19 20:52:57 2008
From: marius at pov.lt (Marius Gedminas)
Date: Wed, 19 Mar 2008 21:52:57 +0200
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <7128D92F-79DB-41CD-A8F5-80F0F0307432@zope.com>
References: <20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<7128D92F-79DB-41CD-A8F5-80F0F0307432@zope.com>
Message-ID: <20080319195257.GD8098@fridge.pov.lt>

On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote:
> On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote:
> ...
> >os.path.join(os.path.dirname(__file__), 'foo') just has too many
> >problems.
> 
> Could you be specific?

I suppose I was thinking about the stale .pyc file problem, mostly.

zc.buildout/setuptools/distutils (I'm not sure which part of the stack
is to blame) likes to compile modules in a temporary directory and then
move the .py and .pyc files to their final location.  As a result,
__file__ points to the now-nonexistent directory.

Aside: I never understood the reason for storing the value of __file__
in the .pyc/.pyo file itself.  Why not set it to the correct value on
import time?

Marius Gedminas
-- 
If you have to run heating in winter, you don't own enough computers.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080319/bd0f4445/attachment.pgp 

From pje at telecommunity.com  Wed Mar 19 20:54:37 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 15:54:37 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
Message-ID: <20080319195438.A28DE3A40A5@sparrow.telecommunity.com>

At 10:48 AM 3/19/2008 -0700, Guido van Rossum wrote:
>I don't understand PyPI all that well; it seems poor design that the
>browsing via keywords is emphasized but there is no easy way to
>*search* for a keyword (the list of all packages is not emphasized
>enough on the main page -- it occurs in the side bar but not in the
>main text). I assume there's a programmatic API (XML-RPC?) but I
>haven't found it yet.

   http://wiki.python.org/moin/CheeseShopXmlRpc

There's also a REST API that setuptools uses:

   http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

The API was originally designed for screen-scraping an older version 
of PyPI, but that has been replaced with a "lite" version served from:

   http://pypi.python.org/simple/

The "lite" version is intended for tools such as easy_install to 
process, as it consists strictly of links and can be statically 
cached.  Zope Corp., for example, maintains a static mirror of this 
API, to guard themselves against PyPI outages and slowdowns, since 
their buildouts can involve huge numbers of eggs, both their own and 
external dependencies.


>I'd love it if you could write or point me to code that takes a
>package name and optional version and returns the URL for the source
>archive, and the type (in case it can't be guessed from the filename
>or the Content-type header).

You can probably do that with the XML-RPC API.  There's a function to 
get the versions of a package, given a (case-sensitive) name, and 
there's a function to get information for uploaded archives, given a 
name and a version.  I originally intended to use it for the PEP 365 
approach, but you can get the necessary information in just one 
static roundtrip using the REST (/simple) HTML API, if you're willing 
to parse the URLs for version information.  (The catch of course 
being that distutils source distributions don't have unambiguously 
parseable filenames.)


>Hm. Why not just use the existing convention for running setup.py
>after unpacking? This works great in my experience, and has the
>advantage of having an easy fallback if you end up having to do this
>manually for whatever reason.

Because I want bootstrap-ees to be able to use the bootstrap 
mechanism.  For example, I expect at some point that setuptools will 
use other, non-self-contained packages, and other package managers 
such as zc.buildout et al also want to depend on setuptools without 
bundling it.


> >  * calling the bootstrap module 'bootstrap', as in 'python -m
> >  bootstrap projectname optionalversion'.  The module would expose an
> >  API to allow it to be used programmatically as well as the command
> >  line, so that bootstrapped packages can use the bootstrap process to
> >  locate dependencies if they so desire.  (Today's package management
> >  tools, at least, are all based on setuptools, so if it's not present
> >  they'll need to download that before beginning their own
> >  bootstrapping process.)
>
>This sounds like going beyond bootstrapping. My vision is that you use
>the bootstrap module (with the command line you suggest above) once to
>install setuptools or the alternate package manager of your choice,
>and then you can use easy_install (or whatever alternative) to install
>the rest.

Well, I noticed that the other package managers were writing 
bootstrap scripts that then download setuptools' bootstrap script and 
run it as part of *their* bootstrap process...  and then I got to 
thinking that it sure would be nice for setuptools to not have to be 
a giant monolithic download if I wanted to start using other packages 
in it...  and that it sure would be nice to get rid of all these 
bootstrap scripts downloading other bootstrap scripts...  and then I 
wrote PEP 365.  :)

One other thing that PEP 365 does for these use cases that your 
approach doesn't, is that pkg_resources could detect whether a 
desired package of a usable version was *already* installed, and skip 
it if so.  So, we've already scaled back the intended use cases quite 
a bit, as people will have to write their own "is it already there?" 
and "is it the right version?" checks.


> >  Without one or the other, the bootstrap tool would have to grow a
> >  version parsing scheme of some type, and play guessing games with
> >  file extensions.  (Which is one reason I limited PEP 365's scope to
> >  downloading eggs actually *uploaded* to PyPI, rather than arbitrary
> >  packages *linked* from PyPI.)
>
>There are two version parsers in distutils, referenced by PEP 345, the
>PyPI 1.2 metadata standard.

Yes, and StrictVersion doesn't parse release candidates.  And neither 
LooseVersion nor StrictVersion supports handling multiple 
pre/post-release tags correctly.  (E.g. "1.1a1dev-r2753")


> >  So, if I had to propose something right now, I would be inclined 
> to propose:
> >
> >  * using setuptools' version parsing semantics for interpretation of
> >  alpha/beta/dev/etc. releases
>
>Can you point me to the code for this? What is its advantage over
>distutils.version?

It implements version comparison semantics that are closer to 
programmer expectations.  It has also been far more widely used and 
exposed to more feedback.  distutils.version, as far as I know, is 
really only used by the PEP 345 metadata standard -- which isn't used 
by *any* automated tools as far as I know, and I'm not sure how many 
packages bother declaring it.

In addition to alpha/beta/candidate/dev versions, it also supports 
post-release (patchlevel) tags such as svn revision or date-based tags.

Here is the code; the docstring is actually longer than the bits that 
do anything:

def parse_version(s):
     """Convert a version string to a chronologically-sortable key

     This is a rough cross between distutils' StrictVersion and LooseVersion;
     if you give it versions that would work with StrictVersion, then 
it behaves
     the same; otherwise it acts like a slightly-smarter LooseVersion. It is
     *possible* to create pathological version coding schemes that will fool
     this parser, but they should be very rare in practice.

     The returned value will be a tuple of strings.  Numeric portions of the
     version are padded to 8 digits so they will compare numerically, but
     without relying on how numbers compare relative to strings.  Dots are
     dropped, but dashes are retained.  Trailing zeros between alpha segments
     or dashes are suppressed, so that e.g. "2.4.0" is considered the same as
     "2.4". Alphanumeric parts are lower-cased.

     The algorithm assumes that strings like "-" and any alpha string that
     alphabetically follows "final"  represents a "patch level".  So, "2.4-1"
     is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
     considered newer than "2.4-1", which in turn is newer than "2.4".

     Strings like "a", "b", "c", "alpha", "beta", "candidate" and so on (that
     come before "final" alphabetically) are assumed to be 
pre-release versions,
     so that the version "2.4" is considered newer than "2.4a1".

     Finally, to handle miscellaneous cases, the strings "pre", "preview", and
     "rc" are treated as if they were "c", i.e. as though they were release
     candidates, and therefore are not as new as a version string that does not
     contain them, and "dev" is replaced with an '@' so that it sorts 
lower than
     than any other pre-release tag.
     """
     parts = []
     for part in _parse_version_parts(s.lower()):
         if part.startswith('*'):
             if part<'*final':   # remove '-' before a prerelease tag
                 while parts and parts[-1]=='*final-': parts.pop()
             # remove trailing zeros from each series of numeric parts
             while parts and parts[-1]=='00000000':
                 parts.pop()
         parts.append(part)
     return tuple(parts)

component_re = re.compile(r'(\d+ | [a-z]+ | \.| -)', re.VERBOSE)
replace = {'pre':'c', 'preview':'c','-':'final-','rc':'c','dev':'@'}.get

def _parse_version_parts(s):
     for part in component_re.split(s):
         part = replace(part,part)
         if not part or part=='.':
             continue
         if part[:1] in '0123456789':
             yield part.zfill(8)    # pad for numeric comparison
         else:
             yield '*'+part

     yield '*final'  # ensure that alpha/beta/candidate are before final


To check a parse_version() value for stability, you can just loop 
over it looking for any part <"*foo" where "foo" is the desired 
minimum stability.  That is, if you find a '*a' and you don't want 
alphas, then this version's no good.  This lets you also distinguish 
between a beta that you might accept, from an in-development snapshot 
of a beta, that you wouldn't.



>What's wrong with just running "setup.py install"? I'd rather continue
>existing standards / conventions. Of course, it won't work when
>setup.py requires setuptools;

Actually, it will, if the setup script uses the current ez_setup 
bootstrapping method for setuptools.

However, I'd like to get *rid* of that bootstrapping method, and 
replace it with this one.  That's why I'd prefer that the bootstrap 
approach use a different entry point for launching, and why I want 
the module to expose an API, and why I don't really want the 
bootstrapper to actually "install" anything.

For one thing, it means dealing with installation *options*.  Your 
prototype doesn't pass through any command-line options to the 
script, so people would have to use a ~/.pydistutils.cfg file in 
order to control the installation options, for example.  (Which then 
can break if the packager included a setup.cfg that was supposed to 
be overridden on the command line...)

Probably this seems a lot more messy to me, because I've had my face 
directly planted in the mess for a number of years now, and I know 
that, for example, people bitched and moaned excessively about not 
being able to use --prefix with easy_install, the way they could with 
'setup.py install'.

And maybe my experiences aren't all relevant here; I'm just not very 
good at turning them off.  My skepticism for the setup.py-based 
approach is at close to "new scheme for removing the GIL" level, 
because I've gone through a lot of pain to get easy_install from the 
stage where it looked a lot like your bootstrap prototype, to 
something that actually works, most of the time, for arbitrary 
distutils packages.  :)

And unfortunately, some of the hurdles will require a few release 
cycles to show up.  And hey, if you're okay with that, cool.  I just 
think that as soon as it gets out in the field, people will use it 
far outside anything we expect it to be used for, and if there's not 
a bright line for the *packager* to cross, I think we'll have people 
unhappy with the tool.

If you have to do a special step to make something bootstrappable, 
then when the tool doesn't work, the user will ask the packager to 
take the special step.  However, if the tool allows the user to 
*point* it at any package, and it randomly (from the user's POV) 
fails, then the tool (and Python) will be blamed for the failure.

Because even though the bootstrap tool is "not a package manager", if 
it's close enough to look like "a simpler easy_install", people will 
try to use it as one, and blog about how bootstrap is broken and 
should support installation options, etc.

(I suppose at this point easy_install is something of a 
counter-example to this worry; people can and do now give packagers 
patches to make their setup scripts more compatible with 
easy_install, in cases where the package does extensive distutils 
modification.  OTOH, easy_install is a de facto standard, where 
bootstrap will be de jure.  What does that mean in practice?  Heck if 
I know.  :)  I guess people will hate on you instead of me, then, so 
maybe I should view that as an improvement.  :)  (It also makes it 
easier to understand your reluctance to be in any way associated with 
eggs, but there's a big difference between eggs and easy_install, and 
IMO your approach leans more towards the relative vices of 
easy_install than the relative virtues of eggs.  But oh well.))


From jim at zope.com  Wed Mar 19 20:58:58 2008
From: jim at zope.com (Jim Fulton)
Date: Wed, 19 Mar 2008 15:58:58 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080319195257.GD8098@fridge.pov.lt>
References: <20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<7128D92F-79DB-41CD-A8F5-80F0F0307432@zope.com>
	<20080319195257.GD8098@fridge.pov.lt>
Message-ID: <36F3802F-F289-43F4-8FF5-9C8FB7FCF263@zope.com>


On Mar 19, 2008, at 3:52 PM, Marius Gedminas wrote:

> On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote:
>> On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote:
>> ...
>>> os.path.join(os.path.dirname(__file__), 'foo') just has too many
>>> problems.
>>
>> Could you be specific?
>
> I suppose I was thinking about the stale .pyc file problem, mostly.
>
>
> zc.buildout/setuptools/distutils (I'm not sure which part of the stack
> is to blame) likes to compile modules in a temporary directory and  
> then
> move the .py and .pyc files to their final location.  As a result,
> __file__ points to the now-nonexistent directory.

You are mistaken.  The file information in the code object is wrong,  
which is wildly annoying when looking at traceback or using pdb, but   
__file__ is correct.

BTW, I'm pretty sure the bad code file information is my fault.  I  
definitely plan to fix this.

> Aside: I never understood the reason for storing the value of __file__
> in the .pyc/.pyo file itself.  Why not set it to the correct value on
> import time?

Agreed.  But this has no impact on resource loading.

Jim

--
Jim Fulton
Zope Corporation



From pje at telecommunity.com  Wed Mar 19 21:02:09 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 16:02:09 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
 module)
In-Reply-To: <79990c6b0803191141h7381de22v3070bf35247371df@mail.gmail.co
 m>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<79990c6b0803181401t3fdd4e8fu20c985c13354de7f@mail.gmail.com>
	<20080318222317.8C4B83A409D@sparrow.telecommunity.com>
	<79990c6b0803181544w5ffa81c9y86ae955eab7a4c31@mail.gmail.com>
	<20080319000559.C80223A4074@sparrow.telecommunity.com>
	<79990c6b0803190243q7f2c0f62wea55ee7ccfbdf5a8@mail.gmail.com>
	<20080319160002.74BC03A40A5@sparrow.telecommunity.com>
	<79990c6b0803191004v5df2f25r57194214d657cf3d@mail.gmail.com>
	<20080319183600.67B953A40A5@sparrow.telecommunity.com>
	<79990c6b0803191141h7381de22v3070bf35247371df@mail.gmail.com>
Message-ID: <20080319200210.8CF2D3A40A5@sparrow.telecommunity.com>

At 06:41 PM 3/19/2008 +0000, Paul Moore wrote:
>I may just go for a simple registry of implementation classes.

That's all simplegeneric *is*, in this context.  A way of having 
"simple registries", since more than one was needed there.


>  Sorry, but if you
>couldn't get sufficient agreement to simplegeneric to get it in a
>"proper" location,

It's a private implementation detail of pkgutil, not a stdlib 
feature, which is why no such agreement was requested or required.


From jim at zope.com  Wed Mar 19 20:37:59 2008
From: jim at zope.com (Jim Fulton)
Date: Wed, 19 Mar 2008 15:37:59 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080318185154.GA31013@fridge.pov.lt>
References: <20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
Message-ID: <7128D92F-79DB-41CD-A8F5-80F0F0307432@zope.com>


On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote:
...
> os.path.join(os.path.dirname(__file__), 'foo') just has too many
> problems.

Could you be specific?  It has worked pretty well for us.  It makes it  
hard to provide software in zip files, but that seems overrated. In  
particular, I recently noticed that it took significantly longer to  
start an app from zipped eggs than from unzipped eggs.  And, of  
course, zipped eggs are a lot less convenient for looking at code.

Jim

--
Jim Fulton
Zope Corporation



From dpeterson at enthought.com  Wed Mar 19 21:13:27 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Wed, 19 Mar 2008 15:13:27 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E11659.9000307@v.loewis.de>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
Message-ID: <47E173E7.9060300@enthought.com>

Martin v. L?wis wrote:
>> 1. What is the plan for PyPI when Python 3.0 comes out and
>>     dependencies start getting satisfied from distribution
>>     across the great divide, e.g. a 3.0-specific package
>>     pulls from PyPI a 2.x-specific package to meet some
>>     need?  Are there plans to fork PyPI, apply special
>>     tags to uploads or what?
>>     
>
> I don't see the need to for PyPI. For packages (or "distributions",
> to avoid confusion with Python packages), I see two options:
>
> a) provide a single release that supports both 2.x and 3.x.
>     The precise strategy to do so might vary. If one is going
>     for a single source version, have setup.py run 2to3
>     (or perhaps 3to2). For dual-source packages, have setup.py
>     just install the one for the right version; setup.py itself
>     needs to be written so it runs on both versions (which is
>     easy to do).
> b) switch to Python 3 at some point (i.e. burn your bridges).
>
> You seem to be implying that some projects may release separate
> source distributions. I cannot imagine why somebody would want
> to do that.
>   

While not quite to the same scale as the 2 to 3 transition, this problem 
seems like one that would generally already exist.  If one writes code 
that uses newer 2.4/2.5 features (say decorators for example,) it won't 
build/run on 2.3 or earlier installs.  How have people been handling 
that sort of situation?  Is it only by not using the newer features or 
is there some situation  I'm just not seeing in a brief review of a few 
projects pages on PyPI where there is only one source tarball?

-- Dave

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080319/40a13e12/attachment.htm 

From guido at python.org  Wed Mar 19 22:23:26 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 19 Mar 2008 14:23:26 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
Message-ID: <ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>

On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
[a long message]

I'm back at Google and *really* busy for another week or so, so I'll
have to postpone the rest of this discussion for a while. If other
people want to chime in please do so; if this is just a dialog between
Phillip and me I might incorrectly assume that nobody besides Phillip
really cares.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From marius at pov.lt  Wed Mar 19 22:39:08 2008
From: marius at pov.lt (Marius Gedminas)
Date: Wed, 19 Mar 2008 23:39:08 +0200
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <36F3802F-F289-43F4-8FF5-9C8FB7FCF263@zope.com>
References: <47DEA275.4080306@behnel.de>
	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>
	<20080317174542.89A463A409D@sparrow.telecommunity.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<20080318185154.GA31013@fridge.pov.lt>
	<7128D92F-79DB-41CD-A8F5-80F0F0307432@zope.com>
	<20080319195257.GD8098@fridge.pov.lt>
	<36F3802F-F289-43F4-8FF5-9C8FB7FCF263@zope.com>
Message-ID: <20080319213908.GA28461@fridge.pov.lt>

On Wed, Mar 19, 2008 at 03:58:58PM -0400, Jim Fulton wrote:
> On Mar 19, 2008, at 3:52 PM, Marius Gedminas wrote:
> >On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote:
> >>On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote:
> >>...
> >>>os.path.join(os.path.dirname(__file__), 'foo') just has too many
> >>>problems.
> >>
> >>Could you be specific?
> >
> >I suppose I was thinking about the stale .pyc file problem, mostly.
> >
> >zc.buildout/setuptools/distutils (I'm not sure which part of the stack
> >is to blame) likes to compile modules in a temporary directory and  
> >then
> >move the .py and .pyc files to their final location.  As a result,
> >__file__ points to the now-nonexistent directory.
> 
> You are mistaken.  The file information in the code object is wrong,  
> which is wildly annoying when looking at traceback or using pdb, but   
> __file__ is correct.

Oops, you're right.  My brain stopped working (which often happens
lately, just about when I decide to write an email or comment in a bug
tracker.  Late evenings and lack of sleep may have something to do with
it.).

I retract my statement.  The only problem with
os.path.join(os.path.dirname(__file__), 'foo') is that it's long and
cumbersome to type (and doesn't work in zip files, but I don't much care
about those).

Marius Gedminas
-- 
If it weren't for the last minute, nothing would get done.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080319/9f80cac9/attachment.pgp 

From jeff at taupro.com  Wed Mar 19 23:15:43 2008
From: jeff at taupro.com (Jeff Rush)
Date: Wed, 19 Mar 2008 17:15:43 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080319152101.730BC3A40A5@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E0D571.2070004@taupro.com>
	<20080319152101.730BC3A40A5@sparrow.telecommunity.com>
Message-ID: <47E1908F.5030501@taupro.com>

Phillip J. Eby wrote:
> At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote:
>> Are you open to giving certain others patch view/commit privileges to 
>> setuptools?
> 
> Jim Fulton has such already.  I'm open to extending that to others who 
> have a good grasp of the subtleties involved.
> 
> Truthfully, if we can just get 0.6 put to bed, I could probably open up 
> the trunk a lot wider.

What is needed to put 0.6 to bed?  How can we help accelerate this?


> Probably the most frustrating thing (or "chief amongst the most 
> frustrating things") about setuptools development is that it's a black 
> hole.  By which I mean that backward compatibility and cruft accretion 
> make it difficult to get out of.
> 
> In the beginning, there was the distutils.  Distutils begat setuptools, 
> and setuptools begat virtualenv and zc.buildout and source control 
> plugins.  Etc., etc.

I've found in the past a revisiting of basic principles and objectives, 
communicated in enhanced documentation, can help to clear out such black 
holes. ;-)  I'm pulling something together, from the recent emails and some 
archived threads -- it definitely is tangled though, I'll agree.


> What I think is really needed in the long run is to keep eggs, but get 
> rid of setuptools and the distutils in their current form.  There's a 
> lot of brokenness there, and also a lot of accumulated cruft.  We really 
> need a distutils 3000, and it needs to be built on a better approach.

That will require a lot of concensus building as well as collection of use 
cases so that the architecture team can encompass aspects they are not 
personally aware of.  As you've said, it's hard to address itches that are not 
your own.

It certainly is possible for someone to create a parallel packaging moduleset 
that uses the existing eggs format and PyPI but without the currently 
codebase, and then, once proven to work, lobby for it as distutils 3000.

Frankly I'd like to see setuptools exploded, with those parts of general use 
folded back into the standard library, the creation of a set of 
non-implementation-specific documents of the distribution formats and 
behavior, leaving a small core of one implementation of how to do it and the 
door open for others to compete with their own implementation.


> In truth, my *real* motivation for PEP 365's bootstrap tool isn't so 
> much to support the package management tools we have today, as it is to 
> support a new one tomorrow.  I have a few ideas for ways to shift the 
> paradigm of how individual projects get built, to incorporate many 
> scenarios that don't work well now.  But to implement those things in 
> such a next-generation tool, I will not want to be restricted to just 
> what's in the stdlib or what can be bundled in the tool.

You should document those ideas someplace and start getting community input. 
There are a lot of diverse opinions on the right way to do this and the way 
ahead is quite unclear.


> And I think it's probably getting close to time I stepped down from 
> day-to-day management of the codebase (which is more like month-to-month 
> or quarter-to-quarter for me lately).  It will probably be a lot easier 
> for me to step back and critique stuff that goes in, after the fact, 
> than to go over the stuff beforehand.  :)
> 
> I'm not sure exactly how to go about such a handoff though.  My guess is 
> that we need a bug/patch tracker, and a few people to review, test, and 
> apply.  Maybe a transitional period during which I just say yea or nay 
> and let others do the test and apply, before opening it up entirely.  
> That way, we can perhaps solidify a few principles that I'd like to have 
> stay in place.  (Like no arbitrary post-install code hooks.)

I'll see about a tracker and identify some people to help out.


> btw, offtopic question: are you by any chance the same Jeff Rush who 
> invented EchoMail?

Yep, that's me.  Not many remember the Fidonet days.  I designed EchoMail on a 
napkin during a DFW Sysop pizza party during a conversation on what to do with 
the unused capability of inter-BBS private file transfers.  It too escaped its 
ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-)

-Jeff


From jeff at taupro.com  Wed Mar 19 23:59:10 2008
From: jeff at taupro.com (Jeff Rush)
Date: Wed, 19 Mar 2008 17:59:10 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E11659.9000307@v.loewis.de>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
Message-ID: <47E19ABE.2040800@taupro.com>

Martin v. L?wis wrote:
> 
> I don't see the need to for PyPI. For packages (or "distributions",
> to avoid confusion with Python packages), I see two options:
> 
> a) provide a single release that supports both 2.x and 3.x.
> b) switch to Python 3 at some point (i.e. burn your bridges).
> 
> You seem to be implying that some projects may release separate
> source distributions. I cannot imagine why somebody would want
> to do that.

Yes, I am assuming that existing projects would at some point introduce a 3.x 
version and maybe continue a 2.x version as separate distros, similar to 
Python itself.  Then the large number of existing unqualified dependencies on, 
say SQLObject, would pull in the higher 3.x version and crash.  It's the older 
projects that don't get updated often that are at risk of being destabilized 
by the arrival of 3.x specific code in PyPI.  Are developers for Python 3.x 
encouraged in 3.x guidelines to release 'fat' distributions that combine 2.x 
and 3.x usable versions?

There is also some hassle with 2.x programmers browsing PyPI for useful 
modules to incorporate in their programs, downloading them (w/easy_install so 
they don't see the project website) and getting streams of errors because they 
unknowningly hit a 3.x-specific version.

Perhaps a convention of a keyword or more likely a new trove classifier that 
spells outs 3.x stuff, with indicators on package info pages and query filters 
on PyPI against that?


>> 2. There have been attempts over the years to fix distutils,
>>     with the last one being in 2006 by Anthony Baxter. He
>>     stated that a major hurdle was the strong demand to
>>     respect backward compatibility and he finally gave up.
> 
> Can you kindly refer to some archived discussion for that?

http://mail.python.org/pipermail/python-dev/2006-April/063943.html

   "I started looking at this. The number of complaints I
    got when I started on this that it would break the
    existing distutils based installers totally discouraged
    me. In addition, the existing distutils codebase is ...
    not good.

    It is flatly not possible to "fix" distutils and
    preserve backwards compatibility." -Anthony Baxter


>>     One of the purposes of Python 3.0 was the freedom to
>>     break backward compatibility for the sake of "doing
>>     the right thing".  So is it now permissible to give
>>     distutils a good reworking and stop letting
>>     compatibility issues hold us back?
> 
> I don't know what the proposed changes are, but for some
> changes; in general, I feel that the need for backwards
> compatibility is exaggerated.

A controversial point, I'm afraid.  Perhaps it is time for a parallel rewrite, 
so that those setup.py who import distutils get the old behavior, and those 
who import distutils2 get the new, and we let attrition and the community 
decide which is standard.

-Jeff


From pje at telecommunity.com  Thu Mar 20 00:10:00 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 19 Mar 2008 19:10:00 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47E1908F.5030501@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E0D571.2070004@taupro.com>
	<20080319152101.730BC3A40A5@sparrow.telecommunity.com>
	<47E1908F.5030501@taupro.com>
Message-ID: <20080319231002.07DF43A40A5@sparrow.telecommunity.com>

At 05:15 PM 3/19/2008 -0500, Jeff Rush wrote:
>Phillip J. Eby wrote:
> > At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote:
> >> Are you open to giving certain others patch view/commit privileges to
> >> setuptools?
> >
> > Jim Fulton has such already.  I'm open to extending that to others who
> > have a good grasp of the subtleties involved.
> >
> > Truthfully, if we can just get 0.6 put to bed, I could probably open up
> > the trunk a lot wider.
>
>What is needed to put 0.6 to bed?  How can we help accelerate this?

Get a tracker set up.  I'm already in the main Python one, might as 
well use that.


>It certainly is possible for someone to create a parallel packaging moduleset
>that uses the existing eggs format and PyPI but without the currently
>codebase, and then, once proven to work, lobby for it as distutils 3000.

Yep.  And I believe that something will look rather more like 
zc.buildout than setuptools, actually.  Specifically in being 
data-driven rather than script-driven, and in the flexibility of what 
sort of parts get build and by what methods.  Setuptools is still too 
rooted in distutils' world, the world where you can't depend on any 
other components being around to build things with.


>Frankly I'd like to see setuptools exploded, with those parts of general use
>folded back into the standard library, the creation of a set of
>non-implementation-specific documents of the distribution formats and
>behavior, leaving a small core of one implementation of how to do it and the
>door open for others to compete with their own implementation.

Apart from the exploding part, there are already documents.  The only 
thing that makes them implementation-specific is that they haven't 
passed through any magic blessing process to make them standards.


>You should document those ideas someplace and start getting community input.
>There are a lot of diverse opinions on the right way to do this and the way
>ahead is quite unclear.

We might be talking about different things, as I'm more concerned 
with replacing setuptools and distutils on the build-and-distribute 
side.  What's needed there is more the weeding out of too many ways 
to do simple things, and fixing the complete absence of ways to do 
complex things.  :)  For simple things the distutils are too hard, 
and for slightly-more-complex things, the entry barrier encourages 
people to abandon and replace them.

On the package management side, I'm somewhat more inclined to agree 
with the need for a community approach, though.


> > btw, offtopic question: are you by any chance the same Jeff Rush who
> > invented EchoMail?
>
>Yep, that's me.  Not many remember the Fidonet days.  I designed 
>EchoMail on a
>napkin during a DFW Sysop pizza party during a conversation on what 
>to do with
>the unused capability of inter-BBS private file transfers.  It too 
>escaped its
>ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-)

Ah, so you *do* know what it's like to develop setuptools, then.  I 
might even have met you at the one DFW sysop pizza party I ever 
attended.  Back then, I ran the FreeZone, and before that, "Ferris 
Bueller's Fine Arts Forum", back in the late 80's and early 90's.  My 
wife met me through the D/FW BBS list in the back of Computer 
Shopper, with a modem she bought at Software, Etc., up in Allen or 
wherever that place was.  Not the chain store, the little consignment 
shop.  Those were the days.  But now we're *really* getting off-topic.  :)


From jeff at taupro.com  Thu Mar 20 00:17:54 2008
From: jeff at taupro.com (Jeff Rush)
Date: Wed, 19 Mar 2008 18:17:54 -0500
Subject: [Distutils] Request for Input re Packaging
Message-ID: <47E19F22.5000600@taupro.com>

In researching the state of packaging, I've been reading the archives and all 
the bug reports filed against distutils.

I'd like though to get some examples of particularly troublesome uses of 
setup.py, to pull together and propose some changes to make their use case a 
bit easier.  So far such cases I've been made aware of are Twisted, numpy and 
SciPy.  If you know of a tough case where the developer had to jump through 
hoops to make it work, please point me to it.

I'd also like to get suggestions of improvements to PyPI, which I've not seen 
much discussion about.  A few I've collected are:

   - move to https/ssl

   - improvements re package signing

   - internal parsing/aggregation of metadata for better queries,
     and to stop using the filename for version/platform/etc. information.

   - moving of requirements logic from client into PyPI, where it
     has db access to the dependency, resolves what packages are
     needed and delivers a list back to the client for prompting
     the user for permission, similar to how "yum" interacts today.

   - a db lint-picking walker that looks for problems on PyPI, such
     as binary distros w/o a source distro, lack of binaries for
     those platforms often without compilers, failure to provide a
     link to a version repo for use with "project==dev".

   - some auto-generated reports of access statistics and the mix
     of distutils vs setuptools, those who "registered" w/o
     "uploading", and perhaps if we get a new classifier assigned,
     some idea of Python 2.x vs 3.x packages.

Last, some of the issues with distutils/setuptools can be solved with 
zc.buildout.  If you have found zc.buildout lacking, please tell me where it 
fell short so we can see if anything can be done.

Thanks for your involvement,

-Jeff


From greg.ewing at canterbury.ac.nz  Thu Mar 20 00:05:47 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 20 Mar 2008 11:05:47 +1200
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E19ABE.2040800@taupro.com>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
	<47E19ABE.2040800@taupro.com>
Message-ID: <47E19C4B.4020505@canterbury.ac.nz>

Jeff Rush wrote:
> Perhaps it is time for a parallel rewrite, 
> so that those setup.py who import distutils get the old behavior, and those 
> who import distutils2 get the new,

That sounds good to me. If anyone wants to have a go
at this, I have some ideas on how to structure it
that I'd be happy to discuss.

-- 
Greg

From martin at v.loewis.de  Thu Mar 20 01:48:21 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 Mar 2008 19:48:21 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment	Technology Concerns
In-Reply-To: <CFF57303-0AD8-4B11-BB62-03A9D5A346FB@python.org>
References: <47DEEC6B.5040602@taupro.com>	<20080318004718.56E173A40FF@sparrow.telecommunity.com>	<47E0D571.2070004@taupro.com>
	<CFF57303-0AD8-4B11-BB62-03A9D5A346FB@python.org>
Message-ID: <47E1B455.9030900@v.loewis.de>

> The Python sandbox has a setuptools directory.  Is this the canonical  
> location for the code? 

Yes, it is.

> If so, then anybody who has Python commit  
> privileges can commit to it and help further develop setuptools.

They can, but they shouldn't. Nothing should be committed there
without pje's approval (in whatever form he choses to give such
approval).

> If not, why not and what is the sandbox setuptools used for?

I think it shouldn't be in sandbox, but toplevel, but that's a
minor detail. Maybe I misunderstand the English word "sandbox".

Regards,
Martin


From martin at v.loewis.de  Thu Mar 20 02:05:28 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 Mar 2008 20:05:28 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<ca471dc20803171017j548446a4l64aa9dc7d4c6b41a@mail.gmail.com>	<20080317174542.89A463A409D@sparrow.telecommunity.com>	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>	<20080317184553.913413A409D@sparrow.telecommunity.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
Message-ID: <47E1B858.1090107@v.loewis.de>

> I don't understand PyPI all that well; it seems poor design that the
> browsing via keywords is emphasized but there is no easy way to
> *search* for a keyword (the list of all packages is not emphasized
> enough on the main page -- it occurs in the side bar but not in the
> main text).

I don't understand. What is "browsing via keywords" and how is that
emphasized? (one I know that, I can look into ways for searching
for keywords)

> I assume there's a programmatic API (XML-RPC?) but I
> haven't found it yet.

The recommended "programmatic" API is

http://pypi.python.org/simple/

Not sure what you were trying to achieve programmatically;
"typically" people know what they want to install (e.g.
"threadedcomments"), and then the tool goes directly to

http://pypi.python.org/simple/threadedcomments/

Regards,
Martin


From martin at v.loewis.de  Thu Mar 20 02:14:58 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 Mar 2008 20:14:58 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E173E7.9060300@enthought.com>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
	<47E173E7.9060300@enthought.com>
Message-ID: <47E1BA92.8050709@v.loewis.de>

> While not quite to the same scale as the 2 to 3 transition, this problem 
> seems like one that would generally already exist.  If one writes code 
> that uses newer 2.4/2.5 features (say decorators for example,) it won't 
> build/run on 2.3 or earlier installs.  How have people been handling 
> that sort of situation?  Is it only by not using the newer features or 
> is there some situation  I'm just not seeing in a brief review of a few 
> projects pages on PyPI where there is only one source tarball?

I think packages have taken all sorts of responses to this issue.
Some will list the minimum required Python version in their README,
some might put a test in setup.py that aborts installation if the
Python version is too old, some may just install and let the user
find out at runtime.

Typically, packages try to support all the Python versions that their
users still use. If a user of an older Python version comes along,
they'll just need to fetch the older release (which hopefully is
still online, or can be extracted from the source repository).

Regards,
Martin


From martin at v.loewis.de  Thu Mar 20 02:25:40 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 Mar 2008 20:25:40 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E19ABE.2040800@taupro.com>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
	<47E19ABE.2040800@taupro.com>
Message-ID: <47E1BD14.2030805@v.loewis.de>

> Yes, I am assuming that existing projects would at some point introduce a 3.x 
> version and maybe continue a 2.x version as separate distros, similar to 
> Python itself.  Then the large number of existing unqualified dependencies on, 
> say SQLObject, would pull in the higher 3.x version and crash.  It's the older 
> projects that don't get updated often that are at risk of being destabilized 
> by the arrival of 3.x specific code in PyPI.  Are developers for Python 3.x 
> encouraged in 3.x guidelines to release 'fat' distributions that combine 2.x 
> and 3.x usable versions?

Passive voice is misleading here: encouraged by whom?

*I* encourage people to consider that option, rather than assuming it
couldn't possibly work. Whether it actually works, I don't know.
I hope it would work, and I hope it would not be fat at all.

> Perhaps a convention of a keyword or more likely a new trove classifier that 
> spells outs 3.x stuff, with indicators on package info pages and query filters 
> on PyPI against that?

I'm fine with adding more trove classifiers if that solves the problem
(although I still assume the problem doesn't actually exist).

As always, a classifier should not be added until there actually are
two packages that want to use it.

>> Can you kindly refer to some archived discussion for that?
> 
> http://mail.python.org/pipermail/python-dev/2006-April/063943.html
> 
>    "I started looking at this. The number of complaints I
>     got when I started on this that it would break the
>     existing distutils based installers totally discouraged
>     me. In addition, the existing distutils codebase is ...
>     not good.
> 
>     It is flatly not possible to "fix" distutils and
>     preserve backwards compatibility." -Anthony Baxter

Thanks. I still have the same position as I had then - if
distutils is broken, it should be fixed, not ignored.

> A controversial point, I'm afraid.  Perhaps it is time for a parallel rewrite, 
> so that those setup.py who import distutils get the old behavior, and those 
> who import distutils2 get the new, and we let attrition and the community 
> decide which is standard.

Is there a list of the problems with distutils somewhere?

It always worked fine for me, so I see no reason to fix it in the
first place.

Regards,
Martin

From bignose+hates-spam at benfinney.id.au  Thu Mar 20 04:26:16 2008
From: bignose+hates-spam at benfinney.id.au (Ben Finney)
Date: Thu, 20 Mar 2008 14:26:16 +1100
Subject: [Distutils] Request for Input re Packaging
References: <47E19F22.5000600@taupro.com>
Message-ID: <87zlsuvzxj.fsf@benfinney.id.au>

Jeff Rush <jeff at taupro.com> writes:

> In researching the state of packaging, I've been reading the
> archives and all the bug reports filed against distutils.
> 
> I'd like though to get some examples of particularly troublesome
> uses of setup.py, to pull together and propose some changes to make
> their use case a bit easier. So far such cases I've been made aware
> of are Twisted, numpy and SciPy. If you know of a tough case where
> the developer had to jump through hoops to make it work, please
> point me to it.
> 
> I'd also like to get suggestions of improvements to PyPI, which I've
> not seen much discussion about.

I know that those in Debian who are involved with packaging Python
modules and applications often complain about the state of distutils
and setuptools. I don't know what the specific complaints are, though.

I'll forward this request to the debian-python list in an attempt to
rouse them for feedback.

-- 
 \            "Why was I with her? She reminds me of you. In fact, she |
  `\             reminds me more of you than you do!"  -- Groucho Marx |
_o__)                                                                  |
Ben Finney


From zooko at zooko.com  Thu Mar 20 05:18:47 2008
From: zooko at zooko.com (zooko)
Date: Wed, 19 Mar 2008 22:18:47 -0600
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
Message-ID: <EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>

On Mar 19, 2008, at 3:23 PM, Guido van Rossum wrote:

> If other people want to chime in please do so; if this is just a
> dialog between Phillip and me I might incorrectly assume
> that nobody besides Phillip really cares.

I really care.  I've used setuptools, easy_install, eggs, and  
pkg_resources extensively for the past year or so (and contributed a  
few small patches).  There have been plenty of problems, but I find  
them to be overall useful tools.

It is a great boon to a programming community to lower the costs of  
re-using other people's code.  The Python community will benefit  
greatly once a way to do that becomes widely enough accepted to reach  
a tipping point and become ubiquitous.  Setuptools is already the de  
facto standard, but it hasn't become ubiquitous, possibly in part  
because of "egg hatred", about which more below.

I've interviewed several successful Python hackers who "hate eggs" in  
order to understand what they hate about them, and I've taken notes  
from some of these interviews.  (The list includes MvL, whose name  
was invoked earlier in this thread.)

After filtering out yer basic complaining about bugs (which  
complaints are of course legitimate, but which don't indict  
setuptools as worse than other software of comparable scope and  
maturity), their objections seem to fall into two categories:

1.  "The very notion of package dependency resolution and  
programmable or command-line installation of packages at the language  
level is a bad notion."

This can't really be the case.  If the existence of such  
functionality at the programming language level were an inherently  
bad notion, then we would be hearing some complaints from the Ruby  
folks, where the Gems system is standard and ubiquitous.  We hear no  
complaints -- only murmurs of satisfaction.  One person recently  
reported to me that while there are more packages in Python, he finds  
himself re-using other people's code more often when he works in  
Ruby, because almost all Ruby software is Gemified, but only a  
fraction of Python software is Eggified.

Often this complaint comes with the idea that eggs conflict with  
their system-level package management tools.  (These are usually  
Debian/Ubuntu users.)

Note that Ruby software is not too hard to include in operating  
system packaging schemes -- my Ubuntu Hardy apt-cache shows plenty of  
Ruby software.  A sufficiently mature and widely supported setuptools  
could actually make it easier to integrate Python software into  
Debian -- see stdeb [1].

2.  "Setuptools/eggs give me grief."

What can really be the case is that setuptools causes a host of  
small, unnecessary problems for people who prefer to do things  
differently than PJE does.  Personally, I prefer to use GNU stow, and  
setuptools causes unnecessary, but avoidable, problems for me.  Many  
people object (rightly enough) to a "./setup.py install"  
automatically fetching new software over the Internet by default.   
The fact that easy_install creates a site.py that changes the  
semantics of PYTHONPATH is probably the most widely and deservedly  
hated example of this kind of thing [2].  I could go on with a few  
other common technical complaints of this kind.

These type-2 problems can be fixed by changing setuptools or they can  
be grudgingly accepted by users, while retaining compatibility with  
the large and growing ecosystem of eggy software.  Certainly fixing  
setuptools to play better with others is a more likely path to  
success than setting out to invent a non-egg-compatible alternative.   
Such a project might never be implemented well enough to serve, and  
if it were it would probably never overtake eggs's lead in the Python  
ecosystem, and if it did it would probably not turn out to be a  
better tool.

So, since you asked for my chime, I advise you to publically bless  
eggs, setuptools, and easy_install as plausible future standards and  
solicit patches which address the complaints.  For that matter,  
soliciting specific complaints would be a good start.  I've done so  
in private many times with only partial success as to the "specific"  
part.  One promising approach is to request objections in the form of  
automated tests that setuptools fails, e.g. [3].

Regards,

Zooko O'Whielacronx

[1] http://stdeb.python-hosting.com/
[2] http://www.rittau.org/blog/20070726-02
    And no, PJE's suggested "trivial fix" does not satisfy the  
objectors, as it can't support the use case of "cd somepkg ; python ./ 
setup.py install ; cd .. ; python -c 'import somepkg'".
[3] http://twistedmatrix.com/trac/ticket/2308#comment:5

From jeff at taupro.com  Thu Mar 20 05:41:08 2008
From: jeff at taupro.com (Jeff Rush)
Date: Wed, 19 Mar 2008 23:41:08 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E1BD14.2030805@v.loewis.de>
References: <47E0D883.9030402@taupro.com>
	<47E11659.9000307@v.loewis.de>	<47E19ABE.2040800@taupro.com>
	<47E1BD14.2030805@v.loewis.de>
Message-ID: <47E1EAE4.90802@taupro.com>

Martin v. L?wis wrote:
>> specific code in PyPI.  Are developers for Python 3.x encouraged in 
>> 3.x guidelines to release 'fat' distributions that combine 2.x and 3.x 
>> usable versions?
> 
> Passive voice is misleading here: encouraged by whom?

"... encouraged in __3.x guidelines__ to ...": I presume although I've not 
found them yet that there is some kind of document for developers titled 
something like, "how to migrate your Python code from 2.x to 3.x".  That 
document would be a logical place for advice and consideration of the 
tradeoffs of jumping to 3.x, maintaining two synced versions using 2to3 or 
3to2, and the risks of keeping two independent releases.  Identifying best 
practices would help them make good choices for the community.


> *I* encourage people to consider that option, rather than assuming it
> couldn't possibly work. Whether it actually works, I don't know.
> I hope it would work, and I hope it would not be fat at all.

So we don't have an actual success story of a dual-version distribution, even 
as a prototype, using 2to3 within a distutils package?  I would not encourage 
a practice without at least one such example.


>>> Can you kindly refer to some archived discussion for that?
>>
>> http://mail.python.org/pipermail/python-dev/2006-April/063943.html
>>
> Thanks. I still have the same position as I had then - if
> distutils is broken, it should be fixed, not ignored.

Since the precise API was not documented well and many people began to make 
use of ambiguous internal interfaces, such fixes would indeed break them.  So 
your vote would be to do the right thing, even if it results in some breakage. 
  I can respect that philosophy.


>> A controversial point, I'm afraid.  Perhaps it is time for a parallel 
>> rewrite, so that those setup.py who import distutils get the old 
>> behavior, and those who import distutils2 get the new, and we let 
>> attrition and the community decide which is standard.
> 
> Is there a list of the problems with distutils somewhere?

Unfortunately no.  Much of it is anecdotal, much of it occurs on lists outside 
the Python community by those attempting to package things.  And some of it 
are comments by developers who peeked into the distutils source and blanched.

And some of the problems are not bugs, per se, but disagreement on scope of 
functionality and a lack of well-known alternatives.  So just "fix it if 
broken" doesn't work when there is no agreement on how to expand that scope.

I am working on pulling together such a list however, and getting it into the 
tracker, so that debate with a record of decisions can occur.  I agree that it 
is hard to fix problems if no one is _clearly_ reporting them to us.  Too much 
smoke, not enough light.


> It always worked fine for me, so I see no reason to fix it in the
> first place.

Pardon my lack of knowledge of your background; when you say it always worked 
fine for you, are you referring to personal experiences using it to _install_ 
software or to experiences as a packager in actually distributing complex 
collections of modules on different platforms?

-Jeff


From tseaver at palladion.com  Thu Mar 20 05:58:14 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 20 Mar 2008 00:58:14 -0400
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>	<20080317184553.913413A409D@sparrow.telecommunity.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
Message-ID: <47E1EEE6.5050608@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guido van Rossum wrote:
> On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> [a long message]
> 
> I'm back at Google and *really* busy for another week or so, so I'll
> have to postpone the rest of this discussion for a while. If other
> people want to chime in please do so; if this is just a dialog between
> Phillip and me I might incorrectly assume that nobody besides Phillip
> really cares.

I care, a lot, enough to have volunteered to help with maintenance /
development of setuptols back in September 2007.  I think that, warts an
all, setuptools is a *huge* improvement over bare distutils for nearly
every use case I know about.

A lot of setuptools warts are driven by related design problems in the
distutils, such as the choice to use imperative / procedural code for
everything:  a declarative approach, with hooks for cases which actually
need them (likely 5% of existing packages) would have made writing tools
on top of the framework much simpler.  It is ironic that Python is *too
powerful* a tool for the tasks normally done by distutils / setuptools:
 a more restricted, and thererfore introspectable, configuration-driven
approoach seems much cleaner.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH4e7m+gerLs4ltQ4RAt+hAKDBqIrashlgf8U6XRtfMHjTOaiy4gCeO1Zn
UfdjDYIb2P6vDCcUGSjITTo=
=JTok
-----END PGP SIGNATURE-----


From guido at python.org  Thu Mar 20 06:08:26 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 19 Mar 2008 22:08:26 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E1B858.1090107@v.loewis.de>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<47E1B858.1090107@v.loewis.de>
Message-ID: <ca471dc20803192208q1aca7684o681f542d3d901b32@mail.gmail.com>

I was using the human interface at python.org/pypi. There are two
prominent links at the top of the page: "Browse the tree of packages"
and "Submit package information" followed by the 30 most recently
changed packages. What I was looking for was the page for a specific
package. The "Browse the tree of packages" link was no help. Finally I
realized that in the side bar, in a small unobtrusive font, is a link
to "List packages" which links to a list of *all* packages, in
alphabetical order. I found my package there. I think repeating that
link right below "browse the tree" would have been sufficient. But it
would have been cool if there had been a search box (also in the start
page) where I could type (part of) the name of the package and it
would have given me the nearest matches.

On Wed, Mar 19, 2008 at 6:05 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > I don't understand PyPI all that well; it seems poor design that the
>  > browsing via keywords is emphasized but there is no easy way to
>  > *search* for a keyword (the list of all packages is not emphasized
>  > enough on the main page -- it occurs in the side bar but not in the
>  > main text).
>
>  I don't understand. What is "browsing via keywords" and how is that
>  emphasized? (one I know that, I can look into ways for searching
>  for keywords)
>
>
>  > I assume there's a programmatic API (XML-RPC?) but I
>  > haven't found it yet.
>
>  The recommended "programmatic" API is
>
>
>  http://pypi.python.org/simple/
>
>  Not sure what you were trying to achieve programmatically;
>  "typically" people know what they want to install (e.g.
>  "threadedcomments"), and then the tool goes directly to
>
>  http://pypi.python.org/simple/threadedcomments/
>
>  Regards,
>  Martin
>
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Thu Mar 20 07:57:31 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 19 Mar 2008 23:57:31 -0700
Subject: [Distutils] [Python-Dev] platform management
In-Reply-To: <-3692898589838545524@unknownmsgid>
References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E160CEE20@EXMBX04.exchhosting.com>
	<-3692898589838545524@unknownmsgid>
Message-ID: <ca471dc20803192357n3c8a7a7fq3d72f656c0f418ef@mail.gmail.com>

Great idea! Sounds like a PEP (informational, probably) would be good idea.

On Tue, Mar 18, 2008 at 4:59 PM, Bill Janssen <janssen at parc.com> wrote:
> I don't think this is bike-shedding.
>
>  The debate about "AMD64" vs. "amd64" vs. "x86_64" reminded me that
>  I've been bit more and more frequently by bits of platform-specific
>  knowledge scattered around the standard library.  The latest is the
>  code in distutils.unixccompiler that tries to figure out what flags to
>  send to the linker in order to add a dynamic library path lookup to a
>  shared library.
>
>  There are lots of different ways of figuring out which platform is
>  being used, and they're all over the place.  The code in
>  distutils.unixccompiler uses "sys.platform[:6]", and looks for
>  "darwin"; the code in urllib.py uses "os.name", and expects it to be
>  "mac", but later looks for "sys.platform == 'darwin'; posixfile
>  believes that platforms come with version numbers ("linux2", "aix4");
>  pydoc and tarfile have tests for "sys.platform == 'mac'".  tempfile
>  looks at os.sys.platform *and* os.name.
>
>  Could well be that all of these are correct (I believe that "mac", for
>  instance, refers to the generations of the Mac OS before OS X).  But
>  it means that when someone tries to update "Python" to a new major
>  version release for, say, OS X, it's really easy to miss things.  And
>  the fact that the platform version is sometimes included and sometimes
>  not is also problematic; darwin9 is different from darwin8 in some
>  important aspects.
>
>  It would be nice to
>
>   (a) come up with a list of standard platform symbols,
>   (b) put those symbols somewhere in the documentation,
>   (c) have one way of discovering them, via sys or os or platform or
>       whichever module,
>   (d) add a standard way of discovering the platform version, and
>   (e) stamp out all the other ways that have crept in over the years.
>
>  Bill
>  _______________________________________________
>  Python-Dev mailing list
>  Python-Dev at python.org
>  http://mail.python.org/mailman/listinfo/python-dev
>  Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From david at ar.media.kyoto-u.ac.jp  Thu Mar 20 06:35:38 2008
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 20 Mar 2008 14:35:38 +0900
Subject: [Distutils] Request for Input re Packaging
In-Reply-To: <47E19F22.5000600@taupro.com>
References: <47E19F22.5000600@taupro.com>
Message-ID: <47E1F7AA.7000705@ar.media.kyoto-u.ac.jp>

Jeff Rush wrote:
> In researching the state of packaging, I've been reading the archives and all 
> the bug reports filed against distutils.
>
> I'd like though to get some examples of particularly troublesome uses of 
> setup.py, to pull together and propose some changes to make their use case a 
> bit easier.  So far such cases I've been made aware of are Twisted, numpy and 
> SciPy.  If you know of a tough case where the developer had to jump through 
> hoops to make it work, please point me to it.
>   

Hi,

    My name is David Cournapeau, and I am one of the developer of numpy 
(I am not one of the core developer, but I have been heavily involved 
with a new build system for both numpy and scipy in the last few months, 
so I think I have one or two things to say in this respect).

    My first contact with distutils was because I wanted to add some 
functionalities to numpy.distutils, which is numpy own extensions to 
distutils for numpy needs (things like fortran support, etc...). I 
wanted to add support for building ctypes extensions (.so on linux, 
.dylib on mac os X, .dll on windows, etc...), etc... I quickly gave up 
because of the complexity of distutils, and took a different approach 
(using scons within distutils to build all our compiled code, distutils 
still doing the packaging).

Here are some things which I find frustrating with distutils:

1 extending distutils is not documented at all. Sure, you have a few 
words on distutils commands, but once you want to use compilers in your 
own commands, you are on your own. For example, a working example on how 
to extend distutils with a new command to build something from C would 
be a really good addition.

The relationship between Distribution classes, Command classes and 
Compiler classes should be documented somewhere. The relationship 
between the different distutils commands should be documented somewhere: 
I wanted to do something as simple as adding a distutils command to add 
a whole directory of files: doing it such as it works with sdist, 
install, distutils and setuptools was found impossible, and I found 
easier to regenerate MANIFEST.in with a shell script. That's something 
that should be doable in a hour or two for anyone who does not know 
anything about distutils; today, I am not sure it is doable by anyone 
without a deep knowledge of distutils.
 
2 The only way to understand how distutils works is to run code, because 
a lot of code is based on adding attributes at run-time, etc... 
Basically, a lot of distutils feels like magic to me. For example:
    - in numpy, we want to have tight control of compiler flags: this is 
extremely complicated to do with distutils, because flags are added from 
everywhere in the code, and understanding it enough to change it wo 
breaking anything is nearly impossible. Removing the magic would be 
great (all the configuration in some separate configuration files, for 
example, and the customization at runtime in one clearly separated 
module, for example). But this is a difficult problem: I don't see how 
to change this (in distutils) without breaking someone else code. 
Ideally, it should be easy to customize compiler flags from the command 
line (I bet this is one of the rpm/deb maintainers complain); every few 
days, some people complain on numpy/scipy ML because they use CFLAGS, it 
does not work as they expect it to work, and it breaks the build.
    - compiler usage is not documented. Some functions (initialized) 
have to be called in some order with compiler instances to get some of 
their characteristics; of course, neither the order or which function to 
call for which characteristic is documented anywhere; worse, it depends 
on the compiler (unix vs windows). I don't understand the point of 
adding attributes on runtime, differently, in different cases. Maybe I 
am missing something here
    - why msvc is different from everything else ? In particular, why it 
is not possible to have access to msvc flags in the same manner than all 
other platforms ? Instead, it is burried in the MSVCCompiler code...
    - generally, it is not specified what is public interface and what 
is not. Everything is leaking everywhere, there is no specification.

3 Some code to detect libraries would be good. For example, you write 
code which depends on libfoo: we have our own code in numpy.distutils, 
but that's something which I think many people would like to be able to 
do. A helper tool to parse pkg-config would be good, too.

The magic behavior + lack of documentation really is the main problem 
for me: if there was a small core of functionalities that we could 
extend, the situation would be better. It is difficult to say one 
particular thing is broken, because almost any distutils functionality 
is linked to something else; I cannot find a more precise description 
than magic, and the above points are the first which come to my mind (I 
can find other ones if necessary, but they are all linked to this magic 
thing and lack of precise interfaces).

But changing this in a backward-compatible manner may be extremely 
difficult, maybe even impossible. To be frank, I was secretly hoping 
something would be done on this front for python 3k... I would certainly 
be happy to help if there was some work on a distutils2.

cheers,

David

From ziade.tarek at gmail.com  Thu Mar 20 09:51:25 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 20 Mar 2008 09:51:25 +0100
Subject: [Distutils] Request for Input re Packaging
In-Reply-To: <47E19F22.5000600@taupro.com>
References: <47E19F22.5000600@taupro.com>
Message-ID: <94bdd2610803200151o5efa8589y6341a2ac9013f4c2@mail.gmail.com>

On Thu, Mar 20, 2008 at 12:17 AM, Jeff Rush <jeff at taupro.com> wrote:

[cut]


>   - move to https/ssl
>

Semi-related:

There are a few problems in this area, also related to indexing
we need to work out imho:

When a package defines a https://... link into the url meta-data, the link
will
be added in the Simple index besides other links. For instance, people
that uses sourceforge can have such urls. Even if the package egg or tarball
is available at PyPI, the home page url will appear at #1 on the index page.

This will make tools like easy_install read this link before it reaches the
egg/tarball.

This is OK as long as the users behind the firewalls are allowed to call
htppS...

so that's a PyPI server *and/or* setuptools issue


>   [cut]
>   - some auto-generated reports of access statistics and the mix
>     of distutils vs setuptools, those who "registered" w/o
>     "uploading", and perhaps if we get a new classifier assigned,
>     some idea of Python 2.x vs 3.x packages.
>

In this area, I have  proposed a few months ago to make the classifiers
permissive
see http://wiki.python.org/moin/EnhancedPyPI. Just wanted to say again, that
this work
is in my pile for turning PyPI errors into warnings when a classifier is
unknown.

The last thing I wanted to express in the state of packaging is that it took
me
a long time to understand where all pieces of code belonged to, between
distutils and setuptools.

I have found some bugs in distutils, and did add some issues, patches, and
discovered
after that setuptools had them resolved in its code.

I am feeling very frustrated on this because the boundary between the two
package
is very fuzzy. Maybe this is just a doc issue.

In the meantime, I might be naive but I
don't really understand why some part of setuptools are not merged into
distutils, when
it does the same things, but in a more robust way.

Cheers
Tarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080320/7a84e2af/attachment-0001.htm 

From p.f.moore at gmail.com  Thu Mar 20 10:33:53 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 20 Mar 2008 09:33:53 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
Message-ID: <79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>

On 20/03/2008, zooko <zooko at zooko.com> wrote:
> On Mar 19, 2008, at 3:23 PM, Guido van Rossum wrote:
>
>  > If other people want to chime in please do so; if this is just a
>  > dialog between Phillip and me I might incorrectly assume
>  > that nobody besides Phillip really cares.
>
> I really care.  I've used setuptools, easy_install, eggs, and
>  pkg_resources extensively for the past year or so (and contributed a
>  few small patches).  There have been plenty of problems, but I find
>  them to be overall useful tools.

I'll chime in here, too. I really want to like
setuptools/easy_install, but I don't. I'll try to be specific in my
reasons, in the hope that they can be addressed. I know some of these
are "known about", but one of my meta-dislikes of setuptools is that
known issues never seem to get addressed (I know, patches accepted,
but I haven't got the time either...)

1. No integration with the system packager (Windows, in my case). If I
do easy_install nose, then nose does not show up in add/remove
programs. That significantly affects the way I manage my PC.

2. No uninstaller. After easy_install nose, how do I get rid of it
later? Searching for files to delete (even if there are only a few
obviously named ones) is not good enough.

3. The pkg_resources documentation (in particular, that's the one I've
tried to follow) is extremely hard to read. Partly this is just style,
but it's partly because it is couched in very unfamiliar terms
(distributions, working sets, interfaces, providers, etc). It's also
*huge*. A tutorial style overview, supported by API detail, would be
far better.

4. Hard to use with limited connectivity. At work, I *only* have
access to the internet via Internet Explorer (MS based proxy). There
are workarounds, but ultimately "download an installer, then run it"
is a far simpler approach for me.

5. Auto-discovery doesn't always work. I'm sorry, I really can't
recall the example at the moment, but sometimes easy_install says it
can't find a package I *know* is available.

6. Splitting the community. Windows users rely heavily on binary
installers (at least, I do). We're starting to get a situation where
some projects provide .egg files, and some provide traditional
(bdist_wininst/bdist_msi) installers. This is bad. One way to do it,
and all that :-)

But if these problems are solved, then I have no problem with seeing
the features of setuptools added to the standard library - resource
APIs, plugin/entry point APIs, ways to create executable scripts, and
such things *should* be standardised. Dependency resolution and
automatic installation isn't something I like (probably because as a
Windows user I've never used such a system, so I mistrust it) but if
it works *with* the system and not against it, I don't mind.

I hope this helps,
Paul.

From mal at egenix.com  Thu Mar 20 11:42:48 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 20 Mar 2008 11:42:48 +0100
Subject: [Distutils] [Python-Dev] Consistent platform name for 64bit
 windows (was: distutils.util.get_platform() for Windows)
In-Reply-To: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>
References: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>
Message-ID: <47E23FA8.3090104@egenix.com>

On 2008-03-18 18:05, mhammond at keypoint.com.au wrote:
> I'm reviving a very old thread based on discussions with Martin at pycon.
> 
>> Sent: Monday, 23 July 2007 5:12 PM
>> Subject: Re: [Distutils] distutils.util.get_platform() for Windows
> 
> Rather than forcing everyone to read the context, allow me to summarize:
> On 64bit Windows versions, we need a "string" that identifies the
> platform, and this string should ideally be used consistently.  This
> original thread related to the files created by distutils (eg,
> pywin32-210.win???64??-py2.6.exe) but it seems obvious that we should be
> consistent wherever Python wants to display the platform (eg, in the
> startup banner, in platform.py, etc).
> 
> In the old thread, there was a semi-consensus that 'x86_64' be used by
> distutils (and indeed, Lib/distutils/util.py in get_platform() has been
> changed, by me, to use this string), but the Python 'banner', for example,
> reports AMD64.  Platform.py doesn't report much at all in this area, at
> least when pywin32 isn't installed, but it arguably should.
> 
> Both Martin and I prefer AMD64 as the string, for various reasons. 
> Firstly, it is less ugly than 'x86_64', and doesn't include an '_'/'-'
> which might tend to confuse parsing by humans or computers.  Martin also
> made the point that AMD invented the architecture and AMD64 is their
> preferred name, so we should respect that.
> 
> So, at the risk of painting a bike-shed, I'd like to propose that we adopt
> 'AMD64' in distutils (needs a change), platform.py (needs a change to use
> sys.getwindowsversion() in preference to pywin32, if possible, anyway),
> and the Python banner (which already uses AMD64).
> 
> Any objections?  Any strong feelings that using 'AMD' will confuse people
> with Intel processors?  Strong feelings about the parsability of the name
> (PJE? <wink>)?  Strong feelings about the color <wink>?

Not really an object, but Microsoft itself uses the term "x64" for
the 64-bit variants of their OS, e.g.

http://www.microsoft.com/windowsxp/64bit/default.mspx

Since the platform name is targeting Windows, I think we should
avoid confusing Windows users more than Intel users ;-)

About the platform.py changes: if someone could provide the return
values of sys.getwindowsversion() for 64bit versions of Windows
XP and Vista, I could add support for it (don't have a 64bit version
of Windows available to check myself).

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 20 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611

From martin at v.loewis.de  Thu Mar 20 12:37:32 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Mar 2008 06:37:32 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E1EAE4.90802@taupro.com>
References: <47E0D883.9030402@taupro.com>	<47E11659.9000307@v.loewis.de>	<47E19ABE.2040800@taupro.com>	<47E1BD14.2030805@v.loewis.de>
	<47E1EAE4.90802@taupro.com>
Message-ID: <47E24C7C.6080209@v.loewis.de>

> "... encouraged in __3.x guidelines__ to ...": I presume although I've not 
> found them yet that there is some kind of document for developers titled 
> something like, "how to migrate your Python code from 2.x to 3.x".  That 
> document would be a logical place for advice and consideration of the 
> tradeoffs of jumping to 3.x, maintaining two synced versions using 2to3 or 
> 3to2, and the risks of keeping two independent releases.  Identifying best 
> practices would help them make good choices for the community.

I don't think any of the core committers is qualified to write such
a document. Instead, it would have to be written by people who
*actually* ported a project from 2 to 3 in some form.

That is not to say that such a document couldn't be part of the 3k
release, or shouldn't be reviewed by core committers.

[Also, it might turn out that some of the core committers writes such
a document, with the theoretical background of what *could* work
for projects. That would be a lot like all those books giving advise
written by people who never followed their own advise because they
never had a chance to].

> So we don't have an actual success story of a dual-version distribution, even 
> as a prototype, using 2to3 within a distutils package?  I would not encourage 
> a practice without at least one such example.

We don't have any success story for Python 3, period. Nobody has ever
attempted to run a significant code base in Python 3, other than the
test suite, AFAIK.

>> It always worked fine for me, so I see no reason to fix it in the
>> first place.
> 
> Pardon my lack of knowledge of your background; when you say it always worked 
> fine for you, are you referring to personal experiences using it to _install_ 
> software or to experiences as a packager in actually distributing complex 
> collections of modules on different platforms?

I've been maintaining a larger project (PyXML) for several years, and
have written/maintained a few smaller projects (iconv, partial,
python-fam), which all used distutils. I have also extended distutils
in the core, with the upload and bdist_msi commands. And then
there is the experience with installing distutils-based packages,
which is usually pleasant (although I prefer to use the Debian
package where available)

Regards,
Martin

From mal at egenix.com  Thu Mar 20 13:55:21 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 20 Mar 2008 13:55:21 +0100
Subject: [Distutils] [Python-Dev] Consistent platform name for 64bit
 windows (was: distutils.util.get_platform() for Windows)
In-Reply-To: <frtm3g$mjl$1@ger.gmane.org>
References: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>	<47E23FA8.3090104@egenix.com>
	<frtm3g$mjl$1@ger.gmane.org>
Message-ID: <47E25EB9.3000301@egenix.com>

On 2008-03-20 13:42, Thomas Heller wrote:
> M.-A. Lemburg schrieb:
>> About the platform.py changes: if someone could provide the return
>> values of sys.getwindowsversion() for 64bit versions of Windows
>> XP and Vista, I could add support for it (don't have a 64bit version
>> of Windows available to check myself).
> 
> This is the output of a 32-bit Python running on "Windows XP Professional
> x64 Edition, Version 2003, Service Pack 2":
> 
> C:\Python24>ver
> 
> Microsoft Windows [Version 5.2.3790]
> 
> C:\Python24>python
> Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.getwindowsversion()
> (5, 2, 3790, 2, 'Service Pack 2')

Thank you !

Anyone with a 64bit Vista ?

Or even better: a page documenting what to expect from the system call
behind the sys.getwindowsversion() API ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 20 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611

From tseaver at palladion.com  Thu Mar 20 14:44:00 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 20 Mar 2008 09:44:00 -0400
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
Message-ID: <47E26A20.1090402@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Moore wrote:

> I'll chime in here, too. I really want to like
> setuptools/easy_install, but I don't. I'll try to be specific in my
> reasons, in the hope that they can be addressed. I know some of these
> are "known about", but one of my meta-dislikes of setuptools is that
> known issues never seem to get addressed (I know, patches accepted,
> but I haven't got the time either...)

Thanks for feedback from the Windows world, from whence I have been
blissfully exiled now for years.

> 1. No integration with the system packager (Windows, in my case). If I
> do easy_install nose, then nose does not show up in add/remove
> programs. That significantly affects the way I manage my PC.

Point taken. Of course, it isn't really a "program" at that point:  it
is an installed "add-on" to Python.  However, if Windows users expect
such add-ons to show up in the "system" list, that is good to know.

I'll note that I use easy_install *only* to work in *non-system*
locations:  if I want to install Python packages to /usr/lib/python2.x/,
I use the standard system installer, e.g. 'apt-get install
python-frobnatz'.  But I routinely create non-system Python environments
for development, using either alternate Pythons or virtualenv:  in those
environments, it works very well for me.

> 2. No uninstaller. After easy_install nose, how do I get rid of it
> later? Searching for files to delete (even if there are only a few
> obviously named ones) is not good enough.

People ask for this on Unix platforms as well, often adding a request
that pacakges installed only as dependencies of the
package-being-removed go away as well.  If you install everything in a
way that works with system package manager, of course, you don't need
this. ;)

Deleting the 'lib/python2.x/site-packages/foo-X.Y.X.egg' directory is
all that is actually required to uninstall an egg that was previouly
added via easy_install.  Cleaning out the equivalent entry in
'easy_install.pth' in that directory is not strictly required.

I wonder if a GUI for managing the add-ons would fit the bill, as an
alternative to packaging them as though they were standalone programs?

> 3. The pkg_resources documentation (in particular, that's the one I've
> tried to follow) is extremely hard to read. Partly this is just style,
> but it's partly because it is couched in very unfamiliar terms
> (distributions, working sets, interfaces, providers, etc). It's also
> *huge*. A tutorial style overview, supported by API detail, would be
> far better.

Many of those terms are distutils jargon, actually.  I think Jeff Rush'
recent work looks like a good start here.

> 4. Hard to use with limited connectivity. At work, I *only* have
> access to the internet via Internet Explorer (MS based proxy). There
> are workarounds, but ultimately "download an installer, then run it"
> is a far simpler approach for me.

I don't know how to make this requirement compatible with using shared
dependencies, except to make it easier for folks to download *all* the
requirements, and later install from the local "distribution cache" (a
directory full of .zip / .egg / .tgs files).  It does turn out to be
quite easy to build a PyPI-style "simple" index for such a cache.  Your
use case would then require:

 1. Run some command to fetch the desired package and the transitive
    closure of its dependencies into a working directory (the cache).

 2. Run another command to build an index for that directory.

 3. Run 'easy_install', pointing to the local index.

> 5. Auto-discovery doesn't always work. I'm sorry, I really can't
> recall the example at the moment, but sometimes easy_install says it
> can't find a package I *know* is available.

Usually this indicates that there are incompatible dependencies between
packages already installed and those on the index.  E.g., if I already
have package foo installed, but its version is not compatible with the
requirements for package bar, then I can't install bar, even though the
distribution is "available."

Because PyPI is not a centrally-managed index of packages, such
conflicts are pretty much inevitable over time for those who don't
subset it in some form (what we've been calling the "known good set"
strategy in Zope-land).

> 6. Splitting the community. Windows users rely heavily on binary
> installers (at least, I do). We're starting to get a situation where
> some projects provide .egg files, and some provide traditional
> (bdist_wininst/bdist_msi) installers. This is bad. One way to do it,
> and all that :-)

If it weren't for the "Add / Remove Programs" requirement you mentioned
above, we would be better off if authors of pure Python packages
uploaded only 'sdist' distributions, which can be cleanly converted to
platform-local eggs at install time, even on Windows.  Packages which
contain C extensions typically must upload the 'bdist_win' version for
the benefit of the vast majority of Windows users who can't bulid the
extensions locally.

Uploading any other binary distribution is pretty much a lose, because
the underlying platform dependencies (UCS2 vs UCS4, i386 vs x64,
framework vs. universal vs. MacPorts vs. Fink, etc) lead to
combinatorial expolosions and or segfaults.   Better to let the
installer fetch the source and build it locally.

> But if these problems are solved, then I have no problem with seeing
> the features of setuptools added to the standard library - resource
> APIs, plugin/entry point APIs, ways to create executable scripts, and
> such things *should* be standardised. Dependency resolution and
> automatic installation isn't something I like (probably because as a
> Windows user I've never used such a system, so I mistrust it) but if
> it works *with* the system and not against it, I don't mind.
> 
> I hope this helps,

Very much, thanks.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH4mof+gerLs4ltQ4RApBLAJwI0Be1CtSKgpAYDEyH2qd0K+a+6QCeN/cf
5Pg43ot4H954A87ZWIouwLo=
=S4yF
-----END PGP SIGNATURE-----


From lxander.m at gmail.com  Thu Mar 20 14:50:58 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Thu, 20 Mar 2008 09:50:58 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
	Packaging/Deployment Technology Concerns
In-Reply-To: <47E1908F.5030501@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E0D571.2070004@taupro.com>
	<20080319152101.730BC3A40A5@sparrow.telecommunity.com>
	<47E1908F.5030501@taupro.com>
Message-ID: <525f23e80803200650w555fd368va19d749b429d3969@mail.gmail.com>

On Wed, Mar 19, 2008 at 6:15 PM, Jeff Rush <jeff at taupro.com> wrote:
>  Frankly I'd like to see setuptools exploded, with those parts of general use
>  folded back into the standard library, the creation of a set of
>  non-implementation-specific documents of the distribution formats and
>  behavior, leaving a small core of one implementation of how to do it and the
>  door open for others to compete with their own implementation.

If I hazard an opinion seconding this sentiment. In my use of
setuptools, it definitely feels like it wants to be three (mostly)
independent projects:

1) The project that standardizes the concept now embodied by eggs and
provides the basic machinery to work with them (find them, introspect
metadata, "import" them, etc.), but not install them per se. This is
generally useful as common plug-in framework, if nothing else.
Currently, this "run-time support" functionality is in pkg_resources.
2) The tool you can use to build eggs (but not install them per se).
Currently this is the setuptools extension to distutils.
3) The tool for installing eggs (or their equivalent) and (optionally)
their dependencies (optionally using remote hosts) as well as
uninstalling. Currently this is easy_install (well, except for
uninstalling, which is understandable quite difficult).

Finally, there is the fourth and already separate project of PyPI:
4) The hosted repository of publicly available eggs (or their
equivalent). This should export any metadata required to resolve
dependencies relatively cheeply.

Breaking them apart will make it easier to have two separate projects
for building eggs (or their equivalents) -- one based on distutils and
the other replacing it. Even more importantly, it will make it
possible for multiple installers to be developed that scratch
particular itches. Hopefully one would eventually emerge as the
de-facto standard, but this will ultimately be decided by community
adoption.

Alex

From janssen at parc.com  Thu Mar 20 15:07:45 2008
From: janssen at parc.com (Bill Janssen)
Date: Thu, 20 Mar 2008 07:07:45 PDT
Subject: [Distutils] [Python-Dev] platform management
In-Reply-To: <ca471dc20803192357n3c8a7a7fq3d72f656c0f418ef@mail.gmail.com> 
References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E160CEE20@EXMBX04.exchhosting.com>
	<-3692898589838545524@unknownmsgid>
	<ca471dc20803192357n3c8a7a7fq3d72f656c0f418ef@mail.gmail.com>
Message-ID: <08Mar20.070752pdt."58696"@synergy1.parc.xerox.com>

Looking at http://docs.python.org/lib/module-os.html, I find the following:

  name
  
    The name of the operating system dependent module imported. The
    following names have currently been registered: 'posix', 'nt', 'mac',
    'os2', 'ce', 'java', 'riscos'.

This implies that there's a registry somewhere?

Bill

> Great idea! Sounds like a PEP (informational, probably) would be good idea.
> 
> On Tue, Mar 18, 2008 at 4:59 PM, Bill Janssen <janssen at parc.com> wrote:
> > I don't think this is bike-shedding.
> >
> >  The debate about "AMD64" vs. "amd64" vs. "x86_64" reminded me that
> >  I've been bit more and more frequently by bits of platform-specific
> >  knowledge scattered around the standard library.  The latest is the
> >  code in distutils.unixccompiler that tries to figure out what flags to
> >  send to the linker in order to add a dynamic library path lookup to a
> >  shared library.
> >
> >  There are lots of different ways of figuring out which platform is
> >  being used, and they're all over the place.  The code in
> >  distutils.unixccompiler uses "sys.platform[:6]", and looks for
> >  "darwin"; the code in urllib.py uses "os.name", and expects it to be
> >  "mac", but later looks for "sys.platform == 'darwin'; posixfile
> >  believes that platforms come with version numbers ("linux2", "aix4");
> >  pydoc and tarfile have tests for "sys.platform == 'mac'".  tempfile
> >  looks at os.sys.platform *and* os.name.
> >
> >  Could well be that all of these are correct (I believe that "mac", for
> >  instance, refers to the generations of the Mac OS before OS X).  But
> >  it means that when someone tries to update "Python" to a new major
> >  version release for, say, OS X, it's really easy to miss things.  And
> >  the fact that the platform version is sometimes included and sometimes
> >  not is also problematic; darwin9 is different from darwin8 in some
> >  important aspects.
> >
> >  It would be nice to
> >
> >   (a) come up with a list of standard platform symbols,
> >   (b) put those symbols somewhere in the documentation,
> >   (c) have one way of discovering them, via sys or os or platform or
> >       whichever module,
> >   (d) add a standard way of discovering the platform version, and
> >   (e) stamp out all the other ways that have crept in over the years.
> >
> >  Bill
> >  _______________________________________________
> >  Python-Dev mailing list
> >  Python-Dev at python.org
> >  http://mail.python.org/mailman/listinfo/python-dev
> >  Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> 
> 
> 
> -- 
> --Guido van Rossum (home page: http://www.python.org/~guido/)


From bkline at rksystems.com  Thu Mar 20 14:56:02 2008
From: bkline at rksystems.com (Bob Kline)
Date: Thu, 20 Mar 2008 09:56:02 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E26A20.1090402@palladion.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com>
Message-ID: <47E26CF2.1090701@rksystems.com>

Tres Seaver wrote:
> Point taken. Of course, it isn't really a "program" at that point:  it
> is an installed "add-on" to Python.  However, if Windows users expect
> such add-ons to show up in the "system" list, that is good to know.
>   

Are things really that different in the non-Windows worlds?  If I want 
python-nose, I run "sudo apt-get install python-nose" (and that means I 
can always remove it with "sudo apt-get remove ...").  Seems more 
similar than different (ignoring the silliness of Microsoft's insistence 
on "the GUI is the OOWTDI" even for such administrative tasks as 
installing system-wide software).

-- 
Bob Kline
http://www.rksystems.com
mailto:bkline at rksystems.com


From mail at timgolden.me.uk  Thu Mar 20 15:10:41 2008
From: mail at timgolden.me.uk (Tim Golden)
Date: Thu, 20 Mar 2008 14:10:41 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E26CF2.1090701@rksystems.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>	<47E26A20.1090402@palladion.com>
	<47E26CF2.1090701@rksystems.com>
Message-ID: <47E27061.8080709@timgolden.me.uk>

Bob Kline wrote:
> Are things really that different in the non-Windows worlds?  If I want 
> python-nose, I run "sudo apt-get install python-nose" (and that means I 
> can always remove it with "sudo apt-get remove ...").  Seems more 
> similar than different (ignoring the silliness of Microsoft's insistence 
> on "the GUI is the OOWTDI" even for such administrative tasks as 
> installing system-wide software).

I was going to -- pointedly -- drop in here the help output
for msiexec, which is the commandline version of the MSI
installation graphical stuff. Only... when I did msiexec /?,
the result was that a Window popped up with the information
in it. (Sort of agrees with your point a bit!)

Still, here's the info (cut-and-pasted from that window):

-----

Windows ? Installer. V 3.01.4000.1823

msiexec /Option <Required Parameter> [Optional Parameter]

Install Options
	</package | /i> <Product.msi>
		Installs or configures a product
	/a <Product.msi>
		Administrative install - Installs a product on the network
	/j<u|m> <Product.msi> [/t <Transform List>] [/g <Language ID>]
		Advertises a product - m to all users, u to current user
	</uninstall | /x> <Product.msi | ProductCode>
		Uninstalls the product


[... snip lots of other options ...]

TJG

From martin at v.loewis.de  Thu Mar 20 15:28:18 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Mar 2008 09:28:18 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803192208q1aca7684o681f542d3d901b32@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>	
	<20080317184553.913413A409D@sparrow.telecommunity.com>	
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	
	<20080318223700.C64123A4074@sparrow.telecommunity.com>	
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	
	<47E1B858.1090107@v.loewis.de>
	<ca471dc20803192208q1aca7684o681f542d3d901b32@mail.gmail.com>
Message-ID: <47E27482.9090908@v.loewis.de>

Guido van Rossum schrieb:
> I was using the human interface at python.org/pypi. There are two
> prominent links at the top of the page: "Browse the tree of packages"
> and "Submit package information" followed by the 30 most recently
> changed packages.

Ah, ok. In PyPI parlance, these are "classifiers" (Trove classifiers,
although the word "trove" means nothing to me), not keywords. They
are different from keywords in the sense that they form a hierarchy.

I personally consider trove classifiers over-valued, but apparently,
some people really love them (probably the ones who are more organized
than I am). Developers continuously request addition of new classifiers;
I don't have any statistics whether users actually use them to locate
stuff.

> What I was looking for was the page for a specific
> package. The "Browse the tree of packages" link was no help. Finally I
> realized that in the side bar, in a small unobtrusive font, is a link
> to "List packages" which links to a list of *all* packages, in
> alphabetical order. I found my package there. I think repeating that
> link right below "browse the tree" would have been sufficient. 

I can't change that right now, but created

http://sourceforge.net/tracker/index.php?func=detail&aid=1921108&group_id=66150&atid=513503

> But it
> would have been cool if there had been a search box (also in the start
> page) where I could type (part of) the name of the package and it
> would have given me the nearest matches.

Did you try the search box in the top-right, and did did not work?

What search term did you enter, and what package did you expect to get?

Regards,
Martin


From pje at telecommunity.com  Thu Mar 20 16:07:55 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Mar 2008 11:07:55 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E1EEE6.5050608@palladion.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<47E1EEE6.5050608@palladion.com>
Message-ID: <20080320150754.9F6503A40A5@sparrow.telecommunity.com>

At 12:58 AM 3/20/2008 -0400, Tres Seaver wrote:
>A lot of setuptools warts are driven by related design problems in the
>distutils, such as the choice to use imperative / procedural code for
>everything:  a declarative approach, with hooks for cases which actually
>need them (likely 5% of existing packages) would have made writing tools
>on top of the framework much simpler.  It is ironic that Python is *too
>powerful* a tool for the tasks normally done by distutils / setuptools:
>  a more restricted, and thererfore introspectable, configuration-driven
>approoach seems much cleaner.

+1


From pje at telecommunity.com  Thu Mar 20 16:18:04 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Mar 2008 11:18:04 -0400
Subject: [Distutils] [Distutils-SIG] PYTHONPATH installation (was Re: PEP
 365 (Adding the pkg_resources module))
In-Reply-To: <EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
Message-ID: <20080320151806.F0A223A40A5@sparrow.telecommunity.com>

At 10:18 PM 3/19/2008 -0600, zooko wrote:
>The fact that easy_install creates a site.py that changes the
>semantics of PYTHONPATH is probably the most widely and deservedly
>hated example of this kind of thing [2].

Yep, this was an unfortunate side effect of eggs growing outside 
their original ecological niche.  Without the 'site' hack, it was 
impossible to install eggs to user directories and avoid installation 
conflicts.

Specifically, if someone installed a package to PYTHONPATH with the 
distutils, and then installed a later version using setuptools, the 
setuptools-installed version would always end up on sys.path *after* 
the distutils-installed version.  Detecting this condition and 
handling it properly was a major problem for users of easy_install, 
who wanted it to "just work".

Standardization of a PEP 262-style installation database is still 
needed to address these problems, not to mention 
uninstallation.  Maybe now with some package manager folks paying 
some attention here, we can do something about that.


>[2] http://www.rittau.org/blog/20070726-02
>    And no, PJE's suggested "trivial fix" does not satisfy the
>objectors, as it can't support the use case of "cd somepkg ; python 
>./ setup.py install ; cd .. ; python -c 'import somepkg'".

Well, it replaces the hack being complained about, with the problem 
that the hack was introduced to fix.  :)

Again, to properly fix this, we need a metadata standard for who owns 
what packages -- and it should probably include information about the 
*tool* that did the installation, so that system packagers can either 
tell Python-level tools to keep their hands off, or tell Python how 
to run the tool in question.


From pje at telecommunity.com  Thu Mar 20 16:25:18 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Mar 2008 11:25:18 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
Message-ID: <20080320152518.B79263A40A5@sparrow.telecommunity.com>

At 09:33 AM 3/20/2008 +0000, Paul Moore wrote:
>1. No integration with the system packager (Windows, in my case). If I
>do easy_install nose, then nose does not show up in add/remove
>programs. That significantly affects the way I manage my PC.

The long-term fix here is probably to have a platform-specific 
installer capable of either turning eggs into .msi or .exe 
installers, or of doing the add/remove programs integration 
directly.  (Someone, of course, will have to step up to create such a tool.)


>5. Auto-discovery doesn't always work. I'm sorry, I really can't
>recall the example at the moment, but sometimes easy_install says it
>can't find a package I *know* is available.

Sometimes it does that to me, too.  But then I look at the project's 
page in PyPI, and they don't have a link to a download 
page.  Usually, they've got a link to a page on their site with 
instructions about downloading, but that doesn't directly link to any 
tarballs or anything.  So I grab the link of the real download page 
and paste it into a -f option to easy_install, so it knows where to 
find the link from.



From pje at telecommunity.com  Thu Mar 20 16:31:20 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Mar 2008 11:31:20 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E26A20.1090402@palladion.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com>
Message-ID: <20080320153120.7384B3A40A5@sparrow.telecommunity.com>

At 09:44 AM 3/20/2008 -0400, Tres Seaver wrote:
>I don't know how to make this requirement compatible with using shared
>dependencies, except to make it easier for folks to download *all* the
>requirements, and later install from the local "distribution cache" (a
>directory full of .zip / .egg / .tgs files).  It does turn out to be
>quite easy to build a PyPI-style "simple" index for such a cache.  Your
>use case would then require:
>
>  1. Run some command to fetch the desired package and the transitive
>     closure of its dependencies into a working directory (the cache).
>
>  2. Run another command to build an index for that directory.
>
>  3. Run 'easy_install', pointing to the local index.

Actually, if someone were to develop a patch for PyPI to do this, we 
could perhaps have a "display download dependencies" link for eggs 
shown on PyPI.  That way, someone who wants to do a manual download 
could get a page with links for all the required eggs, and manually 
download them.

(Of course, the other alternative would be for someone to provide an 
IE-controlling extension to urllib2 so that easy_install wouldn't be 
proxy-bound on such machines.) 


From asmodai at in-nomine.org  Thu Mar 20 16:43:55 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Thu, 20 Mar 2008 16:43:55 +0100
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E27482.9090908@v.loewis.de>
References: <20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<47E1B858.1090107@v.loewis.de>
	<ca471dc20803192208q1aca7684o681f542d3d901b32@mail.gmail.com>
	<47E27482.9090908@v.loewis.de>
Message-ID: <20080320154355.GI60713@nexus.in-nomine.org>

-On [20080320 15:29], "Martin v. L?wis" (martin at v.loewis.de) wrote:
>(Trove classifiers, >although the word "trove" means nothing to me)

Isn't that something lifted from SourceForge?

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/
The Eyes of Truth are always watching you...

From asmodai at in-nomine.org  Thu Mar 20 16:45:51 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Thu, 20 Mar 2008 16:45:51 +0100
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E1EEE6.5050608@palladion.com>
References: <20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<47E1EEE6.5050608@palladion.com>
Message-ID: <20080320154551.GJ60713@nexus.in-nomine.org>

-On [20080320 05:58], Tres Seaver (tseaver at palladion.com) wrote:
>I think that, warts an all, setuptools is a *huge* improvement over bare
>distutils for nearly every use case I know about.

Agreed.

I see setuptools (along with PyPI - hopefully much better in near future
though) as the Python equivalent to CPAN and RubyGems.

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/
Sadness is the inner beauty of the untouched tear...

From acapnotic at foobox.net  Thu Mar 20 17:08:31 2008
From: acapnotic at foobox.net (Kevin Turner)
Date: Thu, 20 Mar 2008 09:08:31 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
 module)
In-Reply-To: <EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
Message-ID: <1206029311.9502.91.camel@grinky>

On Wed, 2008-03-19 at 22:18 -0600, zooko wrote:
> 1.  "The very notion of package dependency resolution and  
> programmable or command-line installation of packages at the language  
> level is a bad notion."
> 
> This can't really be the case.  If the existence of such  
> functionality at the programming language level were an inherently  
> bad notion, then we would be hearing some complaints from the Ruby  
> folks, where the Gems system is standard and ubiquitous.  We hear no  
> complaints -- only murmurs of satisfaction.

Okay then, just to fill out your sample -- as the maintainer of a Python
library which is ported to Ruby, I complain equally about eggs and gems.
This isn't really the place for it, but as near as I can tell, the use
of gems requires you to know whether the user has installed your
dependency in the system install or through a gem *at the time you write
your code*, so you know whether to write "require 'dep'" or "require
'rubygems'; gem 'dep'".  This is, IMHO, even worse than the "setuptools
breaks PYTHONPATH" complaint you cited.

> Note that Ruby software is not too hard to include in operating  
> system packaging schemes -- my Ubuntu Hardy apt-cache shows plenty of  
> Ruby software.

Yes, but that software is not installed using the gem management system,
as I confirmed with a recent conversation with my package manager while
we were talking about http://bugs.debian.org/470282 , a quirk which was
hopefully a one-time API breakage, but certainly has not endeared me to
rubygems any further.

I'm sure we could find other people to complain if we look around a
little more.  I know I have commiserators out there.

But, stepping back a bit:

You're right in believing that it is neigh impossible to distribute Ruby
software without providing gems.  So much of your userbase expects it,
especially when you're distributing a library which their applications
will in turn depend on, because *their* users will expect gems, and they
need to be able to use gems to install the dependency.

setuptools seems to perform slightly better here, as, by merely making
sure my pypi entry has a reachable download_url, my package seems to be
available for installation by setuptools users.  Nonetheless, I get a
recurring stream of requests for egg distribution from people who
believe eggs have manifest destiny, and as we heard recently, that "the
controversy is over."

Meanwhile, I beg their continued forgiveness for being hesitant to
require my users to use something not in the standard library for
something as fundamental as "setup.py install."

These folks are the same who gave me bug reports when I put a .tar.bz2
link to my pypi entry, because apparently -- even though bz2 extraction
has been a feature of GNU tar for years -- setuptools (which uses the
standard library tarfile module) on some platforms cannot uncompress bz2
packages.  

the conclusion I am trying to reach here is this:

as a Python package maintainer, I have no idea what the hell to do to
satisfy my users, from those who are using python 2.3 and have no desire
for any new packaging or import semantics, to those who don't mind
having a new ez_setup downloaded on install.  The people who have found
advantages to using the egg-based distribution system are not going
away.  Providing something in the standard library will provide clear
guidance for me, and relieve me of the fear that I am pushing surprising
(<cough>.pth</cough>) or non-standard installation behavior on my users.

so, I hope you work something out.

Love,

 - Kevin


From janssen at parc.com  Thu Mar 20 17:12:51 2008
From: janssen at parc.com (Bill Janssen)
Date: Thu, 20 Mar 2008 09:12:51 PDT
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E27482.9090908@v.loewis.de> 
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<47E1B858.1090107@v.loewis.de>
	<ca471dc20803192208q1aca7684o681f542d3d901b32@mail.gmail.com>
	<47E27482.9090908@v.loewis.de>
Message-ID: <08Mar20.091252pdt."58696"@synergy1.parc.xerox.com>

> although the word "trove" means nothing to me

http://www.askoxford.com/concise_oed/trove?view=uk

"a store of valuable or delightful things"

Bill

From martin at v.loewis.de  Thu Mar 20 17:31:50 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Mar 2008 11:31:50 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E26A20.1090402@palladion.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com>
Message-ID: <47E29176.9050109@v.loewis.de>

> I'll note that I use easy_install *only* to work in *non-system*
> locations:  if I want to install Python packages to /usr/lib/python2.x/,
> I use the standard system installer, e.g. 'apt-get install
> python-frobnatz'. 

This is probably not the Windows way of doing things (at least not how
I use Windows). Windows doesn't really have the notion of "system
location" (or multiple levels of them, where \Windows and 
\Windows\system32 is "more system" than \Program Files, say).

Windows users typically view the entire system as "theirs", and
have no concerns at all about putting things into Program Files,
system32, or, for that matter, \python25. In fact, they don't care
or even know where stuff ends up - they expect that the system will
find it, and they expect that they can remove anything they installed
even without known where it is - because there is a standard place
to look for uninstalling things.

Of course, setuptools is not the only piece of software that doesn't
play well, so Windows users collect all kinds of cruft over time.
Eventually, C: will run out of disk space, and they either get a
new machine, or reinstall from scratch.

> I wonder if a GUI for managing the add-ons would fit the bill, as an
> alternative to packaging them as though they were standalone programs?

On Windows, it is fairly easy to have an uninstaller registered. There
are wrappers to managing that (such as MSI), but it's really only a
set of registry keys that needs to get written on installation time,
one of them being the command to run on uninstallation.

Assuming that you uninstall the package before uninstalling Python, that
uninstall program could be a Python script (although using a cmd.exe
batch file would probably be more resilient).

The concern with "you just need to delete the folder" is "how am I
supposed to know that? and can I be really sure?". If you run the
official uninstall procedure, and it messes things up, you can complain
to setuptools, or the package author that uninstallation "doesn't work".

If you delete stuff manually, and you forgot to remove something in
a remote location you didn't even know it existed, you still think
it's your own fault. So people are hesitant to actually execute the
procedure.

Of course, once you *do* provide an entry to "Add/Remove Programs",
uninstalling won't be mere deletion, as mere deletion would still
leave these registry keys behind (although Windows got more resilient
over time to provide cleanup in that case: I believe it offers to
remove the ARP entry if the uninstall program has been removed)


Regards,
Martin

From martin at v.loewis.de  Thu Mar 20 17:42:26 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Mar 2008 11:42:26 -0500
Subject: [Distutils] [Python-Dev] platform management
In-Reply-To: <08Mar20.070752pdt."58696"@synergy1.parc.xerox.com>
References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E160CEE20@EXMBX04.exchhosting.com>	<-3692898589838545524@unknownmsgid>	<ca471dc20803192357n3c8a7a7fq3d72f656c0f418ef@mail.gmail.com>
	<08Mar20.070752pdt."58696"@synergy1.parc.xerox.com>
Message-ID: <47E293F2.2070000@v.loewis.de>

> Looking at http://docs.python.org/lib/module-os.html, I find the following:
> 
>   name
>   
>     The name of the operating system dependent module imported. The
>     following names have currently been registered: 'posix', 'nt', 'mac',
>     'os2', 'ce', 'java', 'riscos'.
> 
> This implies that there's a registry somewhere?

This is actually the list of names that the "os" module may take.
There are different implementations of the os module, so instead of
"import os", you could write "import posix", "import nt", "import ce"
(assuming you run on one of these systems).

So it really has not much to do with the name of the operating system,
but rather with the name Python gives to the API.

Regards,
Martin

From pwang at enthought.com  Thu Mar 20 17:46:36 2008
From: pwang at enthought.com (Peter Wang)
Date: Thu, 20 Mar 2008 11:46:36 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E29176.9050109@v.loewis.de>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com> <47E29176.9050109@v.loewis.de>
Message-ID: <C8BCCA3F-7BB8-4D11-AE07-EE9D1313FBD3@enthought.com>

On Mar 20, 2008, at 11:31 AM, Martin v. L?wis wrote:

>> I'll note that I use easy_install *only* to work in *non-system*
>> locations:  if I want to install Python packages to /usr/lib/ 
>> python2.x/,
>> I use the standard system installer, e.g. 'apt-get install
>> python-frobnatz'.
>
> This is probably not the Windows way of doing things (at least not how
> I use Windows). Windows doesn't really have the notion of "system
> location" (or multiple levels of them, where \Windows and
> \Windows\system32 is "more system" than \Program Files, say).
>
> Windows users typically view the entire system as "theirs", and
> have no concerns at all about putting things into Program Files,
> system32, or, for that matter, \python25. In fact, they don't care
> or even know where stuff ends up - they expect that the system will
> find it, and they expect that they can remove anything they installed
> even without known where it is - because there is a standard place
> to look for uninstalling things.

While these observations are accurate for most home users, it is worth  
noting that many IT departments deploy locked-down versions of windows  
that either have fine-grained group policies to forbid modifications  
to the system disk (and require the user to write things to a mounted  
network home directory), or that give write access to the system disk  
but then re-image it upon reboot.

IT departments that deploy this sort of setup usually have the  
"hostile user" mentality, and that is strongly correlated, in turn,  
with users that are reluctant to engage IT to allow them install an  
app.  We have run into this a few times, and it would be good to keep  
this scenario in mind.


-Peter


From martin at v.loewis.de  Thu Mar 20 17:48:59 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 20 Mar 2008 11:48:59 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080320153120.7384B3A40A5@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>	<47E26A20.1090402@palladion.com>
	<20080320153120.7384B3A40A5@sparrow.telecommunity.com>
Message-ID: <47E2957B.50501@v.loewis.de>

> Actually, if someone were to develop a patch for PyPI to do this, we 
> could perhaps have a "display download dependencies" link for eggs 
> shown on PyPI.  That way, someone who wants to do a manual download 
> could get a page with links for all the required eggs, and manually 
> download them.

Just to make this position a bit more official (as one of the PyPI
maintainers): it would be fully within the scope of PyPI to integrate
dependency tracking into its database, and present it in any form
that is desired. Any such feature would have to be contributed.

Regards,
Martin


From mal at egenix.com  Thu Mar 20 18:43:37 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 20 Mar 2008 18:43:37 +0100
Subject: [Distutils] [Python-Dev] Consistent platform name for 64bit
 windows (was: distutils.util.get_platform() for Windows)
In-Reply-To: <47E25EB9.3000301@egenix.com>
References: <3504.63.250.241.10.1205859937.squirrel@mail.eftel.com>	<47E23FA8.3090104@egenix.com>	<frtm3g$mjl$1@ger.gmane.org>
	<47E25EB9.3000301@egenix.com>
Message-ID: <47E2A249.3010109@egenix.com>

On 2008-03-20 13:55, M.-A. Lemburg wrote:
> On 2008-03-20 13:42, Thomas Heller wrote:
>> M.-A. Lemburg schrieb:
>>> About the platform.py changes: if someone could provide the return
>>> values of sys.getwindowsversion() for 64bit versions of Windows
>>> XP and Vista, I could add support for it (don't have a 64bit version
>>> of Windows available to check myself).
>> This is the output of a 32-bit Python running on "Windows XP Professional
>> x64 Edition, Version 2003, Service Pack 2":
>>
>> C:\Python24>ver
>>
>> Microsoft Windows [Version 5.2.3790]
>>
>> C:\Python24>python
>> Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on win32
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import sys
>>>>> sys.getwindowsversion()
>> (5, 2, 3790, 2, 'Service Pack 2')
> 
> Thank you !
> 
> Anyone with a 64bit Vista ?
> 
> Or even better: a page documenting what to expect from the system call
> behind the sys.getwindowsversion() API ?

FYI: I added winreg and sys.getwindowsversion() support in r61674.

platform.machine() and .processor() will now use the environment
variables PROCESSOR_ARCHITECTURE and PROCESSOR_IDENTIFIER where
available (should work on Windows XP and later).

According to http://support.microsoft.com/kb/888731 platform.machine()
will return "AMD64", so I guess the "x64" string is just a marketing
name for 64-bit platforms on Windows and the underlying system uses
"AMD64" as machine type name.

For x86 processors, you'll now get "x86" on Windows XP and later.

For Itanium processors, you should get "IA64" according to this
WOW64 page:

http://msdn2.microsoft.com/en-us/library/aa384274(VS.85).aspx

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 20 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611

From stephen at xemacs.org  Thu Mar 20 19:14:37 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 21 Mar 2008 03:14:37 +0900
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E24C7C.6080209@v.loewis.de>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
	<47E19ABE.2040800@taupro.com> <47E1BD14.2030805@v.loewis.de>
	<47E1EAE4.90802@taupro.com> <47E24C7C.6080209@v.loewis.de>
Message-ID: <871w65gt4i.fsf@uwakimon.sk.tsukuba.ac.jp>

"Martin v. L?wis" writes:

 > I don't think any of the core committers is qualified to write such
 > a document. Instead, it would have to be written by people who
 > *actually* ported a project from 2 to 3 in some form.

Note that this is precisely the kind of experience Skip Montanaro is
talking about trying to generate in the context of SpamBayes in a
thread on the python-3000 list entitled "Strategy for porting to 3.0?"
I don't know if he plans to write a HOWTO himself, but he certainly
intends to keep a lab notebook that will be of help in writing such a
document.


From dpeterson at enthought.com  Thu Mar 20 21:25:44 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Thu, 20 Mar 2008 15:25:44 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <20080319012046.7B1233A4074@sparrow.telecommunity.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E05DD5.1090707@enthought.com>
	<20080319012046.7B1233A4074@sparrow.telecommunity.com>
Message-ID: <47E2C848.7090400@enthought.com>

Phillip J. Eby wrote:
> We should probably move this off of Python-Dev, as we're getting into 
> deep details now...

Done.  Only in distutils-sig now.


> At 07:27 PM 3/18/2008 -0500, Dave Peterson wrote:
>> If you really wanted to do a full-tree intersection, it seems to me 
>> that the problem is detecting all the dependencies without having to 
>> spend significant time downloading/building in order to find them 
>> out.   This could be solved by simply extending the cheeseshop 
>> interface to export the set of requirements outside of the egg / 
>> tarball / etc.  We've done this for our own egg repository by 
>> extracting the appropriate meta-data files out of EGG-INFO and 
>> putting it into a separate file.  This info is also useful for users 
>> as it gives them an idea of how much *new* stuff is going to be 
>> installed (a la yum, apt-get, etc.)
>
> ...and now we're more directly competing with them, too.  The original 
> idea Bob and I had was to do XML files ala Eclipse feature 
> repositories, but then later I realized that for what we were doing, 
> HTML was both adequate and already available.  However, I don't see a 
> problem in principle with having "header" files available for this 
> sort of thing.

It seems from latter discussions that Martin v. L?wis agrees that this 
is a reasonable thing to do.  I'll see if I can find time to work on a 
patch to PyPI.  Not having looked at that code before at all, it might 
take me awhile.


>> With our ETS projects, we've run into problems with the current 
>> heuristic.  Perhaps we just don't know how to make it work like we want?
>>
>> We have a set of projects that we want to be individually installable 
>> (to the extent that we limit cross-project dependencies) but we also 
>> want to make it easy to install the complete set.  We use a meta-egg 
>> for the latter.  It's purpose is only to specify the exact versions 
>> of each project that have been explicitly tested to work together -- 
>> you could almost think of it as a source control system tag.
>
> I would think that as long as that meta-egg specifies *all* the 
> required versions (right down to recursive dependencies), then there 
> shouldn't be any problem.  Maybe it's me who's not understanding 
> something?

It actually does specify all the required versions, including those 
recursive dependencies, but we were still getting breakages when new 
versions were released. :-(   I think I explained what we were seeing in 
my original e-mail though it sounds like you're saying that shouldn't be 
possible, right?


> I would think that you could get the appropriate data by running the 
> tl.eggdeps tool.

Getting the version data isn't a problem at all for us, but thanks for 
the pointer to an interesting project.  (We have an internal project 
that actually analyzes import statements within the code within a 
project to ensure that the documented dependencies in a setup.py match 
the declared ones and this solves the problem for us.)



>> A number of projects want to provide various types of files besides 
>> code in their distributable, and they'd like these to end up in 
>> standard locations for that type of file.  Think documentation, 
>> sample data, web templates, configuration settings, etc.   Each of 
>> these should be treated differently at installation time depending on 
>> platform.  On *nix, docs should go in /usr/share/doc whereas we might 
>> need to create a C:\Python2.5\docs on Windows.   With sample data and 
>> templates, you probably just want it accessible outside of the zipped 
>> egg so users can easily look at it, add to it, edit it, etc.  
>> Configuration settings should be installed with some defaults into a 
>> standard configuration directory like /etc on *nix, etc.
>>
>> Basically the issue is that it needs to be easier to include 
>> different sets of files into an egg for different actions to be taken 
>> during installation or packaging into an OS-specific distribution 
>> format.
>
> Yes, it would be nice to define a metadata standard for including 
> installable "datasets" either through copying or symlinking, 
> optionally with entry points for running some code, too.  When you 
> install an egg, these things could get added to a "post-install to-do" 
> list, that you could then read to find out what steps to do, or invoke 
> a tool on to actually do some of those steps.

I agree.   Let's get that setuptools wiki started and start documenting 
some of these ideas as a roadmap so that anyone who wants to help out 
has an idea of what to work on, or factor into what they're currently 
working on.


>> But the docs for easy_install claim that the list of active eggs is 
>> maintained in easy-install.pth.  Also, if I create my own .pth file, 
>> and the user tries to update my version to a new one, will the 
>> easy_install tool modify my .pth file to remove the mention of the 
>> old version from my sys.path and put the new version in the same .pth 
>> file?  Or will it now be listed in both places?  Or will it only in 
>> easy-install.pth?
>
> My understanding of the context of the question was that it applied to 
> *system* packaging tools, which would be exclusively maintaining the 
> .pth entries for the packages they installed.  i.e., a scenario with 
> *no* easy-install.pth.  Setuptools will still detect the presence of 
> their eggs, regardless of the means by which they're added to 
> sys.path.  But it would not *maintain* those .pth files.

I may be confusing the issue then.  I was under the impression that 
system packaging tools would want to install things such that anyone 
used to using setuptools would be able to see the effects of that 
installation in the same way as if it was done via easy_install. i.e. if 
I wanted to temporarily remove it for testing something or other, I 
could de-activate it; or if I wanted to install a second optional 
version of it, I could use easy_install to do so without worrying about 
tracking down the right .pth file.



>> Yes, but as you've already pointed out, they've escaped into a larger 
>> ecosystem and this restriction is a severe limitation -- leading to 
>> significant frustration.  Especially as projects evolve and want to 
>> do something more complex than simply install pure Python code.  Here 
>> at Enthought, we use and ship a number of projects that have 
>> extensions and thus dynamic libraries that need to either be modified 
>> during installation to work from the user's installed location, or 
>> copied elsewhere on the system to avoid the need to modify (which we 
>> also can't do via an egg install) env variables, registries, etc.
>
> By the way, there *is* experimental shared library building support in 
> setuptools, and I recently heard from Andi Vajda that he was 
> successful in using it in his JCC project to make available a C++ 
> library for linkage from JCC-built projects.  (I'm also sitting on his 
> patch that makes it work...)  I'm not sure that it actually fixes the 
> larger problem, in that e.g., if the main project is installed by the 
> system, and then you build or install an egg yourself.  But I think 
> those problems are solvable.

I'm not sure your description matches what we're trying to do here, but 
I can figure that out better from looking at the code.   Is this in the 
0.6 versions or 0.7a?  And where should I start looking at module-wise?


>>    We'd also love to be able to ship end-user enterprise-scale 
>> applications via eggs so that bug fixes and updates don't require 
>> downloading a monolithic 100MB+ installer.  But doing that requires 
>> the ability to update desktop icons, menus, etc. which we also can't 
>> do automatically via an egg.
>
> Yep...  a good post-install mechanism would be handy for wx and 
> pywin32 as well.

Enthought has started a project to provide an API abstraction of doing 
the desktop icon / menu setup on multiple platforms (Windows, Gnome, 
KDE, and hopefully soon OSX) for both system and user installs.   We use 
it in our EPD product (Enthought Python Distribution.)  We could 
probably work on getting this into a more public form with some hints as 
to whether it should be done as a plugin, patch, separate project, etc.


>> If you don't want the burden on setuptools to support, much less 
>> track, all these options, then perhaps it could just support 
>> automatic execution of a post-install script (and pre-uninstall 
>> script if uninstallation ever happens) that allows individual project 
>> developers to do what they need to do?  Let the burden of describing 
>> how those things happen and how to uninstall/relocate/update them 
>> fall to the provider of the projects that do them.
>
> Yeah, that's what I really *don't* want.  I'd like to enable a more 
> trustable mechanism than a blindly-executed script.  I'd rather see a 
> standard that makes a developer document more, and have to at least 
> *convince* the user that their post-install is worthwhile, even if the 
> tool then makes it easy to run.

I'm not sure what you mean by "convince".   If you simply mean that the 
post-install has to default to not doing anything unless the user 
responds in the affirmative to some prompt, then I guess I could live 
with that.   If you mean that other documentation has to convince them 
to run a command, then I think that leads to the issue I was directly 
worried about, which is people complaining because something isn't 
working because they forgot to run the post-install.

My other concern here is how this chains through dependencies.  If I 
install the ETS meta-egg mentioned above, and that causes 4 other eggs 
to install that all have post-install requirements, I'd hate to have the 
user have to step through the same sort of prompts 4 times.   I guess 
this is what you were referring to above by a list of post-install 
tasks, but I just want to be sure.


> Better still, I'd rather have those post-install parts done in such a 
> way that things like icons, menus, manifests, registry stuff, etc., 
> have to get explicitly listed instead of being done programatically.

I assume you mean by declaring lists or dictionaries within the setup.py 
that then get stored as meta-data within a file in EGG-INFO and then get 
acted on during install.  If so, then yes, I'm all for that idea too.


>> Also, IIUC, stow only tries to "contain" the hard files.  It puts 
>> links in multiple standard locations (for man pages, executables, 
>> libraries, etc.)   If setuptools supported these options, I don't 
>> think there'd be any discussion here except for things like "how do I 
>> extend the set of things the tool supports so that my foo-type files 
>> get linked into the standard /os/path/to/foo for the X os?"
>
> Yep.  Having that would be a worthwhile thing, I think.  Discussion 
> leading to specs is most welcome.

I thought I was starting that already. :-)  Or were you saying that it 
needed to happen somewhere else?


>> I should have read ahead.  This sounds close to what I've been 
>> describing except that this leads me to picture a script that prompts 
>> for install locations and allows the user to customize the 
>> destinations rather than one that assumes everything goes in a 
>> standard place.  I'm all for this, and the continuation of the 
>> ability to install an egg into a user-environment vs. a 
>> system-environment.
>
> +1.
>
>
>> The only thing missing here is the ability for the installer to 
>> automatically run that script so that installation isn't a 
>> disjointed, two-step manual process that a user is prone to forgot to 
>> complete.
>
> I don't see a problem with a prompting process, backed by a log file 
> that records what post-install steps are pending, finished, or 
> explicitly rejected by the user.
>
> One possibility, by the way, is that we could overload "extras" for 
> this purpose.  Entry points (such as those for scripts) can require 
> extras; if extras could mean post-install components like docs or 
> icons or what-have-you, then trying to run the script could result in 
> an error message telling you you need to "easy_install 
> foo_package[icons]" or whatever.

While I can see many nice things about using extras for delivery of 
docs, icons, etc. (including reduced size for those who don't want or 
need them,)  I'm not thrilled with the idea of a user getting a message 
saying to run "easy_install ..." anything for them to be installed.   
Couldn't we just have the post-install actually run the easy_install 
command once they accepted the installation of the icons, etc?


>> One of the features of Enthought's Enstaller extension to 
>> easy_install was that it looks for a post_install.py script in 
>> EGG-INFO and if one is found, runs it.  I would think that getting 
>> this into setuptools would be a significant step forward but I 
>> believe you previously rejected that idea.   We'll take a stab at 
>> creating a patch for you if you're more receptive to that idea now.  
>> Just let me know.
>
> No -- I'm not happy with a straight-up executable hook for 
> post-install steps.  My evaluation of the state of PyPI is that I 
> don't trust the community to write non-hazardous setup.py files, let 
> alone post-install scripts.  There should be a high technical and 
> social barrier to including post-install hooks with arbitrary code.

Ouch.  That seems a pretty harsh indictment.   :-)

> For example, if there was a required separation between installer 
> tools and the things they install, such that any post-install 
> operation had to be performed strictly by providing some 
> human-readable data that will be passed to a separately-installed 
> tool, and there was a high social stigma associated with writing your 
> own post-install tool, then that might work.
>
> So, for example, if the community creates an icons and menus installer 
> tool for the various platforms, and then anybody can use it in their 
> project by adding the right data, then the user doesn't have to fully 
> trust arbitrary package authors, only the authors of the post-install 
> tools.
>
> I'm not saying that model is perfect; in fact I can see some potential 
> pitfalls.  But once an automatic post-install hole is opened it will 
> be *very* hard to close, because it will always be *easier* to roll 
> your own crappy post-installer instead of contributing to a set of 
> robust cross-project/cross-platform tools.  So I'd rather keep this 
> particular "itch" in play and try to build up the scratching pressure 
> until some people get together and pay attention long enough to solve 
> the problem in a less hacky way.  :)

I can see what you're saying though I think it cuts off those who need 
to prove the usecase before writing a tool to support it.   Perhaps we'd 
get more scratching pressure for standardizing (safely) some of these 
things if people were free to experiment. :-)  

Anyway, since Enthought is already scratching, I'm fine with the idea of 
building a standard way to do it that is driven by human-readable 
data.   We just need to setup the process to allow that to happen.   So 
far I haven't seen any responses from you in regards to the setup of an 
issue/patch tracker, wiki, process to open up the number of commiters, 
etc. that gives me any confidence I'm not heading off down the wrong 
path somehow.  Perhaps I'm too cautious?


>>> On the other hand, I've been puzzling over how to handle legitimate
>>> post-install features.  On Windows, both wx and pywin32 have a real
>>> need to do some actuall "install" operations.  Some is just copying
>>> files, but pywin32 also has to do some registry stuff.  I don't know
>>> how to allow just what's sensible, without opening up a huge can of
>>> worms, though.
>>>
>>
>> I think there are lots of situations that are legitimate (projects 
>> with extensions, projects that want to put icons on the desktop or in 
>> menus, projects that need to interact with a registry, projects that 
>> want to put configuration information somewhere other than in a zip 
>> file in a site-packages dir, etc.)   I think we should worry less 
>> about preventing developers from shooting themselves in the foot
>
> It's the users' feet that I'm concerned with.  Some people are already 
> paranoid about the fact that PyPI doesn't use SSL and code signing, or 
> that easy_install uses the intarwebs at all.  I can just see the witch 
> hunt when we start executing arbitrary code.  Unh unh.  No way am I 
> letting that happen.  Nope.

Though if we had https, code signing, et al, then they'd be trusting the 
signers of the source anyway and not just "arbitrary code".   That 
doesn't seem bad to me.  "If I trust their code to run on my system, why 
not trust the post-install code as well?"


>>  and more about ensuring that they can hunt for food for their survival.
>
> Right now, if you have a post-install script that's essential, you'll 
> just have to convince your users to run it.  Which nicely keeps 
> easy_install out of what should be a conversation between developer 
> and user.

And how do I do that?  So few users read the documentation to begin 
with, or our wiki, or anything else.  Is there some meta-data we're able 
to be put into our eggs / setup.py that displays when the user installs 
them.  And which doesn't scroll by or get buried in an avalanche of 
cascading dependencies?


> Enstaller is a different case - you are presumably installing an 
> application, and the user is trusting your installer.  easy_install is 
> something else altogether, and is used by other programs such as 
> buildout.

I think there may be some misunderstanding here.  Enstaller is how we 
are distributing third-party libraries as binaries for a community of 
users, as well as our own code libraries, and only finally for 
applications.  Yes, you could view it as doing this primarily for larger 
applications but we have a number of people who use it just to get wx, 
VTK, etc. on platforms like Windows and OSX, as well as those who use it 
to get user-space installs of wx, VTK, etc. on Linux.

I'm failing to see how trusting Enstaller is different than trusting 
easy_install.   I wouldn't hold either responsible for what happened if 
I installed a package built by someone else that mis-used the 
features.   Just like I'm careful to not always blame MS if some other 
application, installed via an .msi, messes up my copy of Windows. :-)


> Actually, I wonder if instead of trying to enhance setuptools for 
> post-install, if maybe we should be looking at buildout recipes and 
> maybe having a way for setuptools dependencies to point to buildout 
> specs.  IIRC, buildout specs can be remotely retrieved from a single 
> URL, too.

I'll need to read up more on buildout to understand this, but my 
understanding was that buildout was not something a user ran to install 
an app, but rather something the developer ran to build and publish an 
app.  The end result of a 'production' buildout is to generate a large 
tarball or rpm that included everything, right?   If so, this goes 
directly against what Enthought was aiming for, which was to allow 
delivery of bug-fixes and minor updates in a large app by downloading 
only smaller units instead of a huge monolithic re-install of everything.  

Having typed that up though, I'm thinking we're probably abusing eggs 
for something that rightly ought to be delivered as an application 
directory scoped patch.


>>    We can always tighten things down after seeing the usecases that 
>> develop, right?
>
> Actually, no, we can't, since backward compatibility would keep us 
> from removing the hook, once people rely on it.
>
> I really feel yours (and others) pain on this issue, but it's one 
> place where the users have to come first, and they need protection 
> from the wilds of PyPI.  Distribution and installation issues are not 
> first on most developers' minds, so the fact that someone writes a 
> great library on PyPI doesn't mean they can write installers worth a 
> crap.  Frankly, I wouldn't trust myself to write a correct 
> post-installer on the first go -- perhaps *because* I have seen so 
> many "simple" things go wrong.

Hell, people can't even write correct code on the first go otherwise we 
wouldn't have bugs in every app, os, and driver.  However, people do fix 
things over time and eventually get it right or else their project dies 
because no one wants to deal with the pain.   Why is python installation 
any different? :-)


-- Dave


From jeff at taupro.com  Thu Mar 20 21:42:10 2008
From: jeff at taupro.com (Jeff Rush)
Date: Thu, 20 Mar 2008 15:42:10 -0500
Subject: [Distutils] Request for Input re Packaging
In-Reply-To: <94bdd2610803200151o5efa8589y6341a2ac9013f4c2@mail.gmail.com>
References: <47E19F22.5000600@taupro.com>
	<94bdd2610803200151o5efa8589y6341a2ac9013f4c2@mail.gmail.com>
Message-ID: <47E2CC22.1080207@taupro.com>

Tarek Ziad? wrote:
> 
> On Thu, Mar 20, 2008 at 12:17 AM, Jeff Rush <jeff at taupro.com 
>
>       - move to https/ssl
> 
> There are a few problems in this area, also related to indexing 
> we need to work out imho:
> 
> When a package defines a https://... link into the url meta-data, the 
> link will
> be added in the Simple index besides other links. For instance, people
> that uses sourceforge can have such urls. Even if the package egg or tarball
> is available at PyPI, the home page url will appear at #1 on the index page.
> 
> This will make tools like easy_install read this link before it reaches 
> the egg/tarball.
> 
> This is OK as long as the users behind the firewalls are allowed to call 
> htppS...

It's not clear to me the correct behavior - help me understand:

1. Are there firewall policies that block *all* https access?   I've
    only encountered more fine-grained firewalls because, to me, use
    of https for _some_ sites is a necessary and expected behavior.

2. If we moved PyPI to serve exclusively over https, for integrity
    reasons, would this have a major negative impact?

3. Would it be better to sort the URLs, to place the https ones at
    the end, a and allow a fetch error to occur, or provide a
    .distutils config option to just quietly skip https sites?

4. Is it not a problem that, when checking for newer versions,
    setuptools would be unable to access a newer version on an
    https site and would have to settle for an older version
    on a non-https site, leading to stale packages?

-Jeff

From p.f.moore at gmail.com  Thu Mar 20 22:04:36 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 20 Mar 2008 21:04:36 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <ca471dc20803191206j2eae1be3rc5745f955d8e0dce@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080317151637.532D23A409D@sparrow.telecommunity.com>
	<ca471dc20803170853s490339f2g20bb1d95122ff77f@mail.gmail.com>
	<20080317161305.22B183A409D@sparrow.telecommunity.com>
	<ca471dc20803170931o6cb7581fj674451c8dd653ee8@mail.gmail.com>
	<47DEC0EB.3000404@palladion.com>
	<440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com>
	<ca471dc20803191025i7ea5af15wc0bdcc4a47632720@mail.gmail.com>
	<79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com>
	<ca471dc20803191206j2eae1be3rc5745f955d8e0dce@mail.gmail.com>
Message-ID: <79990c6b0803201404ub931349k1211d33dffbba0e2@mail.gmail.com>

On 19/03/2008, Guido van Rossum <guido at python.org> wrote:
> I'd only do what __loader__ offers. People can always wrap a StringIO around it.
>
>  >  Once I have a patch, I'll post it to the tracker. What's the best
>  >  approach? Code a patch for 3.0 and backport, or code for 2.6 and let
>  >  the merging process do its stuff?
>
> Code for 2.6, let the merge do its thing.

http://bugs.python.org/issue2439

Let me know if you want any changes. If it's OK, I'll think about
whether grander things would be of use.

Paul.

From jeff at taupro.com  Thu Mar 20 22:36:40 2008
From: jeff at taupro.com (Jeff Rush)
Date: Thu, 20 Mar 2008 16:36:40 -0500
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the
	pkg_resources	module)
In-Reply-To: <79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
Message-ID: <47E2D8E8.8090209@taupro.com>

Paul Moore wrote:
> On 20/03/2008, zooko <zooko at zooko.com> wrote:

> I'll chime in here, too. I really want to like
> setuptools/easy_install, but I don't. I'll try to be specific in my
> reasons, in the hope that they can be addressed. I know some of these
> are "known about", but one of my meta-dislikes of setuptools is that
> known issues never seem to get addressed (I know, patches accepted,
> but I haven't got the time either...)

Clearly explained problems with the existing arrangement is valuable as well 
as patches.  Thanks for taking the time to help us see your viewpoint.


> 1. No integration with the system packager (Windows, in my case). If I
> do easy_install nose, then nose does not show up in add/remove
> programs. That significantly affects the way I manage my PC.

Part of this stems from stretching of the original mission of setuptools, to 
install modules for Python, into a general-purpose application installation 
tool.  The buildout tool is more suited for application installation, although 
not ideal yet.

In your scenario, what happens when one egg pulls in another and another, 
until you have a hundred entries in your add/remove menu?  And you don't 
understand the inter-relationship of those so you cannot do a clean uninstall?

Similarly, or what do you want to appear in that add/remove menu when you are 
using independent sandboxes with various applications in them, some of which 
are accessible only to specific users who are not admins on that box?


> 3. The pkg_resources documentation (in particular, that's the one I've
> tried to follow) is extremely hard to read. Partly this is just style,
> but it's partly because it is couched in very unfamiliar terms
> (distributions, working sets, interfaces, providers, etc). It's also
> *huge*. A tutorial style overview, supported by API detail, would be
> far better.

We'll get better docs.  Of course, that module contains roughly five different 
sets of functionality, some of which can be used unrelated to the others so 
it's not just one API.


> 4. Hard to use with limited connectivity. At work, I *only* have
> access to the internet via Internet Explorer (MS based proxy). There
> are workarounds, but ultimately "download an installer, then run it"
> is a far simpler approach for me.

This is hard to address using independent eggs re setuptools but fits buildout 
which provides for deployment of a set of related eggs as a single entity. 
I'll add it as a use case and see what we can do though.


> 5. Auto-discovery doesn't always work. I'm sorry, I really can't
> recall the example at the moment, but sometimes easy_install says it
> can't find a package I *know* is available.

I've hit a few of these myself, although it wasn't an issue with the 
auto-discovery mechanism but rather quality problems with PyPI itself.  Some 
of the eggs only had binary distributions provided, and they were not for my 
platform so couldn't be used.  Better error messages in this area would help, 
with encouragement to nag the original author to provide better data on PyPI.


> 6. Splitting the community. Windows users rely heavily on binary
> installers (at least, I do). We're starting to get a situation where
> some projects provide .egg files, and some provide traditional
> (bdist_wininst/bdist_msi) installers. This is bad. One way to do it,
> and all that :-)

Reporting and author nagging facilities built into PyPI could help encourage 
more consistent behavior.

-Jeff


From p.f.moore at gmail.com  Thu Mar 20 22:56:15 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 20 Mar 2008 21:56:15 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E2D8E8.8090209@taupro.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E2D8E8.8090209@taupro.com>
Message-ID: <79990c6b0803201456g30f766b1jfd2af474306b7afc@mail.gmail.com>

On 20/03/2008, Jeff Rush <jeff at taupro.com> wrote:
> Paul Moore wrote:
>  > On 20/03/2008, zooko <zooko at zooko.com> wrote:
>  > 1. No integration with the system packager (Windows, in my case). If I
>  > do easy_install nose, then nose does not show up in add/remove
>  > programs. That significantly affects the way I manage my PC.
>
>
> Part of this stems from stretching of the original mission of setuptools, to
>  install modules for Python, into a general-purpose application installation
>  tool.  The buildout tool is more suited for application installation, although
>  not ideal yet.
>
>  In your scenario, what happens when one egg pulls in another and another,
>  until you have a hundred entries in your add/remove menu?  And you don't
>  understand the inter-relationship of those so you cannot do a clean uninstall?

I don't let it. As I've said elsewhere, I prefer to manage
dependencies myself, manually.

Anything with that many dependencies shouldn't be using the system
Python, in my view. It should be packaged as a standalone application
(py2exe style) and as such have a single add/remove entry (and no
effect on the system Python).

>  Similarly, or what do you want to appear in that add/remove menu when you are
>  using independent sandboxes with various applications in them, some of which
>  are accessible only to specific users who are not admins on that box?

Independent sandboxes isn't a concept I can relate to under Windows.
That doesn't mean it's not possible (I don't know if it is) I just
don't have any useful comment to make, beyond saying that I personally
don't care what happens in that situation.

Paul.

From dpeterson at enthought.com  Thu Mar 20 23:06:19 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Thu, 20 Mar 2008 17:06:19 -0500
Subject: [Distutils] Request for Input re Packaging
In-Reply-To: <47E2CC22.1080207@taupro.com>
References: <47E19F22.5000600@taupro.com>	<94bdd2610803200151o5efa8589y6341a2ac9013f4c2@mail.gmail.com>
	<47E2CC22.1080207@taupro.com>
Message-ID: <47E2DFDB.7060501@enthought.com>

Jeff Rush wrote:
> 2. If we moved PyPI to serve exclusively over https, for integrity
>     reasons, would this have a major negative impact?
>   

Given that urllib2 doesn't support https through a proxy, it would 
probably cause a problem for easy_install, etc.  :-)

We had to create a custom urllib2 Handler for a couple of our 
applications.   I just looked to see if I should pursue re-working our 
solution as a patch and saw that Christopher Li has already posted a 
patch.  http://bugs.python.org/issue1424152   And there is also this 
outstanding ticket: http://bugs.python.org/issue1448934.

-- Dave

From jeff at taupro.com  Thu Mar 20 23:31:35 2008
From: jeff at taupro.com (Jeff Rush)
Date: Thu, 20 Mar 2008 17:31:35 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47E2C848.7090400@enthought.com>
References: <47DEEC6B.5040602@taupro.com>	<20080318004718.56E173A40FF@sparrow.telecommunity.com>	<47E05DD5.1090707@enthought.com>	<20080319012046.7B1233A4074@sparrow.telecommunity.com>
	<47E2C848.7090400@enthought.com>
Message-ID: <47E2E5C7.2050503@taupro.com>

Dave Peterson wrote:
> 
> I agree.   Let's get that setuptools wiki started and start documenting 
> some of these ideas as a roadmap so that anyone who wants to help out 
> has an idea of what to work on, or factor into what they're currently 
> working on.

> Anyway, since Enthought is already scratching, I'm fine with the idea of 
> building a standard way to do it that is driven by human-readable 
> data.   We just need to setup the process to allow that to happen.   So 
> far I haven't seen any responses from you in regards to the setup of an 
> issue/patch tracker, wiki, process to open up the number of commiters, 
> etc. that gives me any confidence I'm not heading off down the wrong 
> path somehow.  Perhaps I'm too cautious?

Dave, I'm in the process of getting a tracker for setuptools, and I'll work on 
the wiki shortly, although we have the PackagingBOF wiki for idea collection 
at the moment. Give me a couple of days, including travel from PyCon.  I'm 
fired up to make this happen.


>> Actually, I wonder if instead of trying to enhance setuptools for 
>> post-install, if maybe we should be looking at buildout recipes and 
>> maybe having a way for setuptools dependencies to point to buildout 
>> specs.  IIRC, buildout specs can be remotely retrieved from a single 
>> URL, too.
> 
> I'll need to read up more on buildout to understand this, but my 
> understanding was that buildout was not something a user ran to install 
> an app, but rather something the developer ran to build and publish an 
> app.  The end result of a 'production' buildout is to generate a large 
> tarball or rpm that included everything, right?   If so, this goes 
> directly against what Enthought was aiming for, which was to allow 
> delivery of bug-fixes and minor updates in a large app by downloading 
> only smaller units instead of a huge monolithic re-install of everything.

Your view of a fine-grained application bundle with the ability to dynamically 
download updated eggs without re-pulling the entire thing is an interesting 
contrast to Paul's view of a more monolithic application for easier 
add/remove/uninstall completeness.  Supporting both usage models is going to 
be a challenge but I think is feasible with some thought.

-Jeff


From skip at pobox.com  Thu Mar 20 22:15:00 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 20 Mar 2008 16:15:00 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <871w65gt4i.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <47E0D883.9030402@taupro.com> <47E11659.9000307@v.loewis.de>
	<47E19ABE.2040800@taupro.com> <47E1BD14.2030805@v.loewis.de>
	<47E1EAE4.90802@taupro.com> <47E24C7C.6080209@v.loewis.de>
	<871w65gt4i.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <18402.54228.621089.563344@montanaro-dyndns-org.local>


    >> I don't think any of the core committers is qualified to write such a
    >> document. Instead, it would have to be written by people who
    >> *actually* ported a project from 2 to 3 in some form.

    Stephen> Note that this is precisely the kind of experience Skip
    Stephen> Montanaro is talking about trying to generate in the context of
    Stephen> SpamBayes ...

Correctamundo.  Give that man a cigar.

Skip

From p.f.moore at gmail.com  Fri Mar 21 00:12:25 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 20 Mar 2008 23:12:25 +0000
Subject: [Distutils] [Python-Dev] Wow, I think I actually *get* it now!
In-Reply-To: <20080320224227.601543A4074@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<20080320103823.21558.491968946.divmod.xquotient.8438@joule.divmod.com>
	<20080320150640.379643A40A5@sparrow.telecommunity.com>
	<20080320220844.21558.1047009855.divmod.xquotient.8501@joule.divmod.com>
	<20080320224227.601543A4074@sparrow.telecommunity.com>
Message-ID: <79990c6b0803201612h5a6d57bt82faa4e71c451f07@mail.gmail.com>

On 20/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
>  Actually, this information is VERY helpful.  It makes it blindingly
>  obvious to me now that the difference between loving and hating
>  setuptools is whether you're *intentionally* using it, or whether it
>  shows up in your ecosystem uninvited.

Exactly. I never thought to express that, but it precisely explains my
situation as well.

>  Hm.  So it seems to me that maybe one thing that would help is a
>  "Setuptools Haters' Guide To Setuptools" -- that is, *short*
>  documentation specifically written for people who don't want to use
>  setuptools and want to minimize its impact on their systems.  I could
>  probably write something like that fairly easily, now that I have
>  some idea of what to go in it, more than, "the existing documentation
>  sucks".  :)
>
>  Can I count on some non-assimilated persons' help in critiquing such
>  a document and suggesting any topics I miss?

I would do so. (From a Windows user's perspective).

Paul.

From dpeterson at enthought.com  Fri Mar 21 00:36:46 2008
From: dpeterson at enthought.com (Dave Peterson)
Date: Thu, 20 Mar 2008 18:36:46 -0500
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47E2E5C7.2050503@taupro.com>
References: <47DEEC6B.5040602@taupro.com>	<20080318004718.56E173A40FF@sparrow.telecommunity.com>	<47E05DD5.1090707@enthought.com>	<20080319012046.7B1233A4074@sparrow.telecommunity.com>
	<47E2C848.7090400@enthought.com> <47E2E5C7.2050503@taupro.com>
Message-ID: <47E2F50E.7040309@enthought.com>

Jeff Rush wrote:
> Dave Peterson wrote:
>>
>> I agree.   Let's get that setuptools wiki started and start 
>> documenting some of these ideas as a roadmap so that anyone who wants 
>> to help out has an idea of what to work on, or factor into what 
>> they're currently working on.
>
>> Anyway, since Enthought is already scratching, I'm fine with the idea 
>> of building a standard way to do it that is driven by human-readable 
>> data.   We just need to setup the process to allow that to happen.   
>> So far I haven't seen any responses from you in regards to the setup 
>> of an issue/patch tracker, wiki, process to open up the number of 
>> commiters, etc. that gives me any confidence I'm not heading off down 
>> the wrong path somehow.  Perhaps I'm too cautious?
>
> Dave, I'm in the process of getting a tracker for setuptools, and I'll 
> work on the wiki shortly, although we have the PackagingBOF wiki for 
> idea collection at the moment. Give me a couple of days, including 
> travel from PyCon.  I'm fired up to make this happen.

Awesome.  Travis O came back into the office from being at PyCon today 
and I've finally got the full set of e-mail addys from the people who 
attended the PackagingBOF meetings.   I'd like to e-mail everyone with 
info about the tracker and wiki, and possibly a new dev type mailing 
list seeded with these names.  Anything I can send out now? :-)


>>> Actually, I wonder if instead of trying to enhance setuptools for 
>>> post-install, if maybe we should be looking at buildout recipes and 
>>> maybe having a way for setuptools dependencies to point to buildout 
>>> specs.  IIRC, buildout specs can be remotely retrieved from a single 
>>> URL, too.
>>
>> I'll need to read up more on buildout to understand this, but my 
>> understanding was that buildout was not something a user ran to 
>> install an app, but rather something the developer ran to build and 
>> publish an app.  The end result of a 'production' buildout is to 
>> generate a large tarball or rpm that included everything, right?   If 
>> so, this goes directly against what Enthought was aiming for, which 
>> was to allow delivery of bug-fixes and minor updates in a large app 
>> by downloading only smaller units instead of a huge monolithic 
>> re-install of everything.
>
> Your view of a fine-grained application bundle with the ability to 
> dynamically download updated eggs without re-pulling the entire thing 
> is an interesting contrast to Paul's view of a more monolithic 
> application for easier add/remove/uninstall completeness.  Supporting 
> both usage models is going to be a challenge but I think is feasible 
> with some thought.

Yes, that will be interesting indeed.   I'm not sure we need anything 
more to support what we'd like to do than to resolve some dependency 
lookup issues that I've already talked about on this list and at the 
PackagingBOF.

-- Dave

>
> -Jeff
>

From greg.ewing at canterbury.ac.nz  Fri Mar 21 00:51:00 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 21 Mar 2008 11:51:00 +1200
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <20080320150754.9F6503A40A5@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>
	<20080317184553.913413A409D@sparrow.telecommunity.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<47E1EEE6.5050608@palladion.com>
	<20080320150754.9F6503A40A5@sparrow.telecommunity.com>
Message-ID: <47E2F864.8020900@canterbury.ac.nz>

At 12:58 AM 3/20/2008 -0400, Tres Seaver wrote:

> A lot of setuptools warts are driven by related design problems in the
> distutils, such as the choice to use imperative / procedural code for
> everything

If a distutils replacement is ever written, I'd like to
see it structured as a dependency graph, like a makefile,
with each node in the graph knowing how to transform its
inputs into its outputs.

That would make it a lot easier to extend to accommodate
new things like Pyrex. You'd just have to write a new
node class that knows how to turn .pyx files into .c
files, and the existing machinery would take it from
there.

-- 
Greg

From zooko at zooko.com  Fri Mar 21 01:48:24 2008
From: zooko at zooko.com (zooko)
Date: Thu, 20 Mar 2008 18:48:24 -0600
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <47E26A20.1090402@palladion.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com>
Message-ID: <EA0E6502-25D8-470A-A155-895163E46E6F@zooko.com>

On Mar 20, 2008, at 7:44 AM, Tres Seaver wrote:

> Paul Moore wrote:

>> 4. Hard to use with limited connectivity. At work, I *only* have
>> access to the internet via Internet Explorer (MS based proxy). There
>> are workarounds, but ultimately "download an installer, then run it"
>> is a far simpler approach for me.
>
> I don't know how to make this requirement compatible with using shared
> dependencies,

We've done something like this.

The http://allmydata.org project bundles its easy_installable  
dependencies.  If you get the current trunk from our darcs repository  
[1], or get a release tarball or a snapshot tarball from [2], then it  
comes with a directory named "misc/dependencies" which has the source  
tarballs of our easy_installable dependencies.  You can browse this  
directory on the web: [3].

Therefore, if you manually satisfy the non-easy_installable  
dependencies, you can download an allmydata.org tarball, disconnect  
from the Internet (which we call "moving to a Desert Island"), and  
install it.

This is, as you say, "compatible with using shared dependencies"  
because setuptools will detect if you already have sufficiently new  
versions of some of these dependencies installed (for example, if  
they are installed in Debian packages), and then skip the step of  
installing that dependency from its source tarball.

The remaining dependencies that cannot be satisfied automatically by  
our setup.py are listed in the install.html [4].  They are:

    1.  g++ >= v3.3 -- the Cygwin version of gcc/g++ works for Cygwin  
and for Windows
    2. GNU make
    3. Python >= v2.4.2 including development headers i.e. "Python.h"
    4. Twisted >= v2.4.0 -- from the Twisted "sumo" source tarball
    5. OpenSSL >= v0.9.7, including development headers
    6. PyOpenSSL == v0.6
    7. Crypto++ >= v5.2.1, including development headers

I am hoping that in the future Twisted (see twisted #1286 [5]) and  
pyOpenSSL will be easy_installable, and that our use of setuptools  
plugins will eventually replace our GNUmakefile and thus remove our  
dependency on GNUmake.  That will leave only g++, Python, OpenSSL,  
and Crypto++ as dependencies that a user has to manually deal with in  
order to build allmydata.org from source.

Regards,

Zooko

[1] http://allmydata.org/source/tahoe/trunk/
[2] http://allmydata.org/source/tahoe/tarballs/
[3] http://allmydata.org/trac/tahoe/browser/misc/dependencies
[4] http://allmydata.org/source/tahoe/trunk/docs/install.html
[5] http://twistedmatrix.com/trac/ticket/1286

From pje at telecommunity.com  Thu Mar 20 23:42:20 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 20 Mar 2008 18:42:20 -0400
Subject: [Distutils] Wow, I think I actually *get* it now!
In-Reply-To: <20080320220844.21558.1047009855.divmod.xquotient.8501@joul
	e.divmod.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<20080320103823.21558.491968946.divmod.xquotient.8438@joule.divmod.com>
	<20080320150640.379643A40A5@sparrow.telecommunity.com>
	<20080320220844.21558.1047009855.divmod.xquotient.8501@joule.divmod.com>
Message-ID: <20080320224227.601543A4074@sparrow.telecommunity.com>

At 10:08 PM 3/20/2008 +0000, glyph at divmod.com wrote (off-list):
>No, but in no situation, except one (where I was extremely pressed 
>for time) was I actually attempting to use setuptools to use any of 
>its features.  My experience of it is: "If a project uses distutils 
>or apt, installation probably works.  If it uses setuptools, it 
>probably throws a traceback or a wall of text explaining why my 
>environment is inadequate to perform the installation."  Other 
>people chose to use it and in so doing broke my setup.  Manually 
>copying a few files in these cases was a _lot_ easier than 
>attempting to diagnose and repair software that I didn't even want to use.
>
>I am not interesting in packaging or distribution.  Far from it: I 
>run all of my software out of an SVN checkout and I _detest_ being 
>involved in discussions of deployment or installation.
...
>However, the general message of the negative subjective experience I 
>have had while using setuptools is not FUD.  It's an accurate 
>portrayal of a great deal of frustration.  setuptools has, to this 
>date, not solved a single problem for *me*, personally or 
>professionally, but it has caused many.  distutils, despite its many 
>flaws, has actually solved quite a few.

Actually, this information is VERY helpful.  It makes it blindingly 
obvious to me now that the difference between loving and hating 
setuptools is whether you're *intentionally* using it, or whether it 
shows up in your ecosystem uninvited.  It also makes the difference 
in whether you get involved: with no investment in the tool itself, 
you have minimal motivation to RTFM, ask questions, or fix bugs.  And 
when people in this scenario *do* communicate to me or the 
distutils-sig, they are much more likely to be impatient and hostile, 
and more likely to view the system as "fundamentally broken".

This makes total sense to me now.  I don't have any *solutions* to 
the problem, mind you, but at least now I understand what before 
seemed like some sort of bizarre anomaly where literally thousands of 
people use setuptools and many dozens actually express their 
happiness with or even love for the system, and then others hate it 
like they hate Microsoft, or worse.  ;-)

Meanwhile, from the "outsiders" point of view, setuptools looks like 
the Matrix or the Borg, happily assimilating the masses, who then 
start coming to you and say, "But you'll be so much happier once you 
join us..."  ...and off in the distance, you hear a quiet rumbling of 
zombies chanting "eeeeggs....  eeeeggggs....  mussst havve eggggssss!"  :)

Hm.  So it seems to me that maybe one thing that would help is a 
"Setuptools Haters' Guide To Setuptools" -- that is, *short* 
documentation specifically written for people who don't want to use 
setuptools and want to minimize its impact on their systems.  I could 
probably write something like that fairly easily, now that I have 
some idea of what to go in it, more than, "the existing documentation 
sucks".  :)

Can I count on some non-assimilated persons' help in critiquing such 
a document and suggesting any topics I miss?


From interstar at gmail.com  Fri Mar 21 03:25:05 2008
From: interstar at gmail.com (phil jones)
Date: Thu, 20 Mar 2008 23:25:05 -0300
Subject: [Distutils] [Python-Dev] Wow, I think I actually *get* it now!
In-Reply-To: <79990c6b0803201612h5a6d57bt82faa4e71c451f07@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<20080320103823.21558.491968946.divmod.xquotient.8438@joule.divmod.com>
	<20080320150640.379643A40A5@sparrow.telecommunity.com>
	<20080320220844.21558.1047009855.divmod.xquotient.8501@joule.divmod.com>
	<20080320224227.601543A4074@sparrow.telecommunity.com>
	<79990c6b0803201612h5a6d57bt82faa4e71c451f07@mail.gmail.com>
Message-ID: <dc2a299f0803201925t32d6832kdc73a30d3a0c987e@mail.gmail.com>

Just a thought. What I'd really like to see is a quick 5 / 10 minute
screen-cast of using setup tools. I've been looking for one for a
couple of weeks but haven't found,

Something like this buildout example, perhaps :

http://video.google.com/videoplay?docid=3428163188647461098&total=7&start=0&num=10&so=0&type=search&plindex=0

cheers

phil jones

On Thu, Mar 20, 2008 at 8:12 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 20/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
>  >  Actually, this information is VERY helpful.  It makes it blindingly
>  >  obvious to me now that the difference between loving and hating
>  >  setuptools is whether you're *intentionally* using it, or whether it
>  >  shows up in your ecosystem uninvited.
>
>  Exactly. I never thought to express that, but it precisely explains my
>  situation as well.
>
>
>  >  Hm.  So it seems to me that maybe one thing that would help is a
>  >  "Setuptools Haters' Guide To Setuptools" -- that is, *short*
>  >  documentation specifically written for people who don't want to use
>  >  setuptools and want to minimize its impact on their systems.  I could
>  >  probably write something like that fairly easily, now that I have
>  >  some idea of what to go in it, more than, "the existing documentation
>  >  sucks".  :)
>  >
>  >  Can I count on some non-assimilated persons' help in critiquing such
>  >  a document and suggesting any topics I miss?
>
>  I would do so. (From a Windows user's perspective).
>
>  Paul.
>
>
> _______________________________________________
>  Distutils-SIG maillist  -  Distutils-SIG at python.org
>  http://mail.python.org/mailman/listinfo/distutils-sig
>

From janzert at janzert.com  Fri Mar 21 04:50:23 2008
From: janzert at janzert.com (Janzert)
Date: Thu, 20 Mar 2008 23:50:23 -0400
Subject: [Distutils] PEP 365 (Adding the pkg_resources module)
In-Reply-To: <ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>	<20080317184553.913413A409D@sparrow.telecommunity.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
Message-ID: <frvb9t$d1d$1@ger.gmane.org>

Guido van Rossum wrote:
> On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> [a long message]
> 
> I'm back at Google and *really* busy for another week or so, so I'll
> have to postpone the rest of this discussion for a while. If other
> people want to chime in please do so; if this is just a dialog between
> Phillip and me I might incorrectly assume that nobody besides Phillip
> really cares.
> 

Since there seems to be a fair number of negative responses to 
setuptools, I just wanted to add a bit of positive counterbalance. I'm 
just a random python user that happens to track python-dev a bit, so 
take all this with the realization that I probably shouldn't have much 
input into anything. ;)

I've been using python for somewhere around 10 years to write various 
random small scripts, gui applications and recently web applications. 
For me setuptools is the best thing to happen to python since I've been 
using it. I develop and deploy on a seemingly constantly changing mix of 
various flavors of windows and linux. Unlike for others, I love that 
once I get setuptools installed I can just use the same commands to get 
the things I need. I guess the contrast for me is that python is the 
common base that I tend to work from not the underlying OS.

So I don't know if I'm part of a large number of quiet users or just 
happen to be an odd case that works really well with setuptools. I was 
disappointed when setuptools didn't make it into 2.5 and I really hope 
it or something very much like it can make it into a release in the near 
future. Because while setuptools certainly isn't perfect, for me at 
least, it is much, much better than nothing at all.

Brian Haskin


From fumanchu at aminus.org  Fri Mar 21 05:10:09 2008
From: fumanchu at aminus.org (Robert Brewer)
Date: Thu, 20 Mar 2008 21:10:09 -0700
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <frvb9t$d1d$1@ger.gmane.org>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<ca471dc20803171059l725eb0a8g8081e907ad2d0864@mail.gmail.com>	<20080317184553.913413A409D@sparrow.telecommunity.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com><ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<frvb9t$d1d$1@ger.gmane.org>
Message-ID: <F1962646D3B64642B7C9A06068EE1E6402A05D7B@ex10.hostedexchange.local>

Janzert wrote:
> Since there seems to be a fair number of negative responses to
> setuptools, I just wanted to add a bit of positive counterbalance. I'm
> just a random python user that happens to track python-dev a bit, so
> take all this with the realization that I probably shouldn't have much
> input into anything. ;)
> 
> I've been using python for somewhere around 10 years to write various
> random small scripts, gui applications and recently web applications.
> For me setuptools is the best thing to happen to python since I've
been
> using it. I develop and deploy on a seemingly constantly changing mix
> of
> various flavors of windows and linux. Unlike for others, I love that
> once I get setuptools installed I can just use the same commands to
get
> the things I need. I guess the contrast for me is that python is the
> common base that I tend to work from not the underlying OS.
> 
> So I don't know if I'm part of a large number of quiet users or just
> happen to be an odd case that works really well with setuptools. I was
> disappointed when setuptools didn't make it into 2.5 and I really hope
> it or something very much like it can make it into a release in the
> near future. Because while setuptools certainly isn't perfect, for me
> at least, it is much, much better than nothing at all.

My interpretation of this is that setuptools suffers from the same
malaise all flexible apps do (but especially CLI apps it seems):
frequent users love the power and high volume of options, infrequent
users despise it. If you're installing apps all day, you probably use it
a lot more often than library devs like me who use it once every other
month (if we're forced to).


Robert Brewer
fumanchu at aminus.org


From tjreedy at udel.edu  Fri Mar 21 01:37:51 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 20 Mar 2008 20:37:51 -0400
Subject: [Distutils] PEP 365 (Adding thepkg_resources	module)
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>	<20080318223700.C64123A4074@sparrow.telecommunity.com>	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com><79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E2D8E8.8090209@taupro.com>
Message-ID: <frv00r$inr$1@ger.gmane.org>


"Jeff Rush" <jeff at taupro.com> wrote in message 
news:47E2D8E8.8090209 at taupro.com...

| In your scenario, what happens when one egg pulls in another and another,
| until you have a hundred entries in your add/remove menu?

As I said in another response, I don't think such things belong in 
add/remove.




From ziade.tarek at gmail.com  Fri Mar 21 12:35:15 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 21 Mar 2008 12:35:15 +0100
Subject: [Distutils] Proposal: a 'release' command in distutils
Message-ID: <94bdd2610803210435o20a8b0aag91ec2f07c7e3b5f3@mail.gmail.com>

Hello,

Working with several PyPI servers is making the command lines quite long and
hard to keep up.

For instance, If I want to upload an egg on two servers I have to do this:

$ python setup.py register sdist bdist_egg upload -r http://my.cool.server/
$ python setup.py register sdist bdist_egg upload -r
http://pypi.python.org/pypi

Brrrr.... :)

The setuptools `alias` command make things easier, but we still need to add
in setup.cfg these
alias, and have to call several commands to commit just one egg to several
places.

I would like to propose a new command in distutils to make things easier.
This proposal is based on the `.pypirc` syntax of my previous proposal
(http://wiki.python.org/moin/EnhancedPyPI)

1. Defining releasing strategies for each repository

For each repository, we could add `release` value
to tell distutils the sequence to be run when `release` is called::

  [distutils]
  index-servers =
    pypi
    my-other-server
    my-other-server-with-its-own-realm

  [pypi]
  repository:http://pypi.python.org/pypi/

  username:tarek2
  password:secret
  release:register sdist build_mo bdist_egg upload
  release-strategy : *

  [my-other-server]
  username:tarek2
  password:secret
  repository:http://example.com/repository
  release:register sdist build_mo bdist_egg upload
  release-strategy:
        acme.*
        specific.package

  [my-other-server-with-its-own-realm]
  username:tarek3
  password:secret3
  repository:http://example2.com/repository
  realm:acme
  release:register sdist build_mo bdist_egg upload
  release-strategy:
        acme.*
        acme2.*

The `release-strategy` value would be a set of glob-like expressions
that will be used to match the package being released, in order
to qualify it for the given repository.

- when `release` is not provided, the repository is ommited by the release
command
- when `release` is provided, but not `release-strategy`, the default value
  for `release-strategy` would be *

Of course these are just optional shorcuts

2. Making a release

This call:

  $ cd acme.package
  $ python setup.py release

Would then look in .pypirc where "acme.package" (defined in the package
meta-data name) needs to be released, and will do this equivalent sequence:

  $ python setup.py register sdist build_mo bdist_egg upload -r pypi
  $ python setup.py register sdist build_mo bdist_egg upload -r
my-other-server-with-its-own-realm

The interesting thing about glob expressions is that it allows setting a
release strategy
upon the namespaces. For example, it makes it possible to upload the eggs of
a given
company to a dedicated server, and defines specific eggs to be released at
pypi.


Tarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080321/0fd2224a/attachment.htm 

From p.f.moore at gmail.com  Fri Mar 21 13:33:31 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Mar 2008 12:33:31 +0000
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <fruvlo$hvu$1@ger.gmane.org>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com> <fruvlo$hvu$1@ger.gmane.org>
Message-ID: <79990c6b0803210533q3f9310c8tf119300a77030ccf@mail.gmail.com>

On 21/03/2008, Terry Reedy <tjreedy at udel.edu> wrote:
> However, this Windows user, and I expect most, do NOT expect add-ons
>  (things under the /Pythonx.y tree) to show up in the add/remove list.

That's an interesting counterpoint to my comments. I presume from this
that you dislike (and/or never use) bdist_msi and bdist_wininst
precisely because they do add such items to the add/remove programs
list?

My argument is essentially that bdist_wininst set a de facto standard
for this. Then, bdist_msi followed it. Now setuptools is trying to
change that standard by ignoring it, rather than by discussion of the
pros and cons.

Personally, I like the current approach, but that's less relevant.

>  The standard (and to me, preferable)  way of dealing with such things is to
>  have an 'installation manager' that can reinstall as well as delete and
>  that has a check box for various things to delete.  This is what Python
>  needs.

I'd dispute strongly that this is a "standard". It may be preferable,
but I'm not sure where you see evidence of it being a standard.

Could I also point out that *if* such a standard is set up for Python,
bdist_wininst and bdist_msi should be modified to follow it.
Otherwise, it's not a standard, more of  competing approach.

As you can see, my main concern is for consistency :-)

Paul.

From pje at telecommunity.com  Fri Mar 21 13:38:15 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 08:38:15 -0400
Subject: [Distutils] [Python-Dev] Capsule Summary of Some
 Packaging/Deployment Technology Concerns
In-Reply-To: <47E2E5C7.2050503@taupro.com>
References: <47DEEC6B.5040602@taupro.com>
	<20080318004718.56E173A40FF@sparrow.telecommunity.com>
	<47E05DD5.1090707@enthought.com>
	<20080319012046.7B1233A4074@sparrow.telecommunity.com>
	<47E2C848.7090400@enthought.com> <47E2E5C7.2050503@taupro.com>
Message-ID: <20080321123816.8BC0C3A4074@sparrow.telecommunity.com>

At 05:31 PM 3/20/2008 -0500, Jeff Rush wrote:
>Dave, I'm in the process of getting a tracker for setuptools,

Can't we just use the Python tracker?  I'm already registered in 
that, and emailing back on issues works.  If we can just put me on 
the nosy list for all setuptools issues, I'll see everything that 
goes by and comment without having to pop into a browser.

If we can't use the Python tracker, let's please use something that 
lets me comment via email.  (And of course, I'll want to subscribe to 
the wiki via email too, but I assume we'll be using a wiki that allows that.)


From pje at telecommunity.com  Fri Mar 21 13:49:42 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 08:49:42 -0400
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <79990c6b0803210533q3f9310c8tf119300a77030ccf@mail.gmail.co
 m>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com> <fruvlo$hvu$1@ger.gmane.org>
	<79990c6b0803210533q3f9310c8tf119300a77030ccf@mail.gmail.com>
Message-ID: <20080321124944.ABDF03A4074@sparrow.telecommunity.com>

At 12:33 PM 3/21/2008 +0000, Paul Moore wrote:
>On 21/03/2008, Terry Reedy <tjreedy at udel.edu> wrote:
> >  The standard (and to me, preferable)  way of dealing with such 
> things is to
> >  have an 'installation manager' that can reinstall as well as delete and
> >  that has a check box for various things to delete.  This is what Python
> >  needs.
>
>I'd dispute strongly that this is a "standard". It may be preferable,
>but I'm not sure where you see evidence of it being a standard.

I presume he means that there are a lot of entries in his Add/Remove 
Programs that work like that, and that it's an emerging standard for 
Windows.  (Certainly I've seen quite a few entries like that in mine, 
although more often than not they only have one checkbox!)


>Could I also point out that *if* such a standard is set up for Python,
>bdist_wininst and bdist_msi should be modified to follow it.
>Otherwise, it's not a standard, more of  competing approach.

The best thing to do would be to get a standard (ala PEP 262, but 
modified by the benefit of experience now) for tracking installed 
Python package distributions.  Then we can standardize on platform 
tools for managing this data, and include them in the relevant 
platform distributions.  (And that would include making bdist_wininst 
and bdist_msi follow this installation DB standard.)


From asmodai at in-nomine.org  Fri Mar 21 13:54:02 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Fri, 21 Mar 2008 13:54:02 +0100
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <frua34$b7q$1@ger.gmane.org>
References: <ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com> <47E29176.9050109@v.loewis.de>
	<frua34$b7q$1@ger.gmane.org>
Message-ID: <20080321125402.GB83712@nexus.in-nomine.org>

-On [20080320 19:24], Steve Holden (steve at holdenweb.com) wrote:
>We need to stop protesting that our installation tools are easy enough 
>and try to get behind the various platforms, be it with Windows 
>installers, rpms, or other support. We probably aren't doing this 
>because it's work nobody particularly relishes, and has relatively low 
>visibility in the developer world. Non-developer Python programmers and 
>end-users would thank us, though.

FreeBSD offers through install of Perl through its ports system a Perl
module called 'bsdpan' which registers every module as a package under
FreeBSD's package system.

Normally ports installs modules as p5-ModuleName, but now it becomes:
/var/db/pkg/bsdpan-B-Lint-1.09

And from that point on I can use the pkg* tools.

Quite elegant in my opinion.

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/
To regard the fundamental as the essence, to regard things as coarse, to
regard accumulation as deficiency, and to dwell quietly alone with the
spiritual and the intelligent -- herein lie the techniques of Tao of
the ancients.

From pje at telecommunity.com  Fri Mar 21 14:47:46 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 09:47:46 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
Message-ID: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>

So, after having some time to absorb the Python-Dev threads about 
setuptools, bootstrap, and all the rest, I think I see an opportunity 
to let people route around the "damage" of eggs, while still making 
it possible for the people who want to use easy_install or to put 
dependencies in their setup.py to get what they want, too.  (And 
without them needing to install eggs, either.)  At the same time, we 
can address the issues that remain around uninstalling packages, 
system vs. user packages, PYTHONPATH and site.py woes, and really 
pretty much every other complaint I've heard in the last few days 
about setuptools stomping on other people's stuff.  (Even Paul's 
Windows issues, hopefully.)

Now, you might be asking, "Okay, so why are you telling me about 
this?  Why not just go fix setuptools?"  Well, I *can't*.  Not 
without some help from Python-Dev and the Distutils-SIG, to create an 
updated standard for installed package metadata, by updating PEP 262 
("A Database of Installed Python Packages") to include input from the 
system packaging folks, support for namespace packages, and support 
for setuptools-compatible dependency information.

What's that got to do with anything?  Well, without it, setuptools 
can't support uninstall or conflict management without using eggs to 
compartmentalize the installed files.  And because it has to use eggs 
to do *that*, it has to munge .pth files and install its own site.py 
when installing to PYTHONPATH.  All of this ugliness follows directly 
from the absence of a PEP 262-style installation database.

Sure, setuptools could create its own version of this, and I almost 
did that four years ago.  (If you look at the "open issues" part of 
PEP 262, you'll see my comments from back then.)  I decided not to 
for two reasons: first, the distutils didn't support it yet, so it 
didn't help for conflict detection and avoidance in the real world at 
that point.

Second, there were no uninstall tools for it, so I'd have had to 
write one myself.  (Zed's "easy_f'ing_uninstall" to the contrary, it 
ain't easy, and I have an aversion to deleting stuff on people's 
systems without knowing what will break.  There's a big difference 
between them typing 'rm -rf' themselves, and me doing it.)

However, if tools exist and are distributed for such a "database", 
and *everybody* agrees to use it as an officially-blessed standard, 
then it should be possible for setuptools to co-exist with that 
framework, and we're all happy campers.

In particular, the "installing eggs sucks" camp should be happy, 
because it'll be possible for me (or anyone else) to write a version 
of easy_install that doesn't install eggs any more, and 
setuptools-based packages can go back to having "setup.py install" 
install things the old way by default.

So, to accomplish this, we (for some value of "we") need to:

1. Hash out consensus around what changes or enhancements are needed 
to PEP 262, to resolve the previously-listed open issues, those that 
have come up since (namespace packages, dependency specifications, 
canonical name/version forms), and anything else that comes up.

2. Update or replace the implementation as appropriate, and modify 
the distutils to support it in Python 2.6 and beyond.  And "support 
it" means, "ensure that 'install' and *all* bdist commands update the 
database".  The bdist_rpm, bdist_wininst, and bdist_msi commands, 
even bdist_dumb.  (This should probably also include the add/remove 
programs stuff in the Windows case.)

3. Create a document for system packagers referencing the PEP and 
introducing them to what/why/how of the standard, in case they 
weren't one of the original participants in creating this.

It will probably take some non-trivial work to do all this for Python 
2.6, but it's probably possible, if we start now.  I don't think it's 
critical to have an uninstall tool distributed with 2.6, as long as 
there's a reasonable way to bootstrap its installation later.

Questions, comments...  volunteers?   :)


From optilude at gmx.net  Fri Mar 21 15:38:27 2008
From: optilude at gmx.net (Martin Aspeli)
Date: Fri, 21 Mar 2008 14:38:27 +0000
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <fs0h93$r3d$1@ger.gmane.org>

Phillip J. Eby wrote:

> Questions, comments...  volunteers?   :)

This makes a lot of sense. I don't really have anything to add in terms 
of implementation, but I wonder if we can learn something from how apt 
or rpms or ports work, and how other programming languages (Ruby gems?) 
solve this.

It's not an easy problem space since you're dealing with disparate and 
inconsistent target platforms, but it's one that has been solved before 
by others.

Cheers,
Martin


From him at online.de  Fri Mar 21 16:15:43 2008
From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=)
Date: Fri, 21 Mar 2008 16:15:43 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <47E3D11F.4030503@online.de>

Phillip J. Eby wrote:
> Second, there were no uninstall tools for it, so I'd have had to 
> write one myself.  (Zed's "easy_f'ing_uninstall" to the contrary, it 
> ain't easy, and I have an aversion to deleting stuff on people's 
> systems without knowing what will break.  There's a big difference 
> between them typing 'rm -rf' themselves, and me doing it.)
>   
I think, the uninstall should _not_ 'rm -rf' but only 'rm' the files 
(and 'rmdir' directories, but not
recursively) that it created, and that have not been modified in the 
meantime (after
the installation). This can be easily achieved by recording a checksum 
(eg. md5 or sha)
upon installation and only deleting a file if the checksum is correct 
and only deleting directories when they
are empty (after the installed files in them have been deleted). 
Otherwise, the uninstall should
complain and leave the modified files installed.

Joachim

From ziade.tarek at gmail.com  Fri Mar 21 16:34:32 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 21 Mar 2008 16:34:32 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <94bdd2610803210834g454cc1f3o5e3c4454b93e5030@mail.gmail.com>

On Fri, Mar 21, 2008 at 2:47 PM, Phillip J. Eby <pje at telecommunity.com>
wrote:

> Second, there were no uninstall tools for it, so I'd have had to
> write one myself.  (Zed's "easy_f'ing_uninstall" to the contrary, it
> ain't easy, and I have an aversion to deleting stuff on people's
> systems without knowing what will break.  There's a big difference
> between them typing 'rm -rf' themselves, and me doing it.)
>

Agree. I tried a while ago to write a "easy_uninstall" but this
is not possible from an application-point of view, to do clean things.

zc.buildout resolves this by installing eggs locally, to an isolated
environment, so my main Python installation doesn't hold any extensions
at all.

So if a database of installed package is created, I would be in favor of
an application-oriented system where it is possible to create, update,
install,
a set of packages dedicated to an environment (default would be Python).
Maybe by having a namespace tied to a list of versions.

In other words; having the same feature virtualenv provides, in Python
itself, and define somehow how to switch to it.

$ easy_install the.package  --environment MyApp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080321/8eb07989/attachment.htm 

From waterbug at pangalactic.us  Fri Mar 21 16:30:53 2008
From: waterbug at pangalactic.us (Stephen Waterbury)
Date: Fri, 21 Mar 2008 11:30:53 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <47E3D4AD.8000607@pangalactic.us>

Phillip J. Eby wrote:
> ... if tools exist and are distributed for such a [PEP 262] "database", 
> and *everybody* agrees to use it as an officially-blessed standard, 
> then it should be possible for setuptools to co-exist with that 
> framework, and we're all happy campers.

I like this idea and the 3 items proposed to accomplish it.

> 2. Update or replace the implementation as appropriate  ...

After some googling and digging around, I found:

<http://svn.python.org/projects/sandbox/trunk/pep262>

Is that what you meant by "the implementation"?

> Questions, comments...  volunteers?   :)

I'll try to help, if this is agreed to and if I'm able.

Steve

From skip at pobox.com  Fri Mar 21 17:21:49 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 21 Mar 2008 11:21:49 -0500
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E3D11F.4030503@online.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
Message-ID: <18403.57501.22865.578079@montanaro-dyndns-org.local>


    Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
    Joachim> files (and 'rmdir' directories, but not recursively) that it
    Joachim> created, and that have not been modified in the meantime (after
    Joachim> the installation). 

That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
in both packages A and B.

    Install A - this will create C
    Install B - this might overwrite C, saving a copy, or it might retain
                A's copy.
    Uninstall B - this has to know that C is used by A and not touch it

Skip

From lists at cheimes.de  Fri Mar 21 17:59:45 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 21 Mar 2008 17:59:45 +0100
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <47E3E981.5010503@cheimes.de>

Phillip J. Eby schrieb:
> Questions, comments...  volunteers?   :)

I've yet to read the monster package utils thread so I can't comment on
it. However I like to draw some attention to my PEP 370
http://python.org/dev/peps/pep-0370/. It's about a site packages
directory in the users home directory. I think it quite related to the
discussion.

Christian


From stephen at xemacs.org  Fri Mar 21 19:49:26 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 22 Mar 2008 03:49:26 +0900
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <18403.57501.22865.578079@montanaro-dyndns-org.local>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
Message-ID: <87prtngbex.fsf@uwakimon.sk.tsukuba.ac.jp>

skip at pobox.com writes:
 > 
 >     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
 >     Joachim> files (and 'rmdir' directories, but not recursively) that it
 >     Joachim> created, and that have not been modified in the meantime (after
 >     Joachim> the installation). 
 > 
 > That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
 > in both packages A and B.
 > 
 >     Install A - this will create C
 >     Install B - this might overwrite C, saving a copy, or it might retain
 >                 A's copy.
 >     Uninstall B - this has to know that C is used by A and not touch it

MacPorts has an expensive but interesting approach.  When it finds a
file used by another package, it backs it up to sth like $file.`date`.

From floris.bruynooghe at gmail.com  Fri Mar 21 19:46:48 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 21 Mar 2008 18:46:48 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <18403.57501.22865.578079@montanaro-dyndns-org.local>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
Message-ID: <20080321184648.GA31634@laurie.devork>

On Fri, Mar 21, 2008 at 11:21:49AM -0500, skip at pobox.com wrote:
> 
>     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
>     Joachim> files (and 'rmdir' directories, but not recursively) that it
>     Joachim> created, and that have not been modified in the meantime (after
>     Joachim> the installation). 
> 
> That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
> in both packages A and B.
> 
>     Install A - this will create C
>     Install B - this might overwrite C, saving a copy, or it might retain
>                 A's copy.

An install should never overwrite an existing file.  The sane thing is
to fail installing B by default (dpkg does this for example).  An
option to override this might be useful though.

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From pje at telecommunity.com  Fri Mar 21 20:02:25 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 15:02:25 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <18403.57501.22865.578079@montanaro-dyndns-org.local>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
Message-ID: <20080321190226.4198D3A4074@sparrow.telecommunity.com>

At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
>     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
>     Joachim> files (and 'rmdir' directories, but not recursively) that it
>     Joachim> created, and that have not been modified in the meantime (after
>     Joachim> the installation).
>
>That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
>in both packages A and B.
>
>     Install A - this will create C
>     Install B - this might overwrite C, saving a copy, or it might retain
>                 A's copy.
>     Uninstall B - this has to know that C is used by A and not touch it

Correct.  However, in practice, B should not touch C, unless the file 
is shared between them.

This is a key issue for support of namespace packages, at least if we 
want to avoid using .pth files.  (Which is what setuptools-built 
system packages do for namespace packages currently.)

Of course, one possible solution is for both A and B to depend on a 
"virtual package" that contains C, such that both A and B can install 
it if it's not there, and list it in their dependencies.  But this is 
one of the handful of open issues that needs to be resolved with Real 
Life Package Management people, such as Debian, Fedora, etc.

Neither overwriting, refusing to install, nor backups will properly 
address this issue.  However, this is properly a topic for the 
Distutils-SIG or whatever SIG the actual spec goes to.  On Python-Dev 
I'm only looking for a go/no-go on the overall approach.


From ziade.tarek at gmail.com  Fri Mar 21 20:03:07 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 21 Mar 2008 20:03:07 +0100
Subject: [Distutils] Request for Input re Packaging
In-Reply-To: <47E2CC22.1080207@taupro.com>
References: <47E19F22.5000600@taupro.com>
	<94bdd2610803200151o5efa8589y6341a2ac9013f4c2@mail.gmail.com>
	<47E2CC22.1080207@taupro.com>
Message-ID: <94bdd2610803211203k1cb2ae1di25882fe8d75c7950@mail.gmail.com>

oups, forgot to cc to the list

On Fri, Mar 21, 2008 at 12:28 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

>
>
> On Thu, Mar 20, 2008 at 9:42 PM, Jeff Rush <jeff at taupro.com> wrote:
>
> > Tarek Ziad? wrote:
> > >
> > > On Thu, Mar 20, 2008 at 12:17 AM, Jeff Rush <jeff at taupro.com
> > >
> > >       - move to https/ssl
> > >
> > > There are a few problems in this area, also related to indexing
> > > we need to work out imho:
> > >
> > > When a package defines a https://... link into the url meta-data, the
> > > link will
> > > be added in the Simple index besides other links. For instance, people
> > > that uses sourceforge can have such urls. Even if the package egg or
> > tarball
> > > is available at PyPI, the home page url will appear at #1 on the index
> > page.
> > >
> > > This will make tools like easy_install read this link before it
> > reaches
> > > the egg/tarball.
> > >
> > > This is OK as long as the users behind the firewalls are allowed to
> > call
> > > htppS...
> >
> > It's not clear to me the correct behavior - help me understand:
> >
> > 1. Are there firewall policies that block *all* https access?   I've
> >    only encountered more fine-grained firewalls because, to me, use
> >    of https for _some_ sites is a necessary and expected behavior.
>
>
> That happened last week for a developer on one project at a customer
> place. I am not saying it is the right behavior, but that's how I found
> the problem.
>
> Now maybe that such a firewall is too restrictive anyway to allow the
> usage of a web based repository such as PyPI
>
>
> > 2. If we moved PyPI to serve exclusively over https, for integrity
> >    reasons, would this have a major negative impact?
> >
>
> Related to 1. I guess it is a choice. As long as it is easy to
> created mirrors of PyPI. That's what we do in some projects.
>
> Now for https, like Dave says, we cannot create at this time
> a robust auth handler for it, and our PyPI implementation uses http auth.
>
> So if this patch is pushed it is very cool :)
>
>
> > 3. Would it be better to sort the URLs, to place the https ones at
> >    the end, a and allow a fetch error to occur, or provide a
> >    .distutils config option to just quietly skip https sites?
>
>
> I think ordering the URLs and puting the *.egg, *.tar.gz, etc..
> at first would be good yes, as easy_install fetches them in order.
>
> It will also make the system quicker I think, if easy_install
> would not fetch external home URLs when the right packages
> are available on the page.
>
> Maybe those could be dropped when the dists are uploaded
> That's what I am doing on the PyPI server I work on.
>
>
>
> > 4. Is it not a problem that, when checking for newer versions,
> >    setuptools would be unable to access a newer version on an
> >    https site and would have to settle for an older version
> >    on a non-https site, leading to stale packages?
> >
>
> Good point. But I guess that as long as the system allows
> external urls, we can't prevent from such failures.
>
> We have some mirrors for that as a matter of fact,
> not to rely on third party servers that are sometimes down
> or moving things around.
>
>
>
>
> >
> > -Jeff
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080321/db9c2e23/attachment-0001.htm 

From mal at egenix.com  Fri Mar 21 20:06:14 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 21 Mar 2008 20:06:14 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <47E40726.1090209@egenix.com>

On 2008-03-21 14:47, Phillip J. Eby wrote:
> So, to accomplish this, we (for some value of "we") need to:
> 
> 1. Hash out consensus around what changes or enhancements are needed 
> to PEP 262, to resolve the previously-listed open issues, those that 
> have come up since (namespace packages, dependency specifications, 
> canonical name/version forms), and anything else that comes up.
> 
> 2. Update or replace the implementation as appropriate, and modify 
> the distutils to support it in Python 2.6 and beyond.  And "support 
> it" means, "ensure that 'install' and *all* bdist commands update the 
> database".  The bdist_rpm, bdist_wininst, and bdist_msi commands, 
> even bdist_dumb.  (This should probably also include the add/remove 
> programs stuff in the Windows case.)

The bdist commands don't need to touch that database in any way,
since they don't install anything, nor do they upload things
anywhere. They simply package code and put the result into
the dist/ subdir. That's all.

What you probably mean is that the installers, pre/post-scripts,
etc. run when installing one of those packages should update
the database of installed packages.

Note that there are several package formats which do not execute
any code when installing them - the user simply unzips them in
some directory. These packages won't be able to register themselves
with a database.

I guess the only way to support all of these variants is
to use a filesystem based approach, e.g. by placing a file
with a special extension into some dir on sys.path.
The "database" logic could then scan sys.path for these
files, read the data and provide an interface to it.

All bdist formats would then have to include these files.

distutils already writes .egg-info files when running
python setup.py install, so perhaps that's a start (though
I'd prefer a three letter extension such as ".pkg").

.egg-info files currently only include the package meta-data
(the PKG-INFO section from PEP 262).

We'd have to add a list of files making up the package (FILES
section in PEP 262) and also some extra information about any
extra files the package creates that can safely be removed in
the uninstall process (e.g. .pyo and .pyc files, temporary files,
database files, configuration data, registry entries, etc.) -
this is currently not covered in PEP 262.

I don't think the REQUIRES and PROVIDES sections from the
PEP 262 are needed. That info can easily go into the PKG-INFO
section.

A separate FILES section also doesn't seem to be necessary -
we could just add one or more entries or the format:

CreatesDir abc/
CreatesFile abc/xyz1.py
CreatesDir abc/def/
CreatesFile abc/def/xyz2.py
CreatesFile abc/def/xyz3.py
CreatesFile abc/def/xyz4.ini

(BTW: wininst writes such a file for the uninstall process)

So to keep things simple, the rfc822 approach defined in
PEP 241 would easily cover everything needed and we could
trim down the PEP 262 format to a simple rfc822 header
list.

In other words: the .egg-info files already provide the basis
and only need to be extended with a list of created files,
directories (and possibly other resources) as well as a list
of resources which may be removed even if not installed
explicitly such as byte-code files, etc.

> 3. Create a document for system packagers referencing the PEP and 
> introducing them to what/why/how of the standard, in case they 
> weren't one of the original participants in creating this.

This should probably be a new PEP defining all the bits and
pieces making up the installation "database".

> It will probably take some non-trivial work to do all this for Python 
> 2.6, but it's probably possible, if we start now.  I don't think it's 
> critical to have an uninstall tool distributed with 2.6, as long as 
> there's a reasonable way to bootstrap its installation later.

BTW: There's a simple uninstall command in mxSetup.py that we
could contribute to distutils. It works much in the same
way as the install command... except that it removes all the
files it would have installed.

Using pre-built packages, this works without having to rebuild
the package just to be able to determine the list of things
that need to be removed.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 21 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611

From p.f.moore at gmail.com  Fri Mar 21 20:44:55 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Mar 2008 19:44:55 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <79990c6b0803211244g38626197oc454f18dc208e7eb@mail.gmail.com>

On 21/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:

>  Questions, comments...  volunteers?   :)

Sounds good. I won't volunteer as I have neither time nor expertise to
contribute much. But I'd like to see this happen, as it sounds like it
would address all my issues with setuptools (and just to reiterate,
I'm happy if it is implemented *in place of* add/remove programs - I
have no vested interest in that facility as such).

For uninstall support, you might look at how bdist_wininst installers
handle it. In my experience the uninstall is robust, clean, and
reliable, and the supporting metadata is pretty simple (just a text
file).

Paul.

From jjl at pobox.com  Fri Mar 21 20:51:25 2008
From: jjl at pobox.com (John J Lee)
Date: Fri, 21 Mar 2008 19:51:25 +0000 (GMT)
Subject: [Distutils] Wow, I think I actually *get* it now!
In-Reply-To: <20080320224227.601543A4074@sparrow.telecommunity.com>
References: <ca471dc20803161613h75cdec13t65dc62e87bfe271c@mail.gmail.com>
	<79990c6b0803171319k336110cckdc7563d6ff9b387a@mail.gmail.com>
	<ca471dc20803172231l5680cc03o1aa734268c427f88@mail.gmail.com>
	<87r6e7vjzy.fsf@uwakimon.sk.tsukuba.ac.jp>
	<ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<20080320103823.21558.491968946.divmod.xquotient.8438@joule.divmod.com>
	<20080320150640.379643A40A5@sparrow.telecommunity.com>
	<20080320220844.21558.1047009855.divmod.xquotient.8501@joule.divmod.com>
	<20080320224227.601543A4074@sparrow.telecommunity.com>
Message-ID: <alpine.DEB.1.00.0803211936450.6191@alice>

On Thu, 20 Mar 2008, Phillip J. Eby wrote:
[...]
> Hm.  So it seems to me that maybe one thing that would help is a
> "Setuptools Haters' Guide To Setuptools" -- that is, *short*
> documentation specifically written for people who don't want to use
> setuptools and want to minimize its impact on their systems.  I could
[...]
> Can I count on some non-assimilated persons' help in critiquing such
> a document and suggesting any topics I miss?

Speaking as a zombie, I used to suggest that the yet-to-be-assimilated 
download my source distribution and run:

python setup.py easy_install --no-deps .


(and somehow arrange for dependencies to be installed, too)

If that's reasonable and up-to-date advice, perhaps it could be part of 
this "setuptools :q!" guide...


John


From floris.bruynooghe at gmail.com  Fri Mar 21 21:00:21 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 21 Mar 2008 20:00:21 +0000
Subject: [Distutils] PEP 370 thoughts
In-Reply-To: <47E3E981.5010503@cheimes.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3E981.5010503@cheimes.de>
Message-ID: <20080321200021.GB31634@laurie.devork>

On Fri, Mar 21, 2008 at 05:59:45PM +0100, Christian Heimes wrote:
> Phillip J. Eby schrieb:
> > Questions, comments...  volunteers?   :)
> 
> I've yet to read the monster package utils thread so I can't comment on
> it. However I like to draw some attention to my PEP 370
> http://python.org/dev/peps/pep-0370/. It's about a site packages
> directory in the users home directory. I think it quite related to the
> discussion.

I agree that this is related and valuable too.  It would be good to
see it integrated.

Some thoughts I have from PEP 370:

* ~/.local goes against the de-facto standard of just using ~/lib,
  ~/bin etc in UNIX.  The freedesktop.org standard[1] is a very good
  argument however, but in that case that standard should be followed
  more closely which means respecting their env variables i.e. maybe
  putting data in $XDG_DATA_HOME.  Also the spec does not mention
  ~/.local/lib or ~/.local/bin at all, meaning there is no
  standardised env variable to change it (PEP 370 creates
  PYTHONUSERBASE to work round that though).

* The user site directory would end up before the system site
  directory (except for suid apps).  This will be a problem for Debian
  etc as bug reports are guaranteed when users install a newer version
  of a lib locally and accidentally break an app owned by the system.
  It would be better to leave the user site directory last on sys.path
  and require overwriting this using PTYHONPATH, so users do it
  explicitly.

* setup.py install's new --user argument.  A --home already exists,
  can the default of that not be changed to the user site directory?
  Currently you need to do "--home=~" so making it so that "--home"
  would end up using PYTHONUSERBASE would not break things would it?


Regards
Floris


[1] http://www.freedesktop.org/wiki/Specifications/basedir-spec

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From chris at simplistix.co.uk  Fri Mar 21 20:57:46 2008
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 21 Mar 2008 19:57:46 -0000 (GMT)
Subject: [Distutils] metadata/versions
Message-ID: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>

Hi All,

Apologies if these questions have been answered elsewhere, if they have,
please point me at the answers. I've been trying to follow the discussion
but the shear volume has overwhelmed me...

1. Is there a canonical way to tell what version a python package thinks
it is?

2. Does the python package index contain dependency information? If not,
why not? ;-)

cheers,

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
           - http://www.simplistix.co.uk


From pje at telecommunity.com  Fri Mar 21 22:21:20 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 17:21:20 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E40726.1090209@egenix.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
Message-ID: <20080321212127.791B63A4074@sparrow.telecommunity.com>

At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
>I guess the only way to support all of these variants is
>to use a filesystem based approach, e.g. by placing a file
>with a special extension into some dir on sys.path.
>The "database" logic could then scan sys.path for these
>files, read the data and provide an interface to it.
>
>All bdist formats would then have to include these files.

That's the idea behind the current version of PEP 262, yes, and I 
think it should be kept.


>A separate FILES section also doesn't seem to be necessary -
>we could just add one or more entries or the format:
>
>CreatesDir abc/
>CreatesFile abc/xyz1.py
>CreatesDir abc/def/
>CreatesFile abc/def/xyz2.py
>CreatesFile abc/def/xyz3.py
>CreatesFile abc/def/xyz4.ini

I actually think the size and hash information is good, in order to 
be able to tell if you're looking at an original file.  I'm not sure 
how useful the permissions and uid/gid info is.  I'm hoping we'll 
hear from anybody who has a use case for that.

And of course, there are still some issues to be resolved regarding 
requirements, package name/version stuff, etc.  But we can hash those 
out once we reach a quorum on the Distutils-SIG.


From mal at egenix.com  Fri Mar 21 23:13:32 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 21 Mar 2008 23:13:32 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321212127.791B63A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
Message-ID: <47E4330C.5070601@egenix.com>

On 2008-03-21 22:21, Phillip J. Eby wrote:
> At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
>> I guess the only way to support all of these variants is
>> to use a filesystem based approach, e.g. by placing a file
>> with a special extension into some dir on sys.path.
>> The "database" logic could then scan sys.path for these
>> files, read the data and provide an interface to it.
>>
>> All bdist formats would then have to include these files.
> 
> That's the idea behind the current version of PEP 262, yes, and I think 
> it should be kept.
> 
>> A separate FILES section also doesn't seem to be necessary -
>> we could just add one or more entries or the format:
>>
>> CreatesDir abc/
>> CreatesFile abc/xyz1.py
>> CreatesDir abc/def/
>> CreatesFile abc/def/xyz2.py
>> CreatesFile abc/def/xyz3.py
>> CreatesFile abc/def/xyz4.ini
> 
> I actually think the size and hash information is good, in order to be 
> able to tell if you're looking at an original file.  I'm not sure how 
> useful the permissions and uid/gid info is.  I'm hoping we'll hear from 
> anybody who has a use case for that.

You're heading off in the wrong direction: we should not be trying
to rewrite RPM or InnoSetup in Python.

Anything more complicated should be left to tools which are
specifically written to manage complex software setups.

I honestly believe that most people would be happy if we just
provide these two things (and no more):

  * install a package from a local archive, a URL or PyPI

  * uninstall a package in way that doesn't break other
    installed packages

and whatever the mechanism, avoid making any undercover
changes to the Python installation such as adding
.pth files, overriding site.py, etc. - these are
not needed if the tool keeps to the simple task of
installing and uninstalling Python packages.

Examples:

python pypi.py install mypkg-1.0.tgz
python pypi.py install http://www.example.com/mypkg-1.0.tgz
python pypi.py install mypkg-1.0

python pypi.py uninstall mypkg

If there's a dependency problem, the tool should print the
list of other packages it needs. It should not try to install
things automagically.

If a package needs other modules as well, the package docs
can point the user to use e.g.

python pypi.py install mydep1-1.3 mydep2-2.3 mydep4-0.3 mypkg-1.0

instead.

Anything more complicated should be left to specialized
tools such as RPM, apt, MSI or the other such tools out
there - after all the tool should be about Python *package*
installation, not application installation.

We *don't* need the tool to:

  * support multiple versions of a package (that's just bound
    to cause problems with pickle, isinstance() etc.)

  * provide namespace hacking (is a completely separate issue
    and can be handled by the packages rather than the install
    tool)

  * support all kinds of funky version numbers (if a package
    wants to participate in the system, the author better
    make sure that the version string fits the standard format)

  * provide some form of intra-package bus interface (ie.
    "entry points" as you call them)

  * provide support for keeping whole packages in ZIP files
    (doesn't play well with C extensions, clutters up the
    sys.path, is read-only, needs special importers, etc. etc. )

  * try automatic version matching for required packages

  * download things from SourceForge or other sites with special
    download mechanisms

  * scan websites for links

  * make coffee, clean the house, send the kids to school :-)

> And of course, there are still some issues to be resolved regarding 
> requirements, package name/version stuff, etc.  But we can hash those 
> out once we reach a quorum on the Distutils-SIG.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 21 2008)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
            Registered at Amtsgericht Duesseldorf: HRB 46611

From floris.bruynooghe at gmail.com  Fri Mar 21 23:17:12 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 21 Mar 2008 22:17:12 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <20080321190226.4198D3A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
Message-ID: <20080321221712.GC31634@laurie.devork>

On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
> At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
> >     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
> >     Joachim> files (and 'rmdir' directories, but not recursively) that it
> >     Joachim> created, and that have not been modified in the meantime (after
> >     Joachim> the installation).
> >
> >That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
> >in both packages A and B.
> >
> >     Install A - this will create C
> >     Install B - this might overwrite C, saving a copy, or it might retain
> >                 A's copy.
> >     Uninstall B - this has to know that C is used by A and not touch it
> 
> Correct.  However, in practice, B should not touch C, unless the file 
> is shared between them.
> 
> This is a key issue for support of namespace packages, at least if we 
> want to avoid using .pth files.  (Which is what setuptools-built 
> system packages do for namespace packages currently.)
> 
> Of course, one possible solution is for both A and B to depend on a 
> "virtual package" that contains C, such that both A and B can install 
> it if it's not there, and list it in their dependencies.  But this is 
> one of the handful of open issues that needs to be resolved with Real 
> Life Package Management people, such as Debian, Fedora, etc.

Debian (dpkg) does not allow a file to exist in more then one package
(distribution in distutils speak) as I said earlier.  Directories can
however, they will only be removed if they are empty (i.e. when the
last package having that directory is uninstalled).

I'm not familiar with namespace packages as I've never used one, but I
assume it is just an empty directory with one (empty?)  __init__.py
file in it?  The only way I can see Debian handling that is by having
a Debian package providing exactly that __init__.py and all other
Debian packages needing that namespace package will depend on it.

However, this is shouldn't worry anyone other then Debian I think.  If
a distribution needs to depend on somthing it will depend on whoever
provides the module *inside* the namespace package, not on the
namespace itself.  So the fact that no distribution provides the
namespace.__init__.py file shouldn't have to worry other users of the
python installdb.

Since easy_install and others shouldn't be messing with debian
packages in /usr/lib (they should be using /usr/local/lib or
~/.local [1]) this won't be a problem.  For them to interoperate
in their own module search paths they might need a virtual
distribution though (which Debian could use as well then).


Regards
Floris

PS: Thanks to Ben Finney for bringing the discussions here to the
attention of debian-python at lists.debian.org.


[1] I am aware of problems with this as explained in this thread:
    http://lists.debian.org/debian-python/2008/03/msg00021.html
   
    For the ~/.local, see PEP 370 mentioned here before.
    Basically the location of locally installed python modules might
    need to be revised.  Could PEP 370 be extended to cover search
    paths for modules installed by local administrators?


  For the PEP262-style database this would probably indeed mean
the need for a virtual package (err, distribution) in that database.

> 
> Neither overwriting, refusing to install, nor backups will properly 
> address this issue.  However, this is properly a topic for the 
> Distutils-SIG or whatever SIG the actual spec goes to.  On Python-Dev 
> I'm only looking for a go/no-go on the overall approach.
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From pje at telecommunity.com  Fri Mar 21 23:30:33 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 18:30:33 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
 beyond
In-Reply-To: <20080321221712.GC31634@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<20080321221712.GC31634@laurie.devork>
Message-ID: <20080321223739.A50023A4074@sparrow.telecommunity.com>

At 10:17 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
>On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
> > At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
> > >     Joachim> I think, the uninstall should _not_ 'rm -rf' but 
> only 'rm' the
> > >     Joachim> files (and 'rmdir' directories, but not recursively) that it
> > >     Joachim> created, and that have not been modified in the 
> meantime (after
> > >     Joachim> the installation).
> > >
> > >That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
> > >in both packages A and B.
> > >
> > >     Install A - this will create C
> > >     Install B - this might overwrite C, saving a copy, or it might retain
> > >                 A's copy.
> > >     Uninstall B - this has to know that C is used by A and not touch it
> >
> > Correct.  However, in practice, B should not touch C, unless the file
> > is shared between them.
> >
> > This is a key issue for support of namespace packages, at least if we
> > want to avoid using .pth files.  (Which is what setuptools-built
> > system packages do for namespace packages currently.)
> >
> > Of course, one possible solution is for both A and B to depend on a
> > "virtual package" that contains C, such that both A and B can install
> > it if it's not there, and list it in their dependencies.  But this is
> > one of the handful of open issues that needs to be resolved with Real
> > Life Package Management people, such as Debian, Fedora, etc.
>
>Debian (dpkg) does not allow a file to exist in more then one package
>(distribution in distutils speak) as I said earlier.  Directories can
>however, they will only be removed if they are empty (i.e. when the
>last package having that directory is uninstalled).

This is a general problem with system package managers, 
actually.  Few allow a file to be provided by more than one package.

>I'm not familiar with namespace packages as I've never used one, but I
>assume it is just an empty directory with one (empty?)  __init__.py
>file in it?

It's not empty; it has to contain a namespace declaration.  There's 
another approach that can work around the absence of the __init__.py, 
by using .pth files, but it's messy, and I'd like to get rid of it if possible.


>   The only way I can see Debian handling that is by having
>a Debian package providing exactly that __init__.py and all other
>Debian packages needing that namespace package will depend on it.
>
>However, this is shouldn't worry anyone other then Debian I think.

Alas, it'll probably affect most package managers.


>  If
>a distribution needs to depend on somthing it will depend on whoever
>provides the module *inside* the namespace package, not on the
>namespace itself.  So the fact that no distribution provides the
>namespace.__init__.py file shouldn't have to worry other users of the
>python installdb.

Well, somebody still has to create the __init__.py, and own it for 
purposes of uninstallation.


>Since easy_install and others shouldn't be messing with debian
>packages in /usr/lib (they should be using /usr/local/lib or
>~/.local [1]) this won't be a problem.

If the new PEP (262 redux) contains anything about actual directory 
locations on any system of any platform (other than as examples), 
I'll consider it a failure.  Thinking about specific locations while 
talking about the new PEP should be considered a segmentation fault.  :)

The only thing that matters for the install db is where things are in 
relation to an installation target directory.  Which reminds me -- 
the install db should support install-dir-relative paths, and 
probably should require them for anything that's installed under the 
library install dir.  (Many use cases of this stuff will be 
*relative*; packaging an application as a giant tarball and unpacking 
it somewhere else on a system, or copying an application tree from 
point A to point B, should not render the data therein unusable.)


From p.f.moore at gmail.com  Fri Mar 21 23:40:47 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 21 Mar 2008 22:40:47 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E4330C.5070601@egenix.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
Message-ID: <79990c6b0803211540h25242236kcf3b9a5d6364eef0@mail.gmail.com>

On 21/03/2008, M.-A. Lemburg <mal at egenix.com> wrote:
> You're heading off in the wrong direction: we should not be trying
>  to rewrite RPM or InnoSetup in Python.
>
>  Anything more complicated should be left to tools which are
>  specifically written to manage complex software setups.
>
>  I honestly believe that most people would be happy if we just
>  provide these two things (and no more):
>
>   * install a package from a local archive, a URL or PyPI
>
>   * uninstall a package in way that doesn't break other
>     installed packages
>
>  and whatever the mechanism, avoid making any undercover
>  changes to the Python installation such as adding
>  .pth files, overriding site.py, etc. - these are
>  not needed if the tool keeps to the simple task of
>  installing and uninstalling Python packages.

My immediate impression is that I completely agree with this. I'd like
to add one capability - to be able to list all installed packages.

Anything beyond this, I'd need to be persuaded about. That's not to
say that it's not valuable, just that it should be separate from the
installer. That's where setuptools falls down, in my view - it tries
to do too much all in one package.

>  We *don't* need the tool to:
[...]
>   * make coffee, clean the house, send the kids to school :-)

Well, these would be useful :-)

Paul.

From pje at telecommunity.com  Fri Mar 21 23:41:00 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 18:41:00 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E4330C.5070601@egenix.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
Message-ID: <20080321224110.657C63A40A5@sparrow.telecommunity.com>

At 11:13 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
>On 2008-03-21 22:21, Phillip J. Eby wrote:
> > At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
> >> I guess the only way to support all of these variants is
> >> to use a filesystem based approach, e.g. by placing a file
> >> with a special extension into some dir on sys.path.
> >> The "database" logic could then scan sys.path for these
> >> files, read the data and provide an interface to it.
> >>
> >> All bdist formats would then have to include these files.
> >
> > That's the idea behind the current version of PEP 262, yes, and I think
> > it should be kept.
> >
> >> A separate FILES section also doesn't seem to be necessary -
> >> we could just add one or more entries or the format:
> >>
> >> CreatesDir abc/
> >> CreatesFile abc/xyz1.py
> >> CreatesDir abc/def/
> >> CreatesFile abc/def/xyz2.py
> >> CreatesFile abc/def/xyz3.py
> >> CreatesFile abc/def/xyz4.ini
> >
> > I actually think the size and hash information is good, in order to be
> > able to tell if you're looking at an original file.  I'm not sure how
> > useful the permissions and uid/gid info is.  I'm hoping we'll hear from
> > anybody who has a use case for that.
>
>You're heading off in the wrong direction: we should not be trying
>to rewrite RPM or InnoSetup in Python.

I'm making the assumption that the author(s) of PEP 262 had good 
reason for including what they did, rather than assuming that we 
should start the entire process over from scratch.


>Anything more complicated should be left to tools which are
>specifically written to manage complex software setups.

Tools which will need this data, in order to do their work.  Hence, 
the reason for standardizing the data, instead of the tool(s).


>[snip long list of features, both desired and undesired]

Actually, *all* of these features are out of scope for stdlib 
development, because I'm not proposing including *any* tools for this 
in the stdlib, apart from distutils install and bdist_* support.

I'm proposing, rather, that we finish the vision of PEP 262, of 
having a standard specification that *all* tools will abide by -- 
including rpm, dpkg, and what-have-you.

Since *all* of these tools need to abide by that specification, their 
requirements will need to be considered in the formulation of the spec.

And as I said, I'll be happy if all we do is get the distutils to 
abide by the spec for 2.6, even if it means we don't get an uninstall 
tool.  That can always be installed later using Guido's bootstrap tool.  :)


From floris.bruynooghe at gmail.com  Fri Mar 21 23:41:32 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 21 Mar 2008 22:41:32 +0000
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
Message-ID: <20080321224132.GA17226@laurie.devork>

On Fri, Mar 21, 2008 at 09:47:46AM -0400, Phillip J. Eby wrote:
> Questions, comments...  volunteers?   :)

Sounds good, having a PEP626-style install database seems worthwile.
Definately if it will enable setuptools to install just like distutils
for a "install".

Here some notes from my Debian admin/packager point of view (sorry if
I miss perspectives of Windows users etc):

* I'd like clearly defined module paths for: (a) system, (b) local
  admin, (c) user.  PEP370 seems a step in the right direction.  When
  using "setup.py install" it would be good if the correct prefix
  would be chosen automatically for (b) and (c).  On Windows (a) and
  (b) would be the same AIUI.

* While the installdb can be shared between the sytem and the local
  admin (/var/lib seems appropriate) the user can't normally write to
  this.  So at separate user installdb might be needed.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From floris.bruynooghe at gmail.com  Sat Mar 22 00:23:08 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 21 Mar 2008 23:23:08 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6
	and	beyond
In-Reply-To: <20080321223739.A50023A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<20080321221712.GC31634@laurie.devork>
	<20080321223739.A50023A4074@sparrow.telecommunity.com>
Message-ID: <20080321232308.GA21249@laurie.devork>

On Fri, Mar 21, 2008 at 06:30:33PM -0400, Phillip J. Eby wrote:
> At 10:17 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
>> On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
>> > At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
>> > >     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 
>> 'rm' the
>> > >     Joachim> files (and 'rmdir' directories, but not recursively) that it
>> > >     Joachim> created, and that have not been modified in the  
>> meantime (after
>> > >     Joachim> the installation).
>> > >
>> > >That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
>> > >in both packages A and B.
>> > >
>> > >     Install A - this will create C
>> > >     Install B - this might overwrite C, saving a copy, or it might retain
>> > >                 A's copy.
>> > >     Uninstall B - this has to know that C is used by A and not touch it
>> >
>> > Correct.  However, in practice, B should not touch C, unless the file
>> > is shared between them.
>> >
>> > This is a key issue for support of namespace packages, at least if we
>> > want to avoid using .pth files.  (Which is what setuptools-built
>> > system packages do for namespace packages currently.)
>> >
>> > Of course, one possible solution is for both A and B to depend on a
>> > "virtual package" that contains C, such that both A and B can install
>> > it if it's not there, and list it in their dependencies.  But this is
>> > one of the handful of open issues that needs to be resolved with Real
>> > Life Package Management people, such as Debian, Fedora, etc.
[...]
>> I'm not familiar with namespace packages as I've never used one, but I
>> assume it is just an empty directory with one (empty?)  __init__.py
>> file in it?
>
> It's not empty; it has to contain a namespace declaration.  There's  
> another approach that can work around the absence of the __init__.py, by 
> using .pth files, but it's messy, and I'd like to get rid of it if 
> possible.

So __init__.py contains some data, but it's static regardless of which
sub-packages are installed?  So far so good as that doesn't influence
things on package-management level.

I knew about the .pth hack but as you say it's messy and I'm glad you
prefer to get rid of it too :-).

>>   The only way I can see Debian handling that is by having
>> a Debian package providing exactly that __init__.py and all other
>> Debian packages needing that namespace package will depend on it.
>>
>> However, this is shouldn't worry anyone other then Debian I think.
>
> Alas, it'll probably affect most package managers.
>
>
>>  If
>> a distribution needs to depend on somthing it will depend on whoever
>> provides the module *inside* the namespace package, not on the
>> namespace itself.  So the fact that no distribution provides the
>> namespace.__init__.py file shouldn't have to worry other users of the
>> python installdb.
>
> Well, somebody still has to create the __init__.py, and own it for  
> purposes of uninstallation.

Ok, true.  The virtual distribution thing (or an other solution) might
be needed for this.  But what I was trying to say was that other
package managers should keep their hands off Debian's /usr/lib IMHO.

>> Since easy_install and others shouldn't be messing with debian
>> packages in /usr/lib (they should be using /usr/local/lib or
>> ~/.local [1]) this won't be a problem.
>
> If the new PEP (262 redux) contains anything about actual directory  
> locations on any system of any platform (other than as examples), I'll 
> consider it a failure.  Thinking about specific locations while talking 
> about the new PEP should be considered a segmentation fault.  :)

I think I agree that the PEP should not detail locations (and relative
paths inside the installdb is a good point), but I do think it's part
of this discussion so that the installdb can cope with the different
directory situations.  You can't expect the system and a user to be
able to write to the same installdb.

A problem along this line: The system (e.g. Debian) has a namespace
package installed and a few of it's sub-packages (/usr/lib).  The
local admin wants a new sub-pacakge of this namespace which should go
into e.g. /usr/local/lib (and is not available in the system).  You
now have:

$system/namespace/__init__.py
$system/namespace/sub1/__init__.py
$local/namespace/__init__.py
$local/namespace/sub2/__init__.py

But this doesn't work in Python!  The only solution I can think of is
that the local admin has to install the entire namespace package again
and make sure it ends up before the system one on sys.path.  This is
slightly ugly I think.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From pje at telecommunity.com  Sat Mar 22 00:39:47 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 19:39:47 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <20080321224132.GA17226@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<20080321224132.GA17226@laurie.devork>
Message-ID: <20080321234002.1E8BA3A4074@sparrow.telecommunity.com>

At 10:41 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
>On Fri, Mar 21, 2008 at 09:47:46AM -0400, Phillip J. Eby wrote:
> > Questions, comments...  volunteers?   :)
>
>Sounds good, having a PEP626-style install database seems worthwile.
>Definately if it will enable setuptools to install just like distutils
>for a "install".
>
>Here some notes from my Debian admin/packager point of view (sorry if
>I miss perspectives of Windows users etc):
>
>* I'd like clearly defined module paths for: (a) system, (b) local
>   admin, (c) user.  PEP370 seems a step in the right direction.  When
>   using "setup.py install" it would be good if the correct prefix
>   would be chosen automatically for (b) and (c).  On Windows (a) and
>   (b) would be the same AIUI.

This seems to be out-of-scope.  The "install db" should be on a 
per-target directory basis, and not get into questions of relative 
ordering of target directories, or what directory is the default, or 
any of that.  If we go there, we could be talking until Python 4000 
is released.  :)  All I want is the specification for what is 
recorded within *one* installation directory.

IOW, let's please please please keep the PEP 262 replacement 
orthogonal and separate from any PEP 370 discussions.  Any 
relationship between the two is either an illusion or an 
insufficiently modular design.  :)


>* While the installdb can be shared between the sytem and the local
>   admin (/var/lib seems appropriate) the user can't normally write to
>   this.  So at separate user installdb might be needed.

Each sys.path directory should have its own installation data; see 
the "open issues" section of PEP 262.


From tseaver at palladion.com  Sat Mar 22 02:11:11 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 21 Mar 2008 21:11:11 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321223739.A50023A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E3D11F.4030503@online.de>	<18403.57501.22865.578079@montanaro-dyndns-org.local>	<20080321190226.4198D3A4074@sparrow.telecommunity.com>	<20080321221712.GC31634@laurie.devork>
	<20080321223739.A50023A4074@sparrow.telecommunity.com>
Message-ID: <47E45CAF.9090307@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 10:17 PM 3/21/2008 +0000, Floris Bruynooghe wrote:
>> On Fri, Mar 21, 2008 at 03:02:25PM -0400, Phillip J. Eby wrote:
>>> At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
>>>>     Joachim> I think, the uninstall should _not_ 'rm -rf' but 
>> only 'rm' the
>>>>     Joachim> files (and 'rmdir' directories, but not recursively) that it
>>>>     Joachim> created, and that have not been modified in the 
>> meantime (after
>>>>     Joachim> the installation).
>>>>
>>>> That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
>>>> in both packages A and B.
>>>>
>>>>     Install A - this will create C
>>>>     Install B - this might overwrite C, saving a copy, or it might retain
>>>>                 A's copy.
>>>>     Uninstall B - this has to know that C is used by A and not touch it
>>> Correct.  However, in practice, B should not touch C, unless the file
>>> is shared between them.
>>>
>>> This is a key issue for support of namespace packages, at least if we
>>> want to avoid using .pth files.  (Which is what setuptools-built
>>> system packages do for namespace packages currently.)
>>>
>>> Of course, one possible solution is for both A and B to depend on a
>>> "virtual package" that contains C, such that both A and B can install
>>> it if it's not there, and list it in their dependencies.  But this is
>>> one of the handful of open issues that needs to be resolved with Real
>>> Life Package Management people, such as Debian, Fedora, etc.
>> Debian (dpkg) does not allow a file to exist in more then one package
>> (distribution in distutils speak) as I said earlier.  Directories can
>> however, they will only be removed if they are empty (i.e. when the
>> last package having that directory is uninstalled).
> 
> This is a general problem with system package managers, 
> actually.  Few allow a file to be provided by more than one package.
> 
>> I'm not familiar with namespace packages as I've never used one, but I
>> assume it is just an empty directory with one (empty?)  __init__.py
>> file in it?
> 
> It's not empty; it has to contain a namespace declaration.  There's 
> another approach that can work around the absence of the __init__.py, 
> by using .pth files, but it's messy, and I'd like to get rid of it if possible.

If all Python distributions which install PythonPacakges under the
"namespace" package are bundeled as DistroPackages, then the content of
__init__.py won't really matter, will it?  There won't be more than one
copy of the "used-to-be namespace pacakge in this scenario.

If it *does* matter (e.g., the DistroPackages need to support use of
non-DistroPackage PythonPackages which use the same namespace), then all
those DistroPackages will need to depend on a nearly-empty,
generated-by-the-packager DistroPacakge which supplies only that file,
as Florian says next.

>>   The only way I can see Debian handling that is by having
>> a Debian package providing exactly that __init__.py and all other
>> Debian packages needing that namespace package will depend on it.
>>
>> However, this is shouldn't worry anyone other then Debian I think.
> 
> Alas, it'll probably affect most package managers.

Right:  RPM isn't happy about having a file owned by multiple packages
either.

>>  If
>> a distribution needs to depend on somthing it will depend on whoever
>> provides the module *inside* the namespace package, not on the
>> namespace itself.  So the fact that no distribution provides the
>> namespace.__init__.py file shouldn't have to worry other users of the
>> python installdb.
> 
> Well, somebody still has to create the __init__.py, and own it for 
> purposes of uninstallation.

I think that will need to be an "artifact" DistroPackage (one which has
no corresponding Python source distribution).

>> Since easy_install and others shouldn't be messing with debian
>> packages in /usr/lib (they should be using /usr/local/lib or
>> ~/.local [1]) this won't be a problem.




- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH5Fyv+gerLs4ltQ4RAsleAKCyF4qz0lULO1J6vAQMNZyaC+MWfQCbBzfD
/eoWYRwwLws24601BAMhdk0=
=dNgJ
-----END PGP SIGNATURE-----


From martin at v.loewis.de  Sat Mar 22 02:31:34 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 02:31:34 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321224110.657C63A40A5@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
Message-ID: <47E46176.4090205@v.loewis.de>

> I'm making the assumption that the author(s) of PEP 262 had good 
> reason for including what they did, rather than assuming that we 
> should start the entire process over from scratch.

The objections to the PEP remain the same as they were then,
though: In the requirements, it says "we need", without saying
why we need. It then goes on saying "we want" (rephrased)
"to duplicate APT and RPM", without saying why we want that,
or why that brings us closer to what we need.

IOW, the PEP is lacking a rationale.

>> Anything more complicated should be left to tools which are
>> specifically written to manage complex software setups.
> 
> Tools which will need this data, in order to do their work.  Hence, 
> the reason for standardizing the data, instead of the tool(s).

If there was a chance that the infrastructure being developed
actually helps these tools, *that* would be a reasonable goal,
IMO.

However, I'm extremely skeptical that this can ever succeed
to the degree that whoever provides RPMs, .debs, or MSI
files will actually use such data, as they will find that
the data are incomplete, and they have to redo all of it,
anyway.

> I'm proposing, rather, that we finish the vision of PEP 262, of 
> having a standard specification that *all* tools will abide by -- 
> including rpm, dpkg, and what-have-you.

Do you also envision the objective of PEP 262, then? I.e.
to provide a database of installed packages, in .../install-db?

> And as I said, I'll be happy if all we do is get the distutils to 
> abide by the spec for 2.6, even if it means we don't get an uninstall 
> tool.  That can always be installed later using Guido's bootstrap tool.  :)

I'm even more skeptical here. If the assumption is that the package
database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI
should then *not* write to that package database, as they also write
to a different database, out of the scope of the PEP, and this is
what uninstallation should use.

Regards,
Martin

From greg at krypto.org  Sat Mar 22 02:53:13 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 21 Mar 2008 18:53:13 -0700
Subject: [Distutils] [Python-Dev] Adventures with x64,
	VS7 and VS8 on Windows
In-Reply-To: <083301c863d2$12dc2b40$389481c0$@com.au>
References: <009f01c79b51$b1da52c0$1f0a0a0a@enfoldsystems.local>
	<46512B09.1080304@v.loewis.de>
	<022401c85cde$678bcf10$36a36d30$@com.au>
	<083301c863d2$12dc2b40$389481c0$@com.au>
Message-ID: <52dc1c820803211853h42876800ma9104e04a33efd0d@mail.gmail.com>

I'm following up on this thread without checking if there were other
following negating a need to respond... If so, ignore as needed.

+1 from me.  Always build on windows into an architecture specific
PCBuild/XXX directory.  A bonus if the directory name matches the return
value of platform.machine() but I don't care too much about that part.
(that'd mean 'i686' and 'x86_64' and who knows what for Itanic users)

On Wed, Jan 30, 2008 at 11:25 PM, Mark Hammond <mhammond at skippinet.com.au>
wrote:

> I'm wondering if anyone has any comment on this issue?  Its currently
> impossible to use distutils as documented to build x64 extensions, either
> natively or cross-compiled using the trunk and while I'm keen to help fix
> this, I'd like agreement on the direction.
>
> Can I assume silence means general agreement on the compromise proposal I
> outlined?
>
> Thanks,
>
> Mark
>
> > -----Original Message-----
> > From: python-dev-bounces+mhammond=keypoint.com.au at python.org
> > [mailto:python-dev-bounces+mhammond=keypoint.com.au at python.org] On
> > Behalf Of Mark Hammond
> > Sent: Tuesday, 22 January 2008 9:06 PM
> > To: '"Martin v. L?wis"'
> > Cc: distutils-sig at python.org; python-dev at python.org
> > Subject: Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
> >
> > Hi Martin,
> >
> > Way back on Monday, 21 May 2007, you wrote:
> >
> > > This is an issue to be discussed for Python 2.6. I'm personally
> > > hesitant to have the "official" build infrastructure deviate
> > > from the layout that has been in-use for so many years, as a lot
> > > of things depend on it.
> > >
> > > I don't find the need to have separate object directories convincing:
> > > For building the Win32/Win64 binaries, I have separate checkouts
> > > *anyway*, since all the add-on libraries would have to support
> > > multi-arch builds, but I think they don't.
> > ...
> > > > * Re the x64 directory used by the PCBuild8 process.  IMO, it makes
> > > > sense to generate x64 binaries to their own directory - my
> > expectation
> > > > is that cross-compiling between platforms is a reasonable use-case,
> > > > and we should support multiple achitectures for the same compiler
> > > > version.
> > >
> > > See above; I disagree.
> >
> > [For additional context; the question was regarding where the output
> > binaries were created for each platform, and specifically if x64 build
> > should use something like 'PCBuild/x64']
> >
> > I'm bringing this topic up again as I noticed the AMD64 builds for
> > Python
> > 2.6 create their output in the PCBuild/x64 directory, not directly into
> > the
> > PCBuild directory.  When I raised this with Christian, he also
> > expressed a
> > desire to be able to cross-compile in the same directory structure and
> > was
> > unaware of this discussion (but I made him aware of it :)
> >
> > You made excellent points about how tools currently recognize the
> > PCBuild
> > directory, and we can't break them, and I agree.  I'd like to propose a
> > compromise that might be able to keep everyone happy.
> >
> > The main problem I see with all platforms using 'PCBuild' is that in a
> > cross-compilation scenario, it is impossible for the distutils to
> > determine
> > the location of the correct Python libraries to link with (stuff in its
> > own
> > PYTHONHOME is no good; it's for the wrong platform).  Obviously, we can
> > change the distutils so that the location of the correct libraries can
> > be
> > specified, but that implies all other build systems that currently
> > assume
> > PCBuild must also change to work in a cross-compilation scenario.
> > While
> > those external tools *will* work today in a native build on x64,
> > eventually
> > they may need to be upgraded to deal with cross-compilation - and this
> > means
> > that all such tools will also be unable to automatically determine the
> > location of the libraries.  They will all need to take the library dir
> > as a
> > user-specified option, which seems a pain (eg, buildbots would seem to
> > need
> > site-specific configuration information to make this work).  If
> > eventually
> > all such tools are likely to upgrade, it would be ideal if we can offer
> > them
> > a way to upgrade so the correct libraries can be determined
> > automatically,
> > as they can now for native builds.
> >
> > My compromise proposal is this: Each platform builds in its own
> > directory
> > (ie, PCBuild/x86_64, PCBuild/x86).  Every single build configuration
> > has a
> > post-build step that copies the result of the build to the PCBuild
> > directory.  We then add an option to the distutils so that a cross-
> > compile
> > platform can be specified and when it is, to use 'PCBuild/{platform}"
> > instead of PCBuild, and we encourage other tools to use the same
> > directory
> > should they want to support "configure-less" cross-compilation (if
> > distutils
> > on the trunk is not still trying to maintain b/w compat, it should just
> > *always* look in PCBuild/{platform}, and I'd suggest py3k not bother
> > with
> > PCBuild at all; build systems will be touched to upgrade to py3k, so
> > that
> > change isn't painful.  But I digress :)
> >
> > The main advantage of the compromise is that it allows for the status
> > quo to
> > remain in place - just continue to use separate source trees for each
> > platform, and only ever build a single platform in each tree, and tools
> > are
> > free to have ways of specifying which directory should be used.  The
> > official build process need not change.  However, it also supports a
> > way to
> > do cross-compilation in a way that build tools can *automatically*
> > locate
> > the correct platform's libraries, and this will be particularly useful
> > for
> > extension authors.
> >
> > Would something like that be acceptable?
> >
> > Thanks,
> >
> > Mark
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-
> > dev/mhammond%40keypoint.com.au
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080321/ce86f0d0/attachment.htm 

From pje at telecommunity.com  Sat Mar 22 03:04:45 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 21 Mar 2008 22:04:45 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E46176.4090205@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
Message-ID: <20080322020447.3B1033A4074@sparrow.telecommunity.com>

At 02:31 AM 3/22/2008 +0100, Martin v. L?wis wrote:
>>I'm making the assumption that the author(s) of PEP 262 had good 
>>reason for including what they did, rather than assuming that we 
>>should start the entire process over from scratch.
>
>The objections to the PEP remain the same as they were then,
>though: In the requirements, it says "we need", without saying
>why we need. It then goes on saying "we want" (rephrased)
>"to duplicate APT and RPM", without saying why we want that,
>or why that brings us closer to what we need.
>
>IOW, the PEP is lacking a rationale.

Ok, well, I have a rationale for it: make it possible to get rid of 
eggs as a mechanism for supporting easy_install.  Many people 
(yourself included) have criticized eggs as an installation 
mechanism, and this is an alternative that gets rid of .egg files and 
directories in that case, and most of the need for .pth file usage as well.


>If there was a chance that the infrastructure being developed
>actually helps these tools, *that* would be a reasonable goal,
>IMO.

Yes, I'm of course primarily interested in Python-specific tools such 
as virtualenv, easy_install, buildout, and the as-yet-unwritten 
uninstallers, package listers, etc., that can usefully read or write such data.


>However, I'm extremely skeptical that this can ever succeed
>to the degree that whoever provides RPMs, .debs, or MSI
>files will actually use such data, as they will find that
>the data are incomplete, and they have to redo all of it,
>anyway.

The data isn't for them to use to meet their use cases, it's for them 
to *provide* so that Python tools don't stomp on, uninstall, or 
otherwise interfere with files installed by the system.  In other 
words, for system packagers, it's a communication from the system to 
Python, rather than the other way around.  Even though the distutils 
will build the file in the bdist, the system packaging tools would be 
free to generate their own file listing and signatures and such.


>Do you also envision the objective of PEP 262, then? I.e.
>to provide a database of installed packages, in .../install-db?

In each directory relative to a given sys.path directory, yes.  That 
is, installing a distutils distribution to any directory would result 
in a file being added to an install-db within that 
directory.  (Assuming we use the proposed implementation model of PEP 
262, which at the moment I don't see any substantial obstacle to.)



>>And as I said, I'll be happy if all we do is get the distutils to 
>>abide by the spec for 2.6, even if it means we don't get an 
>>uninstall tool.  That can always be installed later using Guido's 
>>bootstrap tool.  :)
>
>I'm even more skeptical here. If the assumption is that the package
>database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI
>should then *not* write to that package database, as they also write
>to a different database, out of the scope of the PEP, and this is
>what uninstallation should use.

I probably should have brought this up, in fact, I think I mentioned 
it in a previous thread, but I would like to see PEP 262 add a way to 
say "this is a system-installed package, *don't touch*".  The idea 
again is not to do the job of the native packaging system, but rather 
to ensure that Python-specific tools (e.g. easy_install and friends) 
do not interfere or conflict with it.

A big problem in the early development of easy_install, even using 
eggs, was that there was no way to tell whether it was safe to delete 
or overwrite an existing file or directory that was already installed 
on the system.  A mechanism like this would allow tools like 
easy_install to say, "oh, your system packager has a conflicting 
package here, you need to use that tool to sort this out if you 
really want to do something here.  I'm not going to touch 
that."  Without something like this, there is no way to tell the 
difference on many systems between a system package and something the 
user has put there with "sudo python setup.py install".


From talin at acm.org  Sat Mar 22 06:20:43 2008
From: talin at acm.org (Talin)
Date: Fri, 21 Mar 2008 22:20:43 -0700
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080321190226.4198D3A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E3D11F.4030503@online.de>	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
Message-ID: <47E4972B.30008@acm.org>

Phillip J. Eby wrote:
> At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
>>     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
>>     Joachim> files (and 'rmdir' directories, but not recursively) that it
>>     Joachim> created, and that have not been modified in the meantime (after
>>     Joachim> the installation).
>>
>> That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
>> in both packages A and B.
>>
>>     Install A - this will create C
>>     Install B - this might overwrite C, saving a copy, or it might retain
>>                 A's copy.
>>     Uninstall B - this has to know that C is used by A and not touch it
> 
> Correct.  However, in practice, B should not touch C, unless the file 
> is shared between them.
> 
> This is a key issue for support of namespace packages, at least if we 
> want to avoid using .pth files.  (Which is what setuptools-built 
> system packages do for namespace packages currently.)
> 
> Of course, one possible solution is for both A and B to depend on a 
> "virtual package" that contains C, such that both A and B can install 
> it if it's not there, and list it in their dependencies.  But this is 
> one of the handful of open issues that needs to be resolved with Real 
> Life Package Management people, such as Debian, Fedora, etc.

I've always thought that the right way to handle the dependency DAG is 
to treat it as a garbage collection problem.

Assume that for each package there is a way to derive the following two 
pieces of information: (a) whether this package was installed explicitly 
by the user, or implicitly as the result of a dependency, and (b) the 
set of dependencies for this package.

Then, starting with the list of 'explicit' packages as the root set, do 
a standard mark & sweep; Any package not marked is a candidate for removal.

-- Talin


From pje at telecommunity.com  Sat Mar 22 03:04:45 2008
From: pje at telecommunity.com (ext Phillip J. Eby)
Date: Fri, 21 Mar 2008 22:04:45 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E46176.4090205@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
Message-ID: <20080322020447.3B1033A4074@sparrow.telecommunity.com>

At 02:31 AM 3/22/2008 +0100, Martin v. L?wis wrote:
>>I'm making the assumption that the author(s) of PEP 262 had good 
>>reason for including what they did, rather than assuming that we 
>>should start the entire process over from scratch.
>
>The objections to the PEP remain the same as they were then,
>though: In the requirements, it says "we need", without saying
>why we need. It then goes on saying "we want" (rephrased)
>"to duplicate APT and RPM", without saying why we want that,
>or why that brings us closer to what we need.
>
>IOW, the PEP is lacking a rationale.

Ok, well, I have a rationale for it: make it possible to get rid of 
eggs as a mechanism for supporting easy_install.  Many people 
(yourself included) have criticized eggs as an installation 
mechanism, and this is an alternative that gets rid of .egg files and 
directories in that case, and most of the need for .pth file usage as well.


>If there was a chance that the infrastructure being developed
>actually helps these tools, *that* would be a reasonable goal,
>IMO.

Yes, I'm of course primarily interested in Python-specific tools such 
as virtualenv, easy_install, buildout, and the as-yet-unwritten 
uninstallers, package listers, etc., that can usefully read or write such data.


>However, I'm extremely skeptical that this can ever succeed
>to the degree that whoever provides RPMs, .debs, or MSI
>files will actually use such data, as they will find that
>the data are incomplete, and they have to redo all of it,
>anyway.

The data isn't for them to use to meet their use cases, it's for them 
to *provide* so that Python tools don't stomp on, uninstall, or 
otherwise interfere with files installed by the system.  In other 
words, for system packagers, it's a communication from the system to 
Python, rather than the other way around.  Even though the distutils 
will build the file in the bdist, the system packaging tools would be 
free to generate their own file listing and signatures and such.


>Do you also envision the objective of PEP 262, then? I.e.
>to provide a database of installed packages, in .../install-db?

In each directory relative to a given sys.path directory, yes.  That 
is, installing a distutils distribution to any directory would result 
in a file being added to an install-db within that 
directory.  (Assuming we use the proposed implementation model of PEP 
262, which at the moment I don't see any substantial obstacle to.)



>>And as I said, I'll be happy if all we do is get the distutils to 
>>abide by the spec for 2.6, even if it means we don't get an 
>>uninstall tool.  That can always be installed later using Guido's 
>>bootstrap tool.  :)
>
>I'm even more skeptical here. If the assumption is that the package
>database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI
>should then *not* write to that package database, as they also write
>to a different database, out of the scope of the PEP, and this is
>what uninstallation should use.

I probably should have brought this up, in fact, I think I mentioned 
it in a previous thread, but I would like to see PEP 262 add a way to 
say "this is a system-installed package, *don't touch*".  The idea 
again is not to do the job of the native packaging system, but rather 
to ensure that Python-specific tools (e.g. easy_install and friends) 
do not interfere or conflict with it.

A big problem in the early development of easy_install, even using 
eggs, was that there was no way to tell whether it was safe to delete 
or overwrite an existing file or directory that was already installed 
on the system.  A mechanism like this would allow tools like 
easy_install to say, "oh, your system packager has a conflicting 
package here, you need to use that tool to sort this out if you 
really want to do something here.  I'm not going to touch 
that."  Without something like this, there is no way to tell the 
difference on many systems between a system package and something the 
user has put there with "sudo python setup.py install".

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/jukka.p.laurila%40nokia.com

From steve at holdenweb.com  Sat Mar 22 11:25:18 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sat, 22 Mar 2008 06:25:18 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <47E4330C.5070601@egenix.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
Message-ID: <47E4DE8E.6010809@holdenweb.com>

M.-A. Lemburg wrote:
> On 2008-03-21 22:21, Phillip J. Eby wrote:
>> At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
>>> I guess the only way to support all of these variants is
>>> to use a filesystem based approach, e.g. by placing a file
>>> with a special extension into some dir on sys.path.
>>> The "database" logic could then scan sys.path for these
>>> files, read the data and provide an interface to it.
>>>
>>> All bdist formats would then have to include these files.
>> That's the idea behind the current version of PEP 262, yes, and I think 
>> it should be kept.
>>
>>> A separate FILES section also doesn't seem to be necessary -
>>> we could just add one or more entries or the format:
>>>
>>> CreatesDir abc/
>>> CreatesFile abc/xyz1.py
>>> CreatesDir abc/def/
>>> CreatesFile abc/def/xyz2.py
>>> CreatesFile abc/def/xyz3.py
>>> CreatesFile abc/def/xyz4.ini
>> I actually think the size and hash information is good, in order to be 
>> able to tell if you're looking at an original file.  I'm not sure how 
>> useful the permissions and uid/gid info is.  I'm hoping we'll hear from 
>> anybody who has a use case for that.
> 
> You're heading off in the wrong direction: we should not be trying
> to rewrite RPM or InnoSetup in Python.
> 
> Anything more complicated should be left to tools which are
> specifically written to manage complex software setups.
> 
We *do* need a way to play nice with all the various platform 
installers. This would surely be welcomed by the many third parties who 
now have to package Python for their platforms.

> I honestly believe that most people would be happy if we just
> provide these two things (and no more):
> 
>   * install a package from a local archive, a URL or PyPI
> 
>   * uninstall a package in way that doesn't break other
>     installed packages
> 
> and whatever the mechanism, avoid making any undercover
> changes to the Python installation such as adding
> .pth files, overriding site.py, etc. - these are
> not needed if the tool keeps to the simple task of
> installing and uninstalling Python packages.
> 
> Examples:
> 
> python pypi.py install mypkg-1.0.tgz
> python pypi.py install http://www.example.com/mypkg-1.0.tgz
> python pypi.py install mypkg-1.0
> 
> python pypi.py uninstall mypkg
> 
> If there's a dependency problem, the tool should print the
> list of other packages it needs. It should not try to install
> things automagically.
> 
... unless explicitly asked to do so? It seems silly to omit this 
capability when it's essentially just omitting a recursive call.

> If a package needs other modules as well, the package docs
> can point the user to use e.g.
> 
> python pypi.py install mydep1-1.3 mydep2-2.3 mydep4-0.3 mypkg-1.0
> 
> instead.
> 
Why would this be better than using --load-dependencies?

> Anything more complicated should be left to specialized
> tools such as RPM, apt, MSI or the other such tools out
> there - after all the tool should be about Python *package*
> installation, not application installation.
> 
> We *don't* need the tool to:
> 
>   * support multiple versions of a package (that's just bound
>     to cause problems with pickle, isinstance() etc.)
> 
Another argument for installing apps as separate components with all 
dependencies. I really don't believe enough consideration has been given 
as to the essential difference between these two activities.

>   * provide namespace hacking (is a completely separate issue
>     and can be handled by the packages rather than the install
>     tool)
> 
>   * support all kinds of funky version numbers (if a package
>     wants to participate in the system, the author better
>     make sure that the version string fits the standard format)
> 
Agreed.

>   * provide some form of intra-package bus interface (ie.
>     "entry points" as you call them)
> 
>   * provide support for keeping whole packages in ZIP files
>     (doesn't play well with C extensions, clutters up the
>     sys.path, is read-only, needs special importers, etc. etc. )
> 
It shouldn't require special importers, though. And if the zip file 
contains compiled code the read-only nature doesn't matter if the zips 
are version-specific.

>   * try automatic version matching for required packages
> 
>   * download things from SourceForge or other sites with special
>     download mechanisms
> 
>   * scan websites for links
> 
>   * make coffee, clean the house, send the kids to school :-)
> 
But a package that *would* do this could be immensely popular.

>> And of course, there are still some issues to be resolved regarding 
>> requirements, package name/version stuff, etc.  But we can hash those 
>> out once we reach a quorum on the Distutils-SIG.
> 
Well, I've probably been killfiled into non-existence on this list by 
now, but it seems to me that we are in danger of answering the wrong 
problem yet again. But that's all I have to say on this topic, so you 
can all heave a sigh a relief and get on with messing it up ;-)

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/

From steve at holdenweb.com  Sat Mar 22 11:25:18 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sat, 22 Mar 2008 06:25:18 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <47E4330C.5070601@egenix.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
Message-ID: <47E4DE8E.6010809@holdenweb.com>

M.-A. Lemburg wrote:
> On 2008-03-21 22:21, Phillip J. Eby wrote:
>> At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote:
>>> I guess the only way to support all of these variants is
>>> to use a filesystem based approach, e.g. by placing a file
>>> with a special extension into some dir on sys.path.
>>> The "database" logic could then scan sys.path for these
>>> files, read the data and provide an interface to it.
>>>
>>> All bdist formats would then have to include these files.
>> That's the idea behind the current version of PEP 262, yes, and I think 
>> it should be kept.
>>
>>> A separate FILES section also doesn't seem to be necessary -
>>> we could just add one or more entries or the format:
>>>
>>> CreatesDir abc/
>>> CreatesFile abc/xyz1.py
>>> CreatesDir abc/def/
>>> CreatesFile abc/def/xyz2.py
>>> CreatesFile abc/def/xyz3.py
>>> CreatesFile abc/def/xyz4.ini
>> I actually think the size and hash information is good, in order to be 
>> able to tell if you're looking at an original file.  I'm not sure how 
>> useful the permissions and uid/gid info is.  I'm hoping we'll hear from 
>> anybody who has a use case for that.
> 
> You're heading off in the wrong direction: we should not be trying
> to rewrite RPM or InnoSetup in Python.
> 
> Anything more complicated should be left to tools which are
> specifically written to manage complex software setups.
> 
We *do* need a way to play nice with all the various platform 
installers. This would surely be welcomed by the many third parties who 
now have to package Python for their platforms.

> I honestly believe that most people would be happy if we just
> provide these two things (and no more):
> 
>   * install a package from a local archive, a URL or PyPI
> 
>   * uninstall a package in way that doesn't break other
>     installed packages
> 
> and whatever the mechanism, avoid making any undercover
> changes to the Python installation such as adding
> .pth files, overriding site.py, etc. - these are
> not needed if the tool keeps to the simple task of
> installing and uninstalling Python packages.
> 
> Examples:
> 
> python pypi.py install mypkg-1.0.tgz
> python pypi.py install http://www.example.com/mypkg-1.0.tgz
> python pypi.py install mypkg-1.0
> 
> python pypi.py uninstall mypkg
> 
> If there's a dependency problem, the tool should print the
> list of other packages it needs. It should not try to install
> things automagically.
> 
... unless explicitly asked to do so? It seems silly to omit this 
capability when it's essentially just omitting a recursive call.

> If a package needs other modules as well, the package docs
> can point the user to use e.g.
> 
> python pypi.py install mydep1-1.3 mydep2-2.3 mydep4-0.3 mypkg-1.0
> 
> instead.
> 
Why would this be better than using --load-dependencies?

> Anything more complicated should be left to specialized
> tools such as RPM, apt, MSI or the other such tools out
> there - after all the tool should be about Python *package*
> installation, not application installation.
> 
> We *don't* need the tool to:
> 
>   * support multiple versions of a package (that's just bound
>     to cause problems with pickle, isinstance() etc.)
> 
Another argument for installing apps as separate components with all 
dependencies. I really don't believe enough consideration has been given 
as to the essential difference between these two activities.

>   * provide namespace hacking (is a completely separate issue
>     and can be handled by the packages rather than the install
>     tool)
> 
>   * support all kinds of funky version numbers (if a package
>     wants to participate in the system, the author better
>     make sure that the version string fits the standard format)
> 
Agreed.

>   * provide some form of intra-package bus interface (ie.
>     "entry points" as you call them)
> 
>   * provide support for keeping whole packages in ZIP files
>     (doesn't play well with C extensions, clutters up the
>     sys.path, is read-only, needs special importers, etc. etc. )
> 
It shouldn't require special importers, though. And if the zip file 
contains compiled code the read-only nature doesn't matter if the zips 
are version-specific.

>   * try automatic version matching for required packages
> 
>   * download things from SourceForge or other sites with special
>     download mechanisms
> 
>   * scan websites for links
> 
>   * make coffee, clean the house, send the kids to school :-)
> 
But a package that *would* do this could be immensely popular.

>> And of course, there are still some issues to be resolved regarding 
>> requirements, package name/version stuff, etc.  But we can hash those 
>> out once we reach a quorum on the Distutils-SIG.
> 
Well, I've probably been killfiled into non-existence on this list by 
now, but it seems to me that we are in danger of answering the wrong 
problem yet again. But that's all I have to say on this topic, so you 
can all heave a sigh a relief and get on with messing it up ;-)

regards
  Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From floris.bruynooghe at gmail.com  Sat Mar 22 12:00:10 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 22 Mar 2008 11:00:10 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <20080322020447.3B1033A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
Message-ID: <20080322110010.GA8513@laurie.devork>

On Fri, Mar 21, 2008 at 10:04:45PM -0400, Phillip J. Eby wrote:
> At 02:31 AM 3/22/2008 +0100, Martin v. L?wis wrote:
> >However, I'm extremely skeptical that this can ever succeed
> >to the degree that whoever provides RPMs, .debs, or MSI
> >files will actually use such data, as they will find that
> >the data are incomplete, and they have to redo all of it,
> >anyway.
> 
> The data isn't for them to use to meet their use cases, it's for them 
> to *provide* so that Python tools don't stomp on, uninstall, or 
> otherwise interfere with files installed by the system.  In other 
> words, for system packagers, it's a communication from the system to 
> Python, rather than the other way around.  Even though the distutils 
> will build the file in the bdist, the system packaging tools would be 
> free to generate their own file listing and signatures and such.

As long as systems (dpkg, rpm, ...) install the .egg-info files they
do communicate which modules/distributions are installed.  The
installdb would just duplicate this information (according to the
current PEP).


> >>And as I said, I'll be happy if all we do is get the distutils to 
> >>abide by the spec for 2.6, even if it means we don't get an 
> >>uninstall tool.  That can always be installed later using Guido's 
> >>bootstrap tool.  :)
> >
> >I'm even more skeptical here. If the assumption is that the package
> >database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI
> >should then *not* write to that package database, as they also write
> >to a different database, out of the scope of the PEP, and this is
> >what uninstallation should use.
> 
> I probably should have brought this up, in fact, I think I mentioned 
> it in a previous thread, but I would like to see PEP 262 add a way to 
> say "this is a system-installed package, *don't touch*".  The idea 
> again is not to do the job of the native packaging system, but rather 
> to ensure that Python-specific tools (e.g. easy_install and friends) 
> do not interfere or conflict with it.
> 
> A big problem in the early development of easy_install, even using 
> eggs, was that there was no way to tell whether it was safe to delete 
> or overwrite an existing file or directory that was already installed 
> on the system.  A mechanism like this would allow tools like 
> easy_install to say, "oh, your system packager has a conflicting 
> package here, you need to use that tool to sort this out if you 
> really want to do something here.  I'm not going to touch 
> that."  Without something like this, there is no way to tell the 
> difference on many systems between a system package and something the 
> user has put there with "sudo python setup.py install".

There is a way of telling if you have to keep you hands off a package
(sorry to bring this up again): installation paths.  /usr/lib is the
system path, the local admin (and hence setuptools) should keep their
hands off it at all times (unless requested with a --prefix or so for
building the debs or rpms but an uninstall in those cases won't be
required from setuptools).

What an installdb could help with is tell setuptools to keep it's
hands off a distribution manually installed (or by another tool) in
the local admin directory (/usr/local) or user directory (~/).  Your
proposal of an installdb for each directory on sys.path would solve
this, but the installdb in /usr/lib will be completely usused.

But frankly, for that just an extra field in the .egg-info
"Installed-By: setuptools" would do no?  Possibly followed by a
"X-Setuptools-Installed-Files:" section if you need that.

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From pje at telecommunity.com  Fri Mar 21 20:02:25 2008
From: pje at telecommunity.com (ext Phillip J. Eby)
Date: Fri, 21 Mar 2008 15:02:25 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <18403.57501.22865.578079@montanaro-dyndns-org.local>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
Message-ID: <20080321190226.4198D3A4074@sparrow.telecommunity.com>

At 11:21 AM 3/21/2008 -0500, skip at pobox.com wrote:
>     Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the
>     Joachim> files (and 'rmdir' directories, but not recursively) that it
>     Joachim> created, and that have not been modified in the meantime (after
>     Joachim> the installation).
>
>That's not sufficient.  Suppose file C (e.g. /usr/local/etc/mime.types) is
>in both packages A and B.
>
>     Install A - this will create C
>     Install B - this might overwrite C, saving a copy, or it might retain
>                 A's copy.
>     Uninstall B - this has to know that C is used by A and not touch it

Correct.  However, in practice, B should not touch C, unless the file 
is shared between them.

This is a key issue for support of namespace packages, at least if we 
want to avoid using .pth files.  (Which is what setuptools-built 
system packages do for namespace packages currently.)

Of course, one possible solution is for both A and B to depend on a 
"virtual package" that contains C, such that both A and B can install 
it if it's not there, and list it in their dependencies.  But this is 
one of the handful of open issues that needs to be resolved with Real 
Life Package Management people, such as Debian, Fedora, etc.

Neither overwriting, refusing to install, nor backups will properly 
address this issue.  However, this is properly a topic for the 
Distutils-SIG or whatever SIG the actual spec goes to.  On Python-Dev 
I'm only looking for a go/no-go on the overall approach.

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/jukka.p.laurila%40nokia.com

From martin at v.loewis.de  Sat Mar 22 12:33:49 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 12:33:49 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080322020447.3B1033A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
Message-ID: <47E4EE9D.2090605@v.loewis.de>

> Ok, well, I have a rationale for it: make it possible to get rid of eggs 
> as a mechanism for supporting easy_install.  Many people (yourself 
> included) have criticized eggs as an installation mechanism, and this is 
> an alternative that gets rid of .egg files and directories in that case, 
> and most of the need for .pth file usage as well.

How so? I cannot see the need for .egg files or .pth files in the
first place. If they exist so that you can import stuff: just install
to site-packages, and be done.

> The data isn't for them to use to meet their use cases, it's for them to 
> *provide* so that Python tools don't stomp on, uninstall, or otherwise 
> interfere with files installed by the system.  In other words, for 
> system packagers, it's a communication from the system to Python, rather 
> than the other way around.  Even though the distutils will build the 
> file in the bdist, the system packaging tools would be free to generate 
> their own file listing and signatures and such.

Ok, that's a reasonable requirement. It will be difficult to implement,
though, as it will require Linux distributors (in particular) to include
the database snippets in their packages.

Essentially, one would have to contribute patches to all the 
distributions (we care about, at least), and then nag the respective
maintainers to include these patches.

> I probably should have brought this up, in fact, I think I mentioned it 
> in a previous thread, but I would like to see PEP 262 add a way to say 
> "this is a system-installed package, *don't touch*".  The idea again is 
> not to do the job of the native packaging system, but rather to ensure 
> that Python-specific tools (e.g. easy_install and friends) do not 
> interfere or conflict with it.

Something like a read-only flag?

For those without the read-only flag, the specification should
explicitly say what manipulation is allowed.

Regards,
Martin

From asmodai at in-nomine.org  Fri Mar 21 13:54:02 2008
From: asmodai at in-nomine.org (ext Jeroen Ruigrok van der Werven)
Date: Fri, 21 Mar 2008 13:54:02 +0100
Subject: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources
	module)
In-Reply-To: <frua34$b7q$1@ger.gmane.org>
References: <ca471dc20803181343l1eea7805nb791d6494e4d4262@mail.gmail.com>
	<20080318223700.C64123A4074@sparrow.telecommunity.com>
	<ca471dc20803191048h14f16357idfca7d1cfbcb937f@mail.gmail.com>
	<20080319195438.A28DE3A40A5@sparrow.telecommunity.com>
	<ca471dc20803191423x7511364hf9df8b5edfd9c4b4@mail.gmail.com>
	<EBFE0802-2AC9-4FED-A641-F7B0E172C257@zooko.com>
	<79990c6b0803200233x5f9bd4b8tcac35b1a837a22fa@mail.gmail.com>
	<47E26A20.1090402@palladion.com> <47E29176.9050109@v.loewis.de>
	<frua34$b7q$1@ger.gmane.org>
Message-ID: <20080321125402.GB83712@nexus.in-nomine.org>

-On [20080320 19:24], Steve Holden (steve at holdenweb.com) wrote:
>We need to stop protesting that our installation tools are easy enough 
>and try to get behind the various platforms, be it with Windows 
>installers, rpms, or other support. We probably aren't doing this 
>because it's work nobody particularly relishes, and has relatively low 
>visibility in the developer world. Non-developer Python programmers and 
>end-users would thank us, though.

FreeBSD offers through install of Perl through its ports system a Perl
module called 'bsdpan' which registers every module as a package under
FreeBSD's package system.

Normally ports installs modules as p5-ModuleName, but now it becomes:
/var/db/pkg/bsdpan-B-Lint-1.09

And from that point on I can use the pkg* tools.

Quite elegant in my opinion.

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/
To regard the fundamental as the essence, to regard things as coarse, to
regard accumulation as deficiency, and to dwell quietly alone with the
spiritual and the intelligent -- herein lie the techniques of Tao of
the ancients.
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/jukka.p.laurila%40nokia.com

From lxander.m at gmail.com  Sat Mar 22 14:21:18 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Sat, 22 Mar 2008 09:21:18 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E46176.4090205@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
Message-ID: <525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>

On Fri, Mar 21, 2008 at 9:31 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>  The objections to the PEP remain the same as they were then,
>  though: In the requirements, it says "we need", without saying
>  why we need. It then goes on saying "we want" (rephrased)
>  "to duplicate APT and RPM", without saying why we want that,
>  or why that brings us closer to what we need.
>
>  IOW, the PEP is lacking a rationale.

It seems to me that this discussion is being undermined by not
acknowledging the many use cases up front. There is no rationale
because there are too many tacit rationales. Nevertheless, the many
use cases are related at some level and would benefit from some form
of lowest-common-denominator support in the standard library. As an
example, IF I wanted to use a library to support multi-version
packages or one to ensure I had all the dependencies, I would need to
know what versions of things were currently installed. I can't create
a library to provided these functionalities and use it downstream of
the package creator without some form of standardization to report
package versions. (Or at least without running into the assimilation
problem that setuptools has).

My personal use case is for multi-version packages. I deploy many
small scripts (not applications) that use an evolving set of base
libraries. I don't want the older scripts to hold me back from pushing
forward with the base libraries, so I peg the scripts to their
respective major versions as I release them.

Regards,
Alex

From floris.bruynooghe at gmail.com  Sat Mar 22 14:24:18 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 22 Mar 2008 13:24:18 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <47E4EE9D.2090605@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
Message-ID: <20080322132418.GB8513@laurie.devork>

On Sat, Mar 22, 2008 at 12:33:49PM +0100, "Martin v. L?wis" wrote:
> > The data isn't for them to use to meet their use cases, it's for them to 
> > *provide* so that Python tools don't stomp on, uninstall, or otherwise 
> > interfere with files installed by the system.  In other words, for 
> > system packagers, it's a communication from the system to Python, rather 
> > than the other way around.  Even though the distutils will build the 
> > file in the bdist, the system packaging tools would be free to generate 
> > their own file listing and signatures and such.
> 
> Ok, that's a reasonable requirement. It will be difficult to implement,
> though, as it will require Linux distributors (in particular) to include
> the database snippets in their packages.
> 
> Essentially, one would have to contribute patches to all the 
> distributions (we care about, at least), and then nag the respective
> maintainers to include these patches.

Not true.  You just need to make sure that "setup.py install" creates
that database.  With the proposed format of the database this is just
a file in the correct location - exactly for this reason.  Next time
the distribution will build the package that database file will be in
place.

I still maintain that an installdb for the system packages doesn't
bring anything to the party as it would be in a system-only directory.
But I've argued that in my other emails...


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From p.f.moore at gmail.com  Sat Mar 22 14:46:42 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 22 Mar 2008 13:46:42 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E4DE8E.6010809@holdenweb.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com> <47E4DE8E.6010809@holdenweb.com>
Message-ID: <79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>

On 22/03/2008, Steve Holden <steve at holdenweb.com> wrote:
> Well, I've probably been killfiled into non-existence on this list by
>  now, but it seems to me that we are in danger of answering the wrong
>  problem yet again. But that's all I have to say on this topic, so you
>  can all heave a sigh a relief and get on with messing it up ;-)

You probably have my company in the killfile, but I have a nagging
feeling in the same direction. My biggest problem is that I can't
express what I believe is the *right* problem, beyond the over-general
statement that it seems crucial to me that there should be a single,
unified way of managing *all* packages installed in a given Python
installation. Whether that's a Python-only solution, or the system
packager, doesn't matter. There should be only one way to do it, to
reuse a well-known phrase :-)

If you know how to state nature of the right problem, that would be useful.

All this talk of "playing nicely with the system packager" seems to
imply that people are designing a second solution, and trying to
manage the interaction, rather than deciding on *one* solution (which
by definition has no interaction to worry about).

It's reasonable to have multiple solutions for multiple Python
installations (system packager for the system python, python packager
for a local install, for example) but that's a different matter.

Oh, and application installation is (should be) completely different.
On Windows, applications should probably be bundled with their own
Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
standard is, so I'd have to defer to others.

Paul.

From martin at v.loewis.de  Sat Mar 22 15:02:33 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 15:02:33 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	
	<47E40726.1090209@egenix.com>	
	<20080321212127.791B63A4074@sparrow.telecommunity.com>	
	<47E4330C.5070601@egenix.com>	
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>	
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
Message-ID: <47E51179.5040601@v.loewis.de>

> It seems to me that this discussion is being undermined by not
> acknowledging the many use cases up front. There is no rationale
> because there are too many tacit rationales.

I honestly, really, cannot imagine what those are. Explicit is
better than implicit.

> Nevertheless, the many
> use cases are related at some level and would benefit from some form
> of lowest-common-denominator support in the standard library. As an
> example, IF I wanted to use a library to support multi-version
> packages or one to ensure I had all the dependencies, I would need to
> know what versions of things were currently installed.

How would you install multiple versions in the first place? Python
supports no such thing, at least not without setting PYTHONPATH,
or otherwise changing sys.path.

> My personal use case is for multi-version packages. I deploy many
> small scripts (not applications) that use an evolving set of base
> libraries. I don't want the older scripts to hold me back from pushing
> forward with the base libraries, so I peg the scripts to their
> respective major versions as I release them.

I could not have imagined that as a use case. I never install multiple
versions of the same thing on the same system, except for Python itself.
I also try to avoid using evolving libraries as much as possible,
and quickly abandon libraries if they change in an incompatible manner
across releases, at least if porting becomes too much of a burden.

Notice that this use case isn't listed in the PEP, and I cannot see
how the PEP can help providing that functionality.

Regards,
Martin

From p.f.moore at gmail.com  Sat Mar 22 15:14:01 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 22 Mar 2008 14:14:01 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
Message-ID: <79990c6b0803220714q7e2f2593wa46694ede30f594b@mail.gmail.com>

On 22/03/2008, Alexander Michael <lxander.m at gmail.com> wrote:
>  >  IOW, the PEP is lacking a rationale.
>
> It seems to me that this discussion is being undermined by not
>  acknowledging the many use cases up front. There is no rationale
>  because there are too many tacit rationales.

Absolutely! It feels like people are trying to design a solution
without having written up the problem spec. Maybe that's because there
are too many problems for a single solution to work in all cases?

>  My personal use case is for multi-version packages. I deploy many
>  small scripts (not applications) that use an evolving set of base
>  libraries. I don't want the older scripts to hold me back from pushing
>  forward with the base libraries, so I peg the scripts to their
>  respective major versions as I release them.

My personal use case is for a Windows system with a number of adhoc
scripts, which may depend on 3rd party libraries, and a number of
"applications" (3rd party or otherwise) written in Python.

I want the applications to use a bundled Python and library, via
py2exe. There should be no dependency on the "system" Python.

Adhoc scripts can depend on libraries installed into the system
Python, but generally there will be a "base" set of libraries that I
will always have installed and which can be assumed to be present
(pywin32, cx_Oracle, py2exe, ...). Scripts can use these without
restriction but should be prepared to work with whatever version is
present. It should (almost) never be a problem to a script if a
library gets upgraded.

Additional libraries may be installed in the system Python, for
one-off jobs, or for testing and development. I will install and
remove these at will, and nothing critical should break ifn I do.

There is generally no other Python present on the system. On my
development machine, I may have alternative versions installed, and I
may have trunk checkouts with a python.exe built, but these will never
be used for anything other than specific development tasks, and are
treated as "throw-away". It's very rare that any 3rd party library
will be installed in these, and special handling when it does happen
is entirely acceptable.

For the system Python, I need:
- a single way to list what's installed (including version)
- a single way to uninstall items as needed
- a way (or more than one) to install 3rd party software *which ties
into the above*

The key maintenance task I do on the system Python is to list
everything installed, uninstall them all, upgrade the system Python,
then reinstall the ones I had installed previously. (Don't bother
telling me there are other ways I could do this - that's not the point
here, this is how I actually work).

The reason setuptools/easy_install breaks this is because it fails the
"single way" test. I have to use bdist_wininst for some packages, so
setuptools *has* to integrate with that. Also, setuptools doesn't have
the list and uninstall features.

So that's my problem/requirements. Anything that solves this gets +1
from me. Anything else gets -1 or at best -0. Anything that adds extra
features while not solving my problem gets a strong -1, as the extra
features will encourage take-up of that solution, exacerbating my
current problem.

Paul.

From martin at v.loewis.de  Sat Mar 22 15:14:05 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 15:14:05 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
 beyond
In-Reply-To: <20080322132418.GB8513@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322132418.GB8513@laurie.devork>
Message-ID: <47E5142D.7010902@v.loewis.de>

>> Essentially, one would have to contribute patches to all the 
>> distributions (we care about, at least), and then nag the respective
>> maintainers to include these patches.
> 
> Not true.  You just need to make sure that "setup.py install" creates
> that database.  With the proposed format of the database this is just
> a file in the correct location - exactly for this reason.  Next time
> the distribution will build the package that database file will be in
> place.

How so? Are you /sure/ the packaging process even *runs* setup.py?
And if they do, why do you think they will pick up the database
file?

Regards,
Martin

From p.f.moore at gmail.com  Sat Mar 22 15:17:20 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 22 Mar 2008 14:17:20 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E51179.5040601@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
Message-ID: <79990c6b0803220717u7e0208a9na15a1531b48fd545@mail.gmail.com>

On 22/03/2008, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> How would you install multiple versions in the first place? Python
>  supports no such thing, at least not without setting PYTHONPATH,
>  or otherwise changing sys.path.

That's an unrelated feature of setuptools, providing a way to
"install" multiple versions of a package and select which version you
want at runtime. It does this by sys.path manipulation, I believe.

It's a good example of the sort of orthogonal feature which makes
setuptools such an issue. It's a genuinely useful feature to some
people, but it's unrelated to packaging and installation, so people
buy into a packaging solution (eggs) for the non-packaging benefits
they provide.

Paul.

From floris.bruynooghe at gmail.com  Sat Mar 22 15:31:16 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 22 Mar 2008 14:31:16 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com> <47E4DE8E.6010809@holdenweb.com>
	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
Message-ID: <20080322143116.GA15221@laurie.devork>

On Sat, Mar 22, 2008 at 01:46:42PM +0000, Paul Moore wrote:
> All this talk of "playing nicely with the system packager" seems to
> imply that people are designing a second solution, and trying to
> manage the interaction, rather than deciding on *one* solution (which
> by definition has no interaction to worry about).
> 
> It's reasonable to have multiple solutions for multiple Python
> installations (system packager for the system python, python packager
> for a local install, for example) but that's a different matter.

My server runs stable version of Debian, etch currently.  It has
python installed and most of the modules I need.  However not all
possible python modules are packaged but I as the local admin might
decide a certain module -say pyprocessing- is really required for us
and stable enough to use.  I now need to install this locally.  What I
don't want in install an entire new python and hunt *all* modules and
then make sure that the correct applications on my server use the
correct python.

> Oh, and application installation is (should be) completely different.
> On Windows, applications should probably be bundled with their own
> Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
> standard is, so I'd have to defer to others.

An application packaged by the system (Debian in my case) will need to
use the system modules.  For the same reason we invented shared
libraries instead of static libraries.  When I have to install an
application as local admin then I try to run it with the system
python, using as many system modules as possible.  If that's not
possible (e.g. I need a sandbox) I'll often seriously reconsider
installing that application.

As a user I have no problems with trying out an app in virualenv or
something like it.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ndbecker2 at gmail.com  Sat Mar 22 15:45:35 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sat, 22 Mar 2008 10:45:35 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6	and beyond
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322132418.GB8513@laurie.devork>
	<47E5142D.7010902@v.loewis.de>
Message-ID: <fs362f$n25$1@ger.gmane.org>

"Martin v. L?wis" wrote:

>>> Essentially, one would have to contribute patches to all the
>>> distributions (we care about, at least), and then nag the respective
>>> maintainers to include these patches.
>> 
>> Not true.  You just need to make sure that "setup.py install" creates
>> that database.  With the proposed format of the database this is just
>> a file in the correct location - exactly for this reason.  Next time
>> the distribution will build the package that database file will be in
>> place.
> 
> How so? Are you /sure/ the packaging process even *runs* setup.py?
> And if they do, why do you think they will pick up the database
> file?
> 

In the case of Fedora rpms, the usual install uses setup.py.


From pje at telecommunity.com  Sat Mar 22 16:12:12 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 22 Mar 2008 11:12:12 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <20080322110010.GA8513@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<20080322110010.GA8513@laurie.devork>
Message-ID: <20080322151218.7EDB73A40A5@sparrow.telecommunity.com>

At 11:00 AM 3/22/2008 +0000, Floris Bruynooghe wrote:
>As long as systems (dpkg, rpm, ...) install the .egg-info files they
>do communicate which modules/distributions are installed.  The
>installdb would just duplicate this information (according to the
>current PEP).

.egg-info/PKG-INFO don't list the specific files, though.


>There is a way of telling if you have to keep you hands off a package
>(sorry to bring this up again): installation paths.  /usr/lib is the
>system path, the local admin (and hence setuptools) should keep their
>hands off it at all times (unless requested with a --prefix or so for
>building the debs or rpms but an uninstall in those cases won't be
>required from setuptools).

As I mentioned previously, if the spec says anything about specific 
paths, it will be full of fail.  The spec MUST be able to work with 
*any* local policy about where Python packages are to be 
installed.  Otherwise, any tool that wants to work with install-dbs 
will end up accumulating a long list of paths to be handled specially 
for each OS vendor and version...  and still not handle everything.  No can do.

This has to be a mechanism, not a policy.  Vendors and admins should 
be able to enforce reasonable policies, without requiring that every 
tool have those policies built in.  For one thing, it's an entry 
barrier to tools.

Basically, what I'm proposing here is like WSGI for package 
management tools -- and building anything about paths into the spec 
would be like WSGI spelling out what pages should be at what URLs!


From pje at telecommunity.com  Sat Mar 22 16:19:17 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 22 Mar 2008 11:19:17 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E4EE9D.2090605@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
Message-ID: <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>

At 12:33 PM 3/22/2008 +0100, Martin v. L?wis wrote:
>>I probably should have brought this up, in fact, I think I 
>>mentioned it in a previous thread, but I would like to see PEP 262 
>>add a way to say "this is a system-installed package, *don't 
>>touch*".  The idea again is not to do the job of the native 
>>packaging system, but rather to ensure that Python-specific tools 
>>(e.g. easy_install and friends) do not interfere or conflict with it.
>
>Something like a read-only flag?

Not exactly.  More like, "package management tool X claims exclusive 
rights to this package".  Python tools would always defer this right 
to the system packager, i.e. a system packager is not obliged to 
respect a Python tool's claim to a file, but not the other way around.

That way, system packaging tools don't need to do anything but mark 
the installed files as belonging to them.

Since most vendors at least *begin* with a "setup.py install", we 
could provide a way to indicate that the installation is being done 
on behalf of a system packaging tool, so that it can provide that indication.


>For those without the read-only flag, the specification should
>explicitly say what manipulation is allowed.

Since a distribution isn't really "mutable", I would think that 
uninstallation and reinstallation would be the only manipulation 
available.  (As distinct from inspection, verification, and other 
read-only activities.)

It's possible, though, that there might also be actions such as 
restoring or relocating scripts or data in shared locations outside 
of the sys.path directory.  That will get clearer as the spec gets defined.


From floris.bruynooghe at gmail.com  Sat Mar 22 16:21:50 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 22 Mar 2008 15:21:50 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6
	and	beyond
In-Reply-To: <47E5142D.7010902@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322132418.GB8513@laurie.devork>
	<47E5142D.7010902@v.loewis.de>
Message-ID: <20080322152150.GA16277@laurie.devork>

On Sat, Mar 22, 2008 at 03:14:05PM +0100, "Martin v. L?wis" wrote:
>>> Essentially, one would have to contribute patches to all the  
>>> distributions (we care about, at least), and then nag the respective
>>> maintainers to include these patches.
>>
>> Not true.  You just need to make sure that "setup.py install" creates
>> that database.  With the proposed format of the database this is just
>> a file in the correct location - exactly for this reason.  Next time
>> the distribution will build the package that database file will be in
>> place.
>
> How so? Are you /sure/ the packaging process even *runs* setup.py?
> And if they do, why do you think they will pick up the database
> file?

I speak for Debian, so for Debian: yes.  The setup.py would have to be
pretty bad for a packager to not use it.  There is no reason to
re-write upstream's installation procedure as you would have to figure
out which files need to be installed where and this would create many
bugs.

The canonical way is something like this:

  $ pythonX.Y setup.py --root=$(pwd)/debian/tmp
  $ # Fixup anything done wrong/badly (for debian) by setup.py
  $ # Make a tarball of $(pwd)/debian/tmp

In reality it's slightly more complicated of course.  At [1] there are
many packages, paste and parallelpython are good examples if you're
interested (look in the debian/rules file).


Regards
Floris

[1] http://svn.debian.org/wsvn/python-modules/packages/?rev=0&sc=0

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From pje at telecommunity.com  Sat Mar 22 16:25:45 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 22 Mar 2008 11:25:45 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <79990c6b0803220714q7e2f2593wa46694ede30f594b@mail.gmail.co
 m>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<79990c6b0803220714q7e2f2593wa46694ede30f594b@mail.gmail.com>
Message-ID: <20080322152549.E4C943A40A5@sparrow.telecommunity.com>

At 02:14 PM 3/22/2008 +0000, Paul Moore wrote:
>For the system Python, I need:
>- a single way to list what's installed (including version)
>- a single way to uninstall items as needed
>- a way (or more than one) to install 3rd party software *which ties
>into the above*

Right, and the PEP effort is devoted to having one way to store the 
information required, to tie these things together.  If there is a 
standard way to store that info on your system, then it doesn't 
matter how many tools there are, you still have your "one way" to 
list what's installed or to uninstall things, because you just pick 
the one lister and/or uninstaller whose UI you prefer.


From martin at v.loewis.de  Sat Mar 22 16:29:57 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 16:29:57 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>
Message-ID: <47E525F5.40705@v.loewis.de>

>> For those without the read-only flag, the specification should
>> explicitly say what manipulation is allowed.
> 
> Since a distribution isn't really "mutable", I would think that 
> uninstallation and reinstallation would be the only manipulation 
> available.  (As distinct from inspection, verification, and other 
> read-only activities.)

Sure, but what is precisely the semantics of uninstallation, in
terms of changes to the system state?

I think any model where uninstallation is merely the removal
of files is too limited to be practical.

Regards,
Martin

From pje at telecommunity.com  Sat Mar 22 16:33:02 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 22 Mar 2008 11:33:02 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>
Message-ID: <20080322153304.B7CAD3A40B1@sparrow.telecommunity.com>

At 11:19 AM 3/22/2008 -0400, Phillip J. Eby wrote:
>Not exactly.  More like, "package management tool X claims exclusive
>rights to this package".  Python tools would always defer this right
>to the system packager, i.e. a system packager is not obliged to
>respect a Python tool's claim to a file, but not the other way around.
>
>That way, system packaging tools don't need to do anything but mark
>the installed files as belonging to them.

This probably needs to be refined a little.  Exclusive right is too 
strong, and it goes against Paul Moore's desire for using a single 
tool.  Perhaps instead what it should be is an "uninstall warning" 
field that must be displayed to a user if an interactive program is 
doing uninstallation, and that a non-interactive program must refuse 
to uninstall unless explicitly requested to go ahead.

Unfortunately, a warning message might then need to be localized.  So 
this idea still needs some work.


From martin at v.loewis.de  Sat Mar 22 16:42:36 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 16:42:36 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <20080322152150.GA16277@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322132418.GB8513@laurie.devork>
	<47E5142D.7010902@v.loewis.de>
	<20080322152150.GA16277@laurie.devork>
Message-ID: <47E528EC.7010605@v.loewis.de>

> I speak for Debian, so for Debian: yes.  The setup.py would have to be
> pretty bad for a packager to not use it.  There is no reason to
> re-write upstream's installation procedure as you would have to figure
> out which files need to be installed where and this would create many
> bugs.
> 
> The canonical way is something like this:
> 
>   $ pythonX.Y setup.py --root=$(pwd)/debian/tmp
>   $ # Fixup anything done wrong/badly (for debian) by setup.py
>   $ # Make a tarball of $(pwd)/debian/tmp
> 
> In reality it's slightly more complicated of course.

More than slightly so, with pycentral, pysupport, and all that.

I still doubt that the database would show up in a directory on
sys.path. IIUC, things get install into
/usr/share/pycentral/package, and then get symlinked into
/usr/lib/pythonX.Y/site-packages at installation time. It's
not all that clear to me how that deals with "non-regular"
files.

> At [1] there are
> many packages, paste and parallelpython are good examples if you're
> interested (look in the debian/rules file).

I started with nevow. It uses cdbs, so its debian/rules is nearly
empty, and does not include a call to setup.py at all.

Instead, the distutils support is in

/usr/share/cdbs/1/class/python-distutils.mk

where setup.py is invoked as

ifeq (,$(DEB_PYTHON_REAL_LIB_PACKAGES))
common-install-arch common-install-indep:: common-install-impl
common-install-impl::
         cd $(DEB_SRCDIR) && python$(DEB_PYTHON_COMPILE_VERSION) 
$(DEB_PYTHON_SETUP_CMD) install --root=$(DEB_DESTDIR) 
$(DEB_PYTHON_INSTALL_ARGS_ALL) $(DEB_PYTHON_INSTALL_ARGS_$(cdbs_curpkg))
else
$(patsubst %,install/%,$(DEB_PYTHON_REAL_LIB_PACKAGES)) :: install/% :
         cd $(DEB_SRCDIR) && python$(cdbs_python_ver) 
$(DEB_PYTHON_SETUP_CMD) install --root=$(CURDIR)/debian/$(cdbs_curpkg) 
$(DEB_PYTHON_INSTALL_ARGS_ALL) $(DEB_PYTHON_INSTALL_ARGS_$(cdbs_curpkg))
endif

$(patsubst %,install/%,$(DEB_PYTHON_SIMPLE_PACKAGES)) :: install/% :
         cd $(DEB_SRCDIR) && python $(DEB_PYTHON_SETUP_CMD) install 
--root=$(CURDIR)/debian/$(cdbs_curpkg) $(DEB_PYTHON_INSTALL_ARGS_ALL) 
$(DEB_PYTHON_INSTALL_ARGS_$(cdbs_curpkg))

I cannot infer from that whether supplying a package database
snippet in setup.py would actually work.

Regards,
Martin


From p.f.moore at gmail.com  Sat Mar 22 16:44:32 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 22 Mar 2008 15:44:32 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080322153304.B7CAD3A40B1@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>
	<20080322153304.B7CAD3A40B1@sparrow.telecommunity.com>
Message-ID: <79990c6b0803220844u3499e83cwae684867d6d1e02d@mail.gmail.com>

On 22/03/2008, Phillip J. Eby <pje at telecommunity.com> wrote:
> This probably needs to be refined a little.  Exclusive right is too
>  strong, and it goes against Paul Moore's desire for using a single
>  tool.

Huh? How's that? Don't forget that I'm on Windows, and on Windows
there is no "system tool" - just bdist_wininst, bdist_msi and
easy_install. The fact that bdist_wininst and bdist_msi link into the
system UI for listing and uninstallation doesn't make packages using
them "system packages". If easy_install put add/remove program info in
place, my "single tool" would be the add/remove list. If something
else (for example, the proposed index of installed package, with a
suitable UI) is deemed the "single tool", then bdist_wininst and
bdist_msi have to be changed to respect that.

The only fly in this ointment is bdist_msi, which uses MSI format,
which is a lot nearer to a Windows "system packager" than anything
else. Whether that means bdist_msi can't be changed to work with a
package index rather than (or as well as, I don't care) add/remove, I
don't know.

>  Unfortunately, a warning message might then need to be localized.  So
>  this idea still needs some work.

The "one way to do it" uninstaller should be able to uninstall
everything. If it needs to call the system uninstaller for a specific
package, there's nothing wrong with doing that. But why tell me to run
a different command? Just do it for me. I only want one UI, the rest
is implementation detail.

(Others may have different preferences, so a choice may need to be
made. If so, and if it goes against me, fair enough, I have to decide
what to do about that for myself. But I'd rather force people to tell
me "no", than leave people thinking they satisfied me and wondering
why I'm still complaining...)

Paul.

From pje at telecommunity.com  Sat Mar 22 16:44:48 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 22 Mar 2008 11:44:48 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E525F5.40705@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>
	<47E525F5.40705@v.loewis.de>
Message-ID: <20080322154449.BDA423A40A5@sparrow.telecommunity.com>

At 04:29 PM 3/22/2008 +0100, Martin v. L?wis wrote:
>>>For those without the read-only flag, the specification should
>>>explicitly say what manipulation is allowed.
>>Since a distribution isn't really "mutable", I would think that 
>>uninstallation and reinstallation would be the only manipulation 
>>available.  (As distinct from inspection, verification, and other 
>>read-only activities.)
>
>Sure, but what is precisely the semantics of uninstallation, in
>terms of changes to the system state?
>
>I think any model where uninstallation is merely the removal
>of files is too limited to be practical.

The distutils only support the *addition* of files, so I'm not sure 
how only removing files is a limit here.  Could you explain?





From martin at v.loewis.de  Sat Mar 22 16:45:08 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 22 Mar 2008 16:45:08 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
 beyond
In-Reply-To: <fs362f$n25$1@ger.gmane.org>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<20080321224110.657C63A40A5@sparrow.telecommunity.com>	<47E46176.4090205@v.loewis.de>	<20080322020447.3B1033A4074@sparrow.telecommunity.com>	<47E4EE9D.2090605@v.loewis.de>	<20080322132418.GB8513@laurie.devork>	<47E5142D.7010902@v.loewis.de>
	<fs362f$n25$1@ger.gmane.org>
Message-ID: <47E52984.3070407@v.loewis.de>

> In the case of Fedora rpms, the usual install uses setup.py.

Ok. Does it then also package all files that get installed into
the RPM file? If it produces multiple RPMs from a single source
package, how does it know which files go into what RPM?

Regards,
Martin

From terry at jon.es  Sat Mar 22 16:45:31 2008
From: terry at jon.es (Terry Jones)
Date: Sat, 22 Mar 2008 16:45:31 +0100
Subject: [Distutils] Question on using distutils.core.run_setup
In-Reply-To: Your message at 11:53:12 on Wednesday, 19 March 2008
References: <18399.21206.481836.992152@jon.es>
	<20080318163429.ABE703A4074@sparrow.telecommunity.com>
	<18400.33406.254201.374962@jon.es>
	<20080319045541.7E9263A4074@sparrow.telecommunity.com>
	<18400.44058.555396.751733@jon.es>
	<20080319155315.D0E9F3A40A5@sparrow.telecommunity.com>
Message-ID: <18405.10651.22256.387600@jon.es>

>>>>> "Phillip" == Phillip J Eby <pje at telecommunity.com> writes:
Phillip> At 07:00 AM 3/19/2008 +0100, Terry Jones wrote:
>> I guess I'm missing something here?

Phillip> Whoops, my bad.  'get_finalized_command' is a distutils *command
Phillip> object* method, not a distribution method.  I usually work with
Phillip> code in command objects much more than with distribution objects.
[snip]

OK.

Thanks a lot again,
Terry

From ndbecker2 at gmail.com  Sat Mar 22 17:00:18 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sat, 22 Mar 2008 12:00:18 -0400
Subject: [Distutils]
	=?utf-8?q?=5BPython-Dev=5D__How_we_can_get_rid_of_egg?=
	=?utf-8?q?s_for_2=2E6=09and_beyond?=
In-Reply-To: <47E52984.3070407@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<fs362f$n25$1@ger.gmane.org> <47E52984.3070407@v.loewis.de>
Message-ID: <200803221200.18772.ndbecker2@gmail.com>

On Saturday 22 March 2008, Martin v. L?wis wrote:
> > In the case of Fedora rpms, the usual install uses setup.py.
>
> Ok. Does it then also package all files that get installed into
> the RPM file? If it produces multiple RPMs from a single source
> package, how does it know which files go into what RPM?
>
> Regards,
> Martin

Offhand, I can't think of any examples that produce multiple RPMS, but in 
general, it doesn't _know_ anything.  What goes in what rpm is either
1) Automated using
  python setup.py install -O1 --root 
$RPM_BUILD_ROOT --prefix %{_prefix} --record=%{name}.files

or

2) Manually listing the files that go into a package.

From martin at v.loewis.de  Sat Mar 22 17:06:05 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 17:06:05 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <79990c6b0803220844u3499e83cwae684867d6d1e02d@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<20080321224110.657C63A40A5@sparrow.telecommunity.com>	<47E46176.4090205@v.loewis.de>	<20080322020447.3B1033A4074@sparrow.telecommunity.com>	<47E4EE9D.2090605@v.loewis.de>	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>	<20080322153304.B7CAD3A40B1@sparrow.telecommunity.com>
	<79990c6b0803220844u3499e83cwae684867d6d1e02d@mail.gmail.com>
Message-ID: <47E52E6D.5090303@v.loewis.de>

> Huh? How's that? Don't forget that I'm on Windows, and on Windows
> there is no "system tool" - just bdist_wininst, bdist_msi and
> easy_install. The fact that bdist_wininst and bdist_msi link into the
> system UI for listing and uninstallation doesn't make packages using
> them "system packages".

In pje's terminology, they do. He uses "system package" as a shorthand
for "package in a format defined by the system vendor", not as
"package supplied by the system vendor" (IIUC). So .msi files and
self-extracting .exe files are all "system packages", as opposed to
.eggs (which are in a format that wasn't defined by the system vendor).

> The only fly in this ointment is bdist_msi, which uses MSI format,
> which is a lot nearer to a Windows "system packager" than anything
> else. Whether that means bdist_msi can't be changed to work with a
> package index rather than (or as well as, I don't care) add/remove, I
> don't know.

If the package database is merely a directory with additional files
in it, one file per package, then most likely both bdist_wininst
and bdist_msi could support that.

If the file needs to contain file names specific to the target system,
then supporting it in bdist_msi is a bit tricky, as one would have
to generate the file at installation time. That's a "custom action";
I'd probably generate a VB script at packaging time which is then run
at installation time to edit the package database.

> The "one way to do it" uninstaller should be able to uninstall
> everything. If it needs to call the system uninstaller for a specific
> package, there's nothing wrong with doing that. But why tell me to run
> a different command? Just do it for me. I only want one UI, the rest
> is implementation detail.

The uninstallation procedure of the system installer probably has
a separate UI which can't really be suppressed.

For example, uninstallation may be rejected as additional applications
rely on the package, or uninstallation could cause automatic removal
of prerequisite packages that are then no longer used, all requiring
user confirmation.

Also, on some systems, it's difficult to know what specific tool
to run to interact with the system packaging. On some systems,
you have a choice of multiple command-line, text mode, and GUI
tools, some of which may not be installed, or may fail to work
(e.g. when you don't have the windowing system running, and the
tool is a windowed one), or may not be the user's preference.

Regards,
Martin


From martin at v.loewis.de  Sat Mar 22 17:10:24 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 17:10:24 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080322154449.BDA423A40A5@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<20080321224110.657C63A40A5@sparrow.telecommunity.com>	<47E46176.4090205@v.loewis.de>	<20080322020447.3B1033A4074@sparrow.telecommunity.com>	<47E4EE9D.2090605@v.loewis.de>	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>	<47E525F5.40705@v.loewis.de>
	<20080322154449.BDA423A40A5@sparrow.telecommunity.com>
Message-ID: <47E52F70.1010806@v.loewis.de>

>> Sure, but what is precisely the semantics of uninstallation, in
>> terms of changes to the system state?
>>
>> I think any model where uninstallation is merely the removal
>> of files is too limited to be practical.
> 
> The distutils only support the *addition* of files, so I'm not sure 
> how only removing files is a limit here.  Could you explain?

For files, yes, it only supports addition. But it supports
arbitrary other actions, such as:
- addition of registry keys
- addition of user accounts
- creation of database tables in a relational database
- updating the shared library loader path
- creation and start of a system service
- integration of documentation into info
- registration of DTDs with the system catalog
- ...

It's turing-complete, and it has full interface to the operating
system, so installation of a distutils package can do *much*
more than merely installing files.

Uninstallation needs to revert anything installation did,
so it is often more than mere removal of files.

HTH,
Martin

From martin at v.loewis.de  Sat Mar 22 17:30:45 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 17:30:45 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>
	<47E4DE8E.6010809@holdenweb.com>
	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
Message-ID: <47E53435.4090104@v.loewis.de>

> Oh, and application installation is (should be) completely different.
> On Windows, applications should probably be bundled with their own
> Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
> standard is, so I'd have to defer to others.

This I disagree with. I think it's an overall bad thing to have all
kinds of applications ship their own copy of Python; see also Aza
Raskin's PyCon keynote.

On Linux, python typically comes with the system pre-installed;
it is not even an option not to have it, except for minimalist
installations. So if you write python scripts, you typically
expect that #!/usr/bin/env python works; you might put python2.5
there if you don't trust that system one is "good enough".

For installing the application, you typically want the choice
between a "system installation" (in /usr/bin, or perhaps
/usr/local/bin), and a "user installation". As distutils supports
both cases, it works fairly well for applications as well.

Regards,
Martin




From floris.bruynooghe at gmail.com  Sat Mar 22 18:13:12 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 22 Mar 2008 17:13:12 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <47E528EC.7010605@v.loewis.de>
References: <20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<20080322020447.3B1033A4074@sparrow.telecommunity.com>
	<47E4EE9D.2090605@v.loewis.de>
	<20080322132418.GB8513@laurie.devork>
	<47E5142D.7010902@v.loewis.de>
	<20080322152150.GA16277@laurie.devork>
	<47E528EC.7010605@v.loewis.de>
Message-ID: <20080322171312.GA19074@laurie.devork>

On Sat, Mar 22, 2008 at 04:42:36PM +0100, "Martin v. L?wis" wrote:
>> I speak for Debian, so for Debian: yes.  The setup.py would have to be
>> pretty bad for a packager to not use it.  There is no reason to
>> re-write upstream's installation procedure as you would have to figure
>> out which files need to be installed where and this would create many
>> bugs.
>>
>> The canonical way is something like this:
>>
>>   $ pythonX.Y setup.py --root=$(pwd)/debian/tmp
>>   $ # Fixup anything done wrong/badly (for debian) by setup.py
>>   $ # Make a tarball of $(pwd)/debian/tmp
>>
>> In reality it's slightly more complicated of course.
>
> More than slightly so, with pycentral, pysupport, and all that.
>
> I still doubt that the database would show up in a directory on
> sys.path.

If not it would be a bug in pycentral/pysupport.  Only two bugs to
file, not that bad I think!

>> At [1] there are
>> many packages, paste and parallelpython are good examples if you're
>> interested (look in the debian/rules file).
>
> I started with nevow. It uses cdbs, so its debian/rules is nearly
> empty, and does not include a call to setup.py at all.
>
> Instead, the distutils support is in
>
> /usr/share/cdbs/1/class/python-distutils.mk
[...]

Again, that would be a bug in CDBS.

For the specific snippet you showed, yes that does essentially
"pythonX.Y setup.py --root=$some_alternate_root" as I said above.  As
an aside I also happen to be in the camp that dislikes CDBS...


The specifics and complications don't matter for this discussion I
think.  If setup.py installs the correct file into the installdb then
it will work in almost all cases, Neal Becker confirmed this is true
for Fedora as well.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From lxander.m at gmail.com  Sat Mar 22 19:52:31 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Sat, 22 Mar 2008 14:52:31 -0400
Subject: [Distutils] metadata/versions
In-Reply-To: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>
References: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>
Message-ID: <525f23e80803221152p21597de3sf54b8a306d7f43e9@mail.gmail.com>

On Fri, Mar 21, 2008 at 3:57 PM, Chris Withers <chris at simplistix.co.uk> wrote:
>  Apologies if these questions have been answered elsewhere, if they have,
>  please point me at the answers. I've been trying to follow the discussion
>  but the shear volume has overwhelmed me...
>
>  1. Is there a canonical way to tell what version a python package thinks
>  it is?

Not in practice.

>  2. Does the python package index contain dependency information? If not,
>  why not? ;-)

Not really.

I am far from the authority on these topics, but my understanding as a
user is that PEPs 241,314,345 attempt to address these issues but are
not used in practice. I do not know how to query an installed package
to determine its version in canonical way (lot's of projects use
__version__, but it isn't required and isn't guaranteed to agree with
what distutils thinks is the package version number -- perhaps even
worse you have to import the module to get at it). While projects
could provide (python package) dependency information with PEP 314, I
don't know which ones do, if any. All of this could be from a lack of
education on my part, but my guess is that if it were widely used and
available it wouldn't be such an issue and I would know more about it
since I tend to read the "official" tutorials and manuals.

From p.f.moore at gmail.com  Sat Mar 22 20:56:45 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 22 Mar 2008 19:56:45 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E53435.4090104@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com> <47E4DE8E.6010809@holdenweb.com>
	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
	<47E53435.4090104@v.loewis.de>
Message-ID: <79990c6b0803221256jf4e6ea1qce463ab711dcdb51@mail.gmail.com>

On 22/03/2008, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > Oh, and application installation is (should be) completely different.
>  > On Windows, applications should probably be bundled with their own
>  > Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
>  > standard is, so I'd have to defer to others.
>
>
> This I disagree with. I think it's an overall bad thing to have all
>  kinds of applications ship their own copy of Python; see also Aza
>  Raskin's PyCon keynote.

Is this on Windows? It's fairly common practice. Can you give me a
pointer to Aza Raskin's keynote? Is it online anywhere? I'd be
interested in his point of view.

Paul.

From martin at v.loewis.de  Sat Mar 22 21:13:24 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 22 Mar 2008 21:13:24 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <79990c6b0803221256jf4e6ea1qce463ab711dcdb51@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	
	<47E40726.1090209@egenix.com>	
	<20080321212127.791B63A4074@sparrow.telecommunity.com>	
	<47E4330C.5070601@egenix.com> <47E4DE8E.6010809@holdenweb.com>	
	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>	
	<47E53435.4090104@v.loewis.de>
	<79990c6b0803221256jf4e6ea1qce463ab711dcdb51@mail.gmail.com>
Message-ID: <47E56864.4080201@v.loewis.de>

>>> Oh, and application installation is (should be) completely different.
>>  > On Windows, applications should probably be bundled with their own
>>  > Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
>>  > standard is, so I'd have to defer to others.
>>
>>
>> This I disagree with. I think it's an overall bad thing to have all
>>  kinds of applications ship their own copy of Python; see also Aza
>>  Raskin's PyCon keynote.
> 
> Is this on Windows? It's fairly common practice.

Unfortunately so, yes. This can be viewed a burden to the adoption
of Python: for a small application, you get this huge download to
bundle.

> Can you give me a
> pointer to Aza Raskin's keynote? Is it online anywhere? I'd be
> interested in his point of view.

Unfortunately no. I was looking for it, but couldn't find it. He
mentioned a website with a "call for action", but I couldn't find
that, either :-(

Regards,
Martin

From jinty at web.de  Sun Mar 23 15:57:15 2008
From: jinty at web.de (Brian Sutherland)
Date: Sun, 23 Mar 2008 15:57:15 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <47E46176.4090205@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
Message-ID: <20080323145715.GD900@mobilista.local>

On Sat, Mar 22, 2008 at 02:31:34AM +0100, "Martin v. L?wis" wrote:
> > Tools which will need this data, in order to do their work.  Hence, 
> > the reason for standardizing the data, instead of the tool(s).
> 
> If there was a chance that the infrastructure being developed
> actually helps these tools, *that* would be a reasonable goal,
> IMO.
> 
> However, I'm extremely skeptical that this can ever succeed
> to the degree that whoever provides RPMs, .debs, or MSI
> files will actually use such data, as they will find that
> the data are incomplete, and they have to redo all of it,
> anyway.

I've found it extremely useful to have access to dependency information
when making Debian packages automagically out of setuptools tarballs.

It's not easy or robust to access/use it, but for my simple pure python
packages, it's been working wonderfully. 

-- 
Brian Sutherland

From martin at v.loewis.de  Wed Mar 19 14:34:17 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 19 Mar 2008 08:34:17 -0500
Subject: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for
	Python 3000?
In-Reply-To: <47E0D883.9030402@taupro.com>
References: <47E0D883.9030402@taupro.com>
Message-ID: <47E11659.9000307@v.loewis.de>

> 1. What is the plan for PyPI when Python 3.0 comes out and
>     dependencies start getting satisfied from distribution
>     across the great divide, e.g. a 3.0-specific package
>     pulls from PyPI a 2.x-specific package to meet some
>     need?  Are there plans to fork PyPI, apply special
>     tags to uploads or what?

I don't see the need to for PyPI. For packages (or "distributions",
to avoid confusion with Python packages), I see two options:

a) provide a single release that supports both 2.x and 3.x.
    The precise strategy to do so might vary. If one is going
    for a single source version, have setup.py run 2to3
    (or perhaps 3to2). For dual-source packages, have setup.py
    just install the one for the right version; setup.py itself
    needs to be written so it runs on both versions (which is
    easy to do).
b) switch to Python 3 at some point (i.e. burn your bridges).

You seem to be implying that some projects may release separate
source distributions. I cannot imagine why somebody would want
to do that.

> 2. There have been attempts over the years to fix distutils,
>     with the last one being in 2006 by Anthony Baxter. He
>     stated that a major hurdle was the strong demand to
>     respect backward compatibility and he finally gave up.

Can you kindly refer to some archived discussion for that?

>     One of the purposes of Python 3.0 was the freedom to
>     break backward compatibility for the sake of "doing
>     the right thing".  So is it now permissible to give
>     distutils a good reworking and stop letting
>     compatibility issues hold us back?

I don't know what the proposed changes are, but for some
changes; in general, I feel that the need for backwards
compatibility is exaggerated.

Regards,
Martin

From lxander.m at gmail.com  Mon Mar 24 21:57:44 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Mon, 24 Mar 2008 16:57:44 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E51179.5040601@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
Message-ID: <525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>

On Sat, Mar 22, 2008 at 10:02 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > It seems to me that this discussion is being undermined by not
>  > acknowledging the many use cases up front. There is no rationale
>  > because there are too many tacit rationales.
>
>  I honestly, really, cannot imagine what those are. Explicit is
>  better than implicit.

Exactly. I was attempting to suggest that we be explicit about "the
problem" being solved so it was easier for everyone interested to
verify that their "needs" (i.e. use case) will be met and that the
proposed solution solves the problem. Since the topic of the thread is
how to get rid of eggs in 2.6, the question (to me, at least) is how
to enable the functionality provided by setuptools to be provided
outside the standard library in a way that does not create pressure
for the entire python community to be assimilated into egg-ified
zombies.

Why do we have the current pressure for egg-ification? The main
pressure appears to arise from a strong desire by the python community
for a simple installation tool like CPAN that downloads and installs
the package of interest *as well as* any dependencies you don't
already have installed. Of course, it is the desire for the
dependencies to be discovered, downloaded, and installed in a manner
that honors the current state of your python environment that creates
all of the problems.

My participation may be unwanted or unwarranted due to my lack of
education/understanding, but I like to live dangerously in the hopes
of actually being helpful. With that preamble, here's my attempt at an
explicit rationale for a database of installed packages (A.K.A. The
New PEP 262):

"""
Rationale
=========
It is often necessary during the course of managing a python
installation for an administrator to determine the following things:

1. If the current installation state satisfies the requirements of a
new package being considered for installation.
2. If it is safe for the administrator to upgrade or remove a package,
and if so, how (e.g. use a system-level package management tool).
3. What files to remove in order to uninstall the package (if a
system-level package management tool was not used to install it).
4. If the current installation was modified in-place or custom
configured in an way, so that such changes can be noted before
upgrading.

Furthermore, many administrators want to do as much of this as
possible with automated tools, without manually inspecting README
files, INSTALL files, or the installed code itself to determine the
list of dependencies and installed versions, so there is a desire to
be able to make the above determinations programmatically.

Current efforts to provide these capabilities without standard library
support have resulted in many users being forced to use non-standard
package management tools because other users desired these
capabilities. This proposal is motivated by a desire to provide the
minimum required infrastructure so that both segments of the python
community can peacefully coexist without getting in each others way
(i.e. the ability to "opt in" to python-based non-system-level package
managers).

Proposal
========
The proposal is to provide in the standard library the following capabilities:

1. List the installed packages, along with the version and dependency
list for each package.
2. Query the ownership of a currently installed package (standard
library, system-level package management tool, etc.).
3. List the files installed be specific package.
4. Recall the original message digest for each installed file to
determine if the file has been modified.

...

Alternatives
============
There are at least three alternatives to providing a database of
installed packages that could also potentially enable administrators
to accomplish the same ultimate goal of installing new packages and
their dependencies without breaking or interfering with the "system"
installation.

Switchable Environments
-----------------------
Eliminate the need to be careful by providing a mechanism for the
creation of and installation into what amounts to user-selectable
site-packages directories that each "inherit" from what is currently
called site-packages (something like an eggless virtualenv with a
python comannd-line option to choose the environment). Need to upgrade
a package provided by the standard library or your system-level
package manager? No Problem! Of course, this won't help you manage
your sand-boxed environment, but you can have lots of them for very
little cost (it's not a whole new python installation) so just create
a new one each time you want to change something.

Ubiquitous System Packages
--------------------------
Eliminate the need to *not* use your OS's system-level package manager
by creating the necessary infrastructure so that python packages can
easily be distributed in the major system-level packager formats (RPM,
DEB, MSI, MPKG, etc.) for all publicly released python packages. A
windows-based developer who releases their pure-python package should
be able to create RPMs, DEBs, etc. automatically (using distutils, for
instance) without access to a computer running the appropriate OS
(python setup.py bdist_all upload).

Remote Hosted Installation Database
-----------------------------------
Eliminate the requirement that your installation be self-describing by
maintaining the information in PyPI necessary for one to figure out
what versions of what things are installed in your system (either by
recipes or a table of message digest values for the directories) in
addition the dependencies and list of installed files for each version
of each package thats ever been released.

"""

Regards,
Alex

From tseaver at palladion.com  Mon Mar 24 23:12:59 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 24 Mar 2008 18:12:59 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <47E52F70.1010806@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<20080321224110.657C63A40A5@sparrow.telecommunity.com>	<47E46176.4090205@v.loewis.de>	<20080322020447.3B1033A4074@sparrow.telecommunity.com>	<47E4EE9D.2090605@v.loewis.de>	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>	<47E525F5.40705@v.loewis.de>	<20080322154449.BDA423A40A5@sparrow.telecommunity.com>
	<47E52F70.1010806@v.loewis.de>
Message-ID: <47E8276B.8030800@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin v. L?wis wrote:
>>> Sure, but what is precisely the semantics of uninstallation, in
>>> terms of changes to the system state?
>>>
>>> I think any model where uninstallation is merely the removal
>>> of files is too limited to be practical.
>> The distutils only support the *addition* of files, so I'm not sure 
>> how only removing files is a limit here.  Could you explain?
> 
> For files, yes, it only supports addition. But it supports
> arbitrary other actions, such as:
> - addition of registry keys
> - addition of user accounts
> - creation of database tables in a relational database
> - updating the shared library loader path
> - creation and start of a system service
> - integration of documentation into info
> - registration of DTDs with the system catalog
> - ...
> 
> It's turing-complete, and it has full interface to the operating
> system, so installation of a distutils package can do *much*
> more than merely installing files.

Which is exactly what is wrong with distutils:  turing completeness in
an installer is an *anti* goal, from the perspective of manageability.
I'd be willing to bet that if you asked system packagers to list the
dozen or so packages which they *hate* to maintain, the large majority
of each list would be packages which acutally use the full power of
distutils.

(Note:  I'm aware that people believe it to be necessary to munge the
Windows registry when installing Python packages;  I just don't agree
with the practice, and don't think we should distort Python's process to
coddle it).

> Uninstallation needs to revert anything installation did,
> so it is often more than mere removal of files.

Practically speacking, nobody but the author of the TC-abusing setup.py
is *ever* going to be able to do this, and most of them won't get the
edge cases right.  Maybe we can just punt on such packages, and make a
tool which actually works for the huge majority of distributions which
don't need to do more than install files.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFH6Cdr+gerLs4ltQ4RAlFzAJi3gs8fzb9o8/Dtct1G9P0EJxNSAKCc7V7m
uT5MgTzltBDhdrgoNxt8nA==
=zgqI
-----END PGP SIGNATURE-----


From tseaver at palladion.com  Mon Mar 24 23:26:42 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 24 Mar 2008 18:26:42 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <47E53435.4090104@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<47E4DE8E.6010809@holdenweb.com>	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
	<47E53435.4090104@v.loewis.de>
Message-ID: <47E82AA2.1020502@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin v. L?wis wrote:

>> Oh, and application installation is (should be) completely different.
>> On Windows, applications should probably be bundled with their own
>> Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
>> standard is, so I'd have to defer to others.
> 
> This I disagree with. I think it's an overall bad thing to have all
> kinds of applications ship their own copy of Python; see also Aza
> Raskin's PyCon keynote.
> 
> On Linux, python typically comes with the system pre-installed;
> it is not even an option not to have it, except for minimalist
> installations. So if you write python scripts, you typically
> expect that #!/usr/bin/env python works; you might put python2.5
> there if you don't trust that system one is "good enough".
> 
> For installing the application, you typically want the choice
> betaween a "system installation" (in /usr/bin, or perhaps
> /usr/local/bin), and a "user installation". As distutils supports
> both cases, it works fairly well for applications as well.

Sharing the system python is hugely problematic on a unix box which
actually *uses* python for its own tools:  the application is not "safe"
from additions / updates / removeals of the packages in
/usr/lib/python2.x/site-packages done to support those system tools.
The problem gets worse as multiple non-system applications install files
there:  e.g., the 'twisted' package on Debian boxes depends on an
ancient version of 'zope.interface', which can't be used with any
currently supported version of Zope.

virtualenv makes using the system python on such systems somewhat more
tolerable, because each virtualenv can be isolated from the
site-packages of the "host" environment (and therefore from the other
applications).  You still have to live with the choices the system
packagers make (UCS4, for instance), but the pain is at least tolerable.

For a long-running Python application (as opposed to desktop tools or
scripts), installing a custom Python is the "safest" choice:  it
insulates the application not only from unexpected system-driven
site-packages changes, but also allows greater control over other Python
configuration choices (the UCS2 / UCS4 knob, etc.).



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH6Cqi+gerLs4ltQ4RAlqZAKCxr2lraLSycVsksYAevtf+urALOgCeLzs9
fE2g7IAb+22B+UbSUFPqj4w=
=re0h
-----END PGP SIGNATURE-----

From tseaver at palladion.com  Mon Mar 24 23:26:42 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 24 Mar 2008 18:26:42 -0400
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <47E53435.4090104@v.loewis.de>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<47E4DE8E.6010809@holdenweb.com>	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
	<47E53435.4090104@v.loewis.de>
Message-ID: <47E82AA2.1020502@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin v. L?wis wrote:

>> Oh, and application installation is (should be) completely different.
>> On Windows, applications should probably be bundled with their own
>> Python interpreter, a la py2exe. On Unix/Linux, I don't know what the
>> standard is, so I'd have to defer to others.
> 
> This I disagree with. I think it's an overall bad thing to have all
> kinds of applications ship their own copy of Python; see also Aza
> Raskin's PyCon keynote.
> 
> On Linux, python typically comes with the system pre-installed;
> it is not even an option not to have it, except for minimalist
> installations. So if you write python scripts, you typically
> expect that #!/usr/bin/env python works; you might put python2.5
> there if you don't trust that system one is "good enough".
> 
> For installing the application, you typically want the choice
> betaween a "system installation" (in /usr/bin, or perhaps
> /usr/local/bin), and a "user installation". As distutils supports
> both cases, it works fairly well for applications as well.

Sharing the system python is hugely problematic on a unix box which
actually *uses* python for its own tools:  the application is not "safe"
from additions / updates / removeals of the packages in
/usr/lib/python2.x/site-packages done to support those system tools.
The problem gets worse as multiple non-system applications install files
there:  e.g., the 'twisted' package on Debian boxes depends on an
ancient version of 'zope.interface', which can't be used with any
currently supported version of Zope.

virtualenv makes using the system python on such systems somewhat more
tolerable, because each virtualenv can be isolated from the
site-packages of the "host" environment (and therefore from the other
applications).  You still have to live with the choices the system
packagers make (UCS4, for instance), but the pain is at least tolerable.

For a long-running Python application (as opposed to desktop tools or
scripts), installing a custom Python is the "safest" choice:  it
insulates the application not only from unexpected system-driven
site-packages changes, but also allows greater control over other Python
configuration choices (the UCS2 / UCS4 knob, etc.).



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH6Cqi+gerLs4ltQ4RAlqZAKCxr2lraLSycVsksYAevtf+urALOgCeLzs9
fE2g7IAb+22B+UbSUFPqj4w=
=re0h
-----END PGP SIGNATURE-----


From floris.bruynooghe at gmail.com  Tue Mar 25 01:33:10 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Tue, 25 Mar 2008 00:33:10 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
	<525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
Message-ID: <20080325003310.GA27433@laurie.devork>

On Mon, Mar 24, 2008 at 04:57:44PM -0400, Alexander Michael wrote:
> With that preamble, here's my attempt at an
> explicit rationale for a database of installed packages (A.K.A. The
> New PEP 262):

Nice effort, thanks.

> Rationale
> =========
> It is often necessary during the course of managing a python
> installation for an administrator to determine the following things:
> 
> 1. If the current installation state satisfies the requirements of a
> new package being considered for installation.
> 2. If it is safe for the administrator to upgrade or remove a package,
> and if so, how (e.g. use a system-level package management tool).
> 3. What files to remove in order to uninstall the package (if a
> system-level package management tool was not used to install it).
> 4. If the current installation was modified in-place or custom
> configured in an way, so that such changes can be noted before
> upgrading.

I agree with 1 and 2, but 3 and 4 are optional I think.  They are not
important for the different tools to interact AIUI.  As long as
everyone only removes files they own all is fine.  By making the
database extensible (e.g. X- headers in the RFC822 case) each tool
could use their own extensions for recording these specifics as they
need them, eliminating required information being spread around the
system.

Not that I think it would do much harm if they where specified by the
PEP, but then I never tried implementing one of these tools so I might
not have the best opinion on how much specifying this might
unnecessarily restrict implementations.


Something that this requirement should probably explicitly describe:
does this database only concern modules, i.e. installed in a directory
that's on sys.path?  My understanding is that it does.  However I'm
unsure as to how that can be combined with supporting applications
that require certain modules.  Would every tool need to keep track of
the apps it installed separately to make sure that, when it's used to
uninstall a module, it doesn't render an app that it installed
useless?

To tie in another part of this discussion, this could mean that module
installation is forced to be a simple list of files while still
allowing a Turing complete installer for applications.

> Alternatives
> ============
[...]
> Ubiquitous System Packages
> --------------------------
> Eliminate the need to *not* use your OS's system-level package manager
> by creating the necessary infrastructure so that python packages can
> easily be distributed in the major system-level packager formats (RPM,
> DEB, MSI, MPKG, etc.) for all publicly released python packages. A
> windows-based developer who releases their pure-python package should
> be able to create RPMs, DEBs, etc. automatically (using distutils, for
> instance) without access to a computer running the appropriate OS
> (python setup.py bdist_all upload).

This is a bad or even impossible approach.  It is very hard or even
impossible to create a system package on a different system.  It is
even harder to keep the releases of *all* python packaging tools in
sync with the current policy of a system.  It is not a surprise that
most of past efforts to create a "bdist_deb" target for setup.py have
failed to gain much traction.

A system might change policies halfway their release cycle and can't
afford the wait for the python world to catch up with that policy
change first.  This gets even worse when the sytem can't really agree
on a policy (which does unfortunately happen).  Systems might also
have different policies for their different versions (e.g. old-stable,
stable and in-development) only making it more impossible for everyone
else to try and play catch-up with the system as they would have to
supply packages for each supported release.

I think it's best to allow systems to create their own packages by
giving them a dependable "setup.py install" command --which most
systems use in their packaging scripts anyway-- to build on.


> Remote Hosted Installation Database
> -----------------------------------
> Eliminate the requirement that your installation be self-describing by
> maintaining the information in PyPI necessary for one to figure out
> what versions of what things are installed in your system (either by
> recipes or a table of message digest values for the directories) in
> addition the dependencies and list of installed files for each version
> of each package thats ever been released.

The easiest way to distribute some python stuff is to quickly write a
setup.py, even if it's only to share some quick and dirty module with
a handful of people.  Not everything will want or should be in PyPI.
Some machines might also not have net access.  So I'm dubious about
this option (it also seems technically quite hard).



Thanks for writing this up, I think it's very good and useful.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From pje at telecommunity.com  Tue Mar 25 03:35:01 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 24 Mar 2008 22:35:01 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
 beyond
In-Reply-To: <20080325003310.GA27433@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
	<525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
	<20080325003310.GA27433@laurie.devork>
Message-ID: <20080325023518.567C13A4062@sparrow.telecommunity.com>

At 12:33 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
>On Mon, Mar 24, 2008 at 04:57:44PM -0400, Alexander Michael wrote:
> > With that preamble, here's my attempt at an
> > explicit rationale for a database of installed packages (A.K.A. The
> > New PEP 262):
>
>Nice effort, thanks.
>
> > Rationale
> > =========
> > It is often necessary during the course of managing a python
> > installation for an administrator to determine the following things:
> >
> > 1. If the current installation state satisfies the requirements of a
> > new package being considered for installation.
> > 2. If it is safe for the administrator to upgrade or remove a package,
> > and if so, how (e.g. use a system-level package management tool).
> > 3. What files to remove in order to uninstall the package (if a
> > system-level package management tool was not used to install it).
> > 4. If the current installation was modified in-place or custom
> > configured in an way, so that such changes can be noted before
> > upgrading.
>
>I agree with 1 and 2, but 3 and 4 are optional I think.  They are not
>important for the different tools to interact AIUI.  As long as
>everyone only removes files they own all is fine.

Not quite - it's necessary to know which files are owned by a given 
package or tool in order to do this.  If you are installing a 
package, you need to know who owns any files that you're about to overwrite.


>Something that this requirement should probably explicitly describe:
>does this database only concern modules, i.e. installed in a directory
>that's on sys.path?

It will probably need to include scripts, data, etc.


>   My understanding is that it does.  However I'm
>unsure as to how that can be combined with supporting applications
>that require certain modules.  Would every tool need to keep track of
>the apps it installed separately to make sure that, when it's used to
>uninstall a module, it doesn't render an app that it installed
>useless?
>
>To tie in another part of this discussion, this could mean that module
>installation is forced to be a simple list of files while still
>allowing a Turing complete installer for applications.

These are currently open issues.


From mhammond at skippinet.com.au  Tue Mar 25 05:15:35 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Tue, 25 Mar 2008 15:15:35 +1100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E8276B.8030800@palladion.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<20080321224110.657C63A40A5@sparrow.telecommunity.com>	<47E46176.4090205@v.loewis.de>	<20080322020447.3B1033A4074@sparrow.telecommunity.com>	<47E4EE9D.2090605@v.loewis.de>	<20080322151918.D9EBC3A40B1@sparrow.telecommunity.com>	<47E525F5.40705@v.loewis.de>	<20080322154449.BDA423A40A5@sparrow.telecommunity.com>	<47E52F70.1010806@v.loewis.de>
	<47E8276B.8030800@palladion.com>
Message-ID: <014c01c88e2e$e4aad5a0$ae0080e0$@com.au>

> (Note:  I'm aware that people believe it to be necessary to munge the
> Windows registry when installing Python packages;  I just don't agree
> with the practice, and don't think we should distort Python's process
> to coddle it).

Whoever thinks it necessary is misguided.  Installing a package doesn't
require munging the registry and none of the popular installation techniques
do.  Installing Python itself does, and some packages have special
requirements (eg, pywin32 registering COM objects), but in general, Python
packages on Windows can ignore the registry.

Mark


From floris.bruynooghe at gmail.com  Tue Mar 25 10:16:33 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Tue, 25 Mar 2008 09:16:33 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6
	and	beyond
In-Reply-To: <20080325023518.567C13A4062@sparrow.telecommunity.com>
References: <47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
	<525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
	<20080325003310.GA27433@laurie.devork>
	<20080325023518.567C13A4062@sparrow.telecommunity.com>
Message-ID: <20080325091633.GA10192@laurie.devork>

On Mon, Mar 24, 2008 at 10:35:01PM -0400, Phillip J. Eby wrote:
> At 12:33 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
>> > 1. If the current installation state satisfies the requirements of a
>> > new package being considered for installation.
>> > 2. If it is safe for the administrator to upgrade or remove a package,
>> > and if so, how (e.g. use a system-level package management tool).
>> > 3. What files to remove in order to uninstall the package (if a
>> > system-level package management tool was not used to install it).
>> > 4. If the current installation was modified in-place or custom
>> > configured in an way, so that such changes can be noted before
>> > upgrading.
>>
>> I agree with 1 and 2, but 3 and 4 are optional I think.  They are not
>> important for the different tools to interact AIUI.  As long as
>> everyone only removes files they own all is fine.
>
> Not quite - it's necessary to know which files are owned by a given  
> package or tool in order to do this.  If you are installing a package, 
> you need to know who owns any files that you're about to overwrite.

What would the motivation be for requiring to overwrite files created
by another tool?  That seems rather dangerous to me.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From lxander.m at gmail.com  Tue Mar 25 13:24:00 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Tue, 25 Mar 2008 08:24:00 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080325091633.GA10192@laurie.devork>
References: <47E40726.1090209@egenix.com> <47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
	<525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
	<20080325003310.GA27433@laurie.devork>
	<20080325023518.567C13A4062@sparrow.telecommunity.com>
	<20080325091633.GA10192@laurie.devork>
Message-ID: <525f23e80803250524s7b7c3e55x4a159442ea745f91@mail.gmail.com>

On Tue, Mar 25, 2008 at 5:16 AM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
> On Mon, Mar 24, 2008 at 10:35:01PM -0400, Phillip J. Eby wrote:
>  > At 12:33 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
> >> > 1. If the current installation state satisfies the requirements of a
>  >> > new package being considered for installation.
>  >> > 2. If it is safe for the administrator to upgrade or remove a package,
>  >> > and if so, how (e.g. use a system-level package management tool).
>  >> > 3. What files to remove in order to uninstall the package (if a
>  >> > system-level package management tool was not used to install it).
>  >> > 4. If the current installation was modified in-place or custom
>  >> > configured in an way, so that such changes can be noted before
>  >> > upgrading.
>  >>
>  >> I agree with 1 and 2, but 3 and 4 are optional I think.  They are not
>  >> important for the different tools to interact AIUI.  As long as
>  >> everyone only removes files they own all is fine.
>  >
>  > Not quite - it's necessary to know which files are owned by a given
>  > package or tool in order to do this.  If you are installing a package,
>  > you need to know who owns any files that you're about to overwrite.
>
>  What would the motivation be for requiring to overwrite files created
>  by another tool?  That seems rather dangerous to me.

As different tools typically shouldn't be allowed to clobber each
other, it seems both reasonable and a nice simplification for the
proposed db not to require registering of installed files by all
package managers/installers if an installer can choose to record its
own lists in the db using a simple extension mechanism when desired.
Nevertheless, it did seem nice to be able to ask the install db if the
current state matches what is on disk by verifying the file lists and
checksums against the file system, but this might be taking us too far
into the abyss of recreating a system-level package manager.But an
additional benefit is that requiring the file lists makes it easier to
change tools by ignoring or rewriting the "ownership" information
which I would tend to view as purely advisory in nature to begin with
(i.e. no enforcement). Of course, this could be accommodated by
establishing "standard" but optional auxiliary fields in the db that
all python oriented tools would likely use.

From chris at simplistix.co.uk  Tue Mar 25 15:14:04 2008
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 25 Mar 2008 14:14:04 +0000
Subject: [Distutils] metadata/versions
In-Reply-To: <525f23e80803221152p21597de3sf54b8a306d7f43e9@mail.gmail.com>
References: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>
	<525f23e80803221152p21597de3sf54b8a306d7f43e9@mail.gmail.com>
Message-ID: <47E908AC.7030709@simplistix.co.uk>

Alexander Michael wrote:
> I am far from the authority on these topics, but my understanding as a
> user is that PEPs 241,314,345 

So PEP 345 is implemented in Python 2.5?

> attempt to address these issues but are
> not used in practice. 

How come? Does distutils in 2.5 or setuptools not generate the required 
PKG-INFO?

cheers,

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk

From oliver.schoenborn at utoronto.ca  Tue Mar 25 15:39:31 2008
From: oliver.schoenborn at utoronto.ca (Oliver Schoenborn)
Date: Tue, 25 Mar 2008 10:39:31 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
Message-ID: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>

I have a problem with "setup.py develop" installation.

I have the following folder tree:

pubsub
    \- setup.py
    \- extern
       \- module0.py
    \- core
       \- __init__.py
       \- module1.py
       \- module2.py
       \- ...

I want my setup.py to install modules from pubsub/extern (such as  
module0.py) into site-packages, and to install the 'core' package in  
site-packages as pubsubcore:

setup(
     ...
     py_modules   = ['module0'],
     packages     = ['pubsubcore'],
     package_dir  = {'': 'extern', 'pubsubcore':'core'},
     scripts      = [],
     ...
)

For sdist, bdist, bdist_egg etc, it works: the output shows module0 is  
in the "root" of the distribution, together with a package named  
"pubsubcore" which contains what is in 'pubsub/core'.

HOWEVER, and here is the problem: when I do "setup.py develop" (this  
is on WinXP with Python 2.4.4 and the latest stable setuptools), only  
the contents of 'extern' get 'installed':

C:\Documents and Settings\schoenb\My Documents\pubsub>setup.py develop
running develop
running egg_info
writing external\PyPubSub.egg-info\PKG-INFO
writing top-level names to external\PyPubSub.egg-info\top_level.txt
writing dependency_links to external\PyPubSub.egg-info\dependency_links.txt
writing manifest file 'external\PyPubSub.egg-info\SOURCES.txt'
running build_ext
Creating c:\python24\lib\site-packages\PyPubSub.egg-link (link to external)
Adding PyPubSub 3.0a6 to easy-install.pth file
Installed c:\documents and settings\schoenb\my documents\pubsub\extern
Processing dependencies for PyPubSub==3.0a6

The above allows me, as the developer, to do "import module0" but not  
"import pubsubcore". Indeed, looking at  
\python24\lib\site-packages\easy-install.pth, there is only one new  
line added, pointing to c:\documents and settings\schoenb\my  
documents\pubsub\extern. There should be a second line added for the  
package, ie

c:\documents and settings\schoenb\my documents\pubsub

so that I (developer) can do "import pubsubcore", just as any user  
would be able to if I had done a full install. When I add the above  
line manually to the .pth file the "developer" installation works as  
expected.

What am I doing wrong? Thanks,

Oliver



From lxander.m at gmail.com  Tue Mar 25 15:54:59 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Tue, 25 Mar 2008 10:54:59 -0400
Subject: [Distutils] metadata/versions
In-Reply-To: <47E908AC.7030709@simplistix.co.uk>
References: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>
	<525f23e80803221152p21597de3sf54b8a306d7f43e9@mail.gmail.com>
	<47E908AC.7030709@simplistix.co.uk>
Message-ID: <525f23e80803250754n2dc593el741241a5abedfeb@mail.gmail.com>

On Tue, Mar 25, 2008 at 10:14 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Alexander Michael wrote:
>  > I am far from the authority on these topics, but my understanding as a
>  > user is that PEPs 241,314,345
>
>  So PEP 345 is implemented in Python 2.5?

Yes.

>  > attempt to address these issues but are
>  > not used in practice.
>
>  How come? Does distutils in 2.5 or setuptools not generate the required
>  PKG-INFO?

IIUC, as far as distutils goes, the metadata shows-up in the sdist,
but not in site-packages, so it isn't really useful for introspecting
an installation. The ability to specify requirements is virtually
undocumented and isn't used in practice to my knowledge. Setuptools
makes both pieces of information introspectable for packages it
installs (but provides its own way of specifying requirements).

From pje at telecommunity.com  Tue Mar 25 16:18:18 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 11:18:18 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
In-Reply-To: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>
References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>
Message-ID: <20080325151819.199753A40FF@sparrow.telecommunity.com>

At 10:39 AM 3/25/2008 -0400, Oliver Schoenborn wrote:
>I have a problem with "setup.py develop" installation.
>
>I have the following folder tree:
>
>pubsub
>     \- setup.py
>     \- extern
>        \- module0.py
>     \- core
>        \- __init__.py
>        \- module1.py
>        \- module2.py
>        \- ...
>
>I want my setup.py to install modules from pubsub/extern (such as
>module0.py) into site-packages, and to install the 'core' package in
>site-packages as pubsubcore:
>
>setup(
>      ...
>      py_modules   = ['module0'],
>      packages     = ['pubsubcore'],
>      package_dir  = {'': 'extern', 'pubsubcore':'core'},
>      scripts      = [],
>      ...
>)

This use of package_dir is not compatible with 'develop'.  For 
'develop' to work on a project, you may not have any non-empty keys 
in package_dir.  If you want to use develop with this project, you 
will have to alter your directory layout so that 'core' is under 
'extern', or else move the 'extern' modules out of 'extern' and up to 
the top-level directory, and update (or get rid of) package_dir accordingly.


From pje at telecommunity.com  Tue Mar 25 16:15:00 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 11:15:00 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <20080325091633.GA10192@laurie.devork>
References: <47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
	<525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
	<20080325003310.GA27433@laurie.devork>
	<20080325023518.567C13A4062@sparrow.telecommunity.com>
	<20080325091633.GA10192@laurie.devork>
Message-ID: <20080325151817.4D2053A40D8@sparrow.telecommunity.com>

At 09:16 AM 3/25/2008 +0000, Floris Bruynooghe wrote:
>What would the motivation be for requiring to overwrite files created
>by another tool?  That seems rather dangerous to me.

The point is to NOT overwrite those files.


From pje at telecommunity.com  Tue Mar 25 16:25:23 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 11:25:23 -0400
Subject: [Distutils] metadata/versions
In-Reply-To: <525f23e80803250754n2dc593el741241a5abedfeb@mail.gmail.com>
References: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>
	<525f23e80803221152p21597de3sf54b8a306d7f43e9@mail.gmail.com>
	<47E908AC.7030709@simplistix.co.uk>
	<525f23e80803250754n2dc593el741241a5abedfeb@mail.gmail.com>
Message-ID: <20080325152522.9AA853A4074@sparrow.telecommunity.com>

At 10:54 AM 3/25/2008 -0400, Alexander Michael wrote:
>On Tue, Mar 25, 2008 at 10:14 AM, Chris Withers 
><chris at simplistix.co.uk> wrote:
> > Alexander Michael wrote:
> >  > I am far from the authority on these topics, but my understanding as a
> >  > user is that PEPs 241,314,345
> >
> >  So PEP 345 is implemented in Python 2.5?
>
>Yes.
>
> >  > attempt to address these issues but are
> >  > not used in practice.
> >
> >  How come? Does distutils in 2.5 or setuptools not generate the required
> >  PKG-INFO?
>
>IIUC, as far as distutils goes, the metadata shows-up in the sdist,
>but not in site-packages, so it isn't really useful for introspecting
>an installation. The ability to specify requirements is virtually
>undocumented and isn't used in practice to my knowledge. Setuptools
>makes both pieces of information introspectable for packages it
>installs (but provides its own way of specifying requirements).

In fact, setuptools' requirements info is 100% separate from the 
PEP-specified requirements info.  Setuptools allows different 
requirements for tests, for running the setup script, and for 
optional add-ons to a project.  These requirements also use a 
different version parsing mechanism etc.

One reason for this is that Python 2.3 doesn't support this version 
stuff, and setuptools was written for 2.3 initially.  Another is that 
the PEP-defined requires/provides data does not allow dependencies to 
be satisfied merely by looking at target filenames -- i.e., without 
needing to download them.  Using a requires/provides model for 
dependency resolution would have required new PyPI infrastructure to 
be built out, whereas a filename-based approach could run on anything.

PyPI and the requires/provides PEPs also appear to carry an implicit 
assumption that these requires/provides are leveled on 
project+version, without any distinction by Python version and target 
platform.  Setuptools OTOH, assumes that specific dependencies are a 
property of a *binary* distribution, not a source 
distribution.  (i.e., the PEPs put the dependency info in a source 
package's PKG-INFO...  where it's pretty much useless for any purpose).

I've previously posted some of these comments about the 
requires/provides PEPs on Python-Dev, Distutils-SIG, and Catalog-SIG, 
with no response from anyone.


From oliver.schoenborn at utoronto.ca  Tue Mar 25 17:07:21 2008
From: oliver.schoenborn at utoronto.ca (Oliver Schoenborn)
Date: Tue, 25 Mar 2008 12:07:21 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
In-Reply-To: <20080325151819.199753A40FF@sparrow.telecommunity.com>
References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>
	<20080325151819.199753A40FF@sparrow.telecommunity.com>
Message-ID: <20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca>

Quoting "Phillip J. Eby" <pje at telecommunity.com>:

> At 10:39 AM 3/25/2008 -0400, Oliver Schoenborn wrote:
>> I have a problem with "setup.py develop" installation.
>>
>> ...
>
> This use of package_dir is not compatible with 'develop'.  For
> 'develop' to work on a project, you may not have any non-empty keys in
> package_dir.  If you want to use develop with this project, you will
> have to alter your directory layout so that 'core' is under 'extern',
> or else move the 'extern' modules out of 'extern' and up to the
> top-level directory, and update (or get rid of) package_dir accordingly.

Thanks for your quick reply. That's a pitty, would it be difficult to  
patch? E.g. from a list of all 'values' in the package_dir then add a  
line to easy-install.pth:

# pseudo code:
...
dirs = [os.path.join(os.getcwd(), dd) for key, dd
         in package_dir.iteritems() if key.find('.') < 0]
for dd in dirs:
     easyinst_pth.write(dd+'\n')
...




From pje at telecommunity.com  Tue Mar 25 17:42:35 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 12:42:35 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
In-Reply-To: <20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca>
References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>
	<20080325151819.199753A40FF@sparrow.telecommunity.com>
	<20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca>
Message-ID: <20080325164235.4C9D13A4074@sparrow.telecommunity.com>

At 12:07 PM 3/25/2008 -0400, Oliver Schoenborn wrote:
>Quoting "Phillip J. Eby" <pje at telecommunity.com>:
>
>>At 10:39 AM 3/25/2008 -0400, Oliver Schoenborn wrote:
>>>I have a problem with "setup.py develop" installation.
>>>
>>>...
>>
>>This use of package_dir is not compatible with 'develop'.  For
>>'develop' to work on a project, you may not have any non-empty keys in
>>package_dir.  If you want to use develop with this project, you will
>>have to alter your directory layout so that 'core' is under 'extern',
>>or else move the 'extern' modules out of 'extern' and up to the
>>top-level directory, and update (or get rid of) package_dir accordingly.
>
>Thanks for your quick reply. That's a pitty, would it be difficult to
>patch? E.g. from a list of all 'values' in the package_dir then add a
>line to easy-install.pth:
>
># pseudo code:
>...
>dirs = [os.path.join(os.getcwd(), dd) for key, dd
>         in package_dir.iteritems() if key.find('.') < 0]
>for dd in dirs:
>     easyinst_pth.write(dd+'\n')
>...

That won't work correctly with respect to linking the package's 
.egg-info to its import path, and of course, uninstall still has to 
be handled.  Implementing uninstall would be further complicated by 
the possibility of package_dir having been changed between the 
install and subsequent uninstall, or any intervening re-install.

In short, it'd be a serious pain, and I haven't see any use cases for 
doing anything with package_dir besides defining a root location for 
packages (e.g. a src/ or lib/ directory).  I have no idea why the 
ability to do that was added to distutils in the first place, and if 
I ever did a full distutils replacement, it'd likely be one of the 
very first things lined up against the wall and shot.  ;-)


From oliver.schoenborn at utoronto.ca  Tue Mar 25 17:52:03 2008
From: oliver.schoenborn at utoronto.ca (Oliver Schoenborn)
Date: Tue, 25 Mar 2008 12:52:03 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
In-Reply-To: <20080325164235.4C9D13A4074@sparrow.telecommunity.com>
References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>
	<20080325151819.199753A40FF@sparrow.telecommunity.com>
	<20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca>
	<20080325164235.4C9D13A4074@sparrow.telecommunity.com>
Message-ID: <20080325125203.iaa6yzdanon48swk@webmail.utoronto.ca>

Quoting "Phillip J. Eby" <pje at telecommunity.com>:

> At 12:07 PM 3/25/2008 -0400, Oliver Schoenborn wrote:
>> Quoting "Phillip J. Eby" <pje at telecommunity.com>:
>>
>>> At 10:39 AM 3/25/2008 -0400, Oliver Schoenborn wrote:
>>>> I have a problem with "setup.py develop" installation.
>>>>
>>>> ...
>>>
>>> This use of package_dir is not compatible with 'develop'.  For
>>> 'develop' to work on a project, you may not have any non-empty keys in
>>> package_dir.  If you want to use develop with this project, you will
>>> have to alter your directory layout so that 'core' is under 'extern',
>>> or else move the 'extern' modules out of 'extern' and up to the
>>> top-level directory, and update (or get rid of) package_dir accordingly.
>>
>> Thanks for your quick reply. That's a pitty, would it be difficult to
>> patch? E.g. from a list of all 'values' in the package_dir then add a
>> line to easy-install.pth:
>>
>> # pseudo code:
>> ...
>> dirs = [os.path.join(os.getcwd(), dd) for key, dd
>>        in package_dir.iteritems() if key.find('.') < 0]
>> for dd in dirs:
>>    easyinst_pth.write(dd+'\n')
>> ...
>
> ...
>
> In short, it'd be a serious pain, and I haven't see any use cases for
> doing anything with package_dir besides defining a root location for
> packages (e.g. a src/ or lib/ directory).  I have no idea why the
> ability to do that was added to distutils in the first place, ...

Well, I ended up restructuring the source to reflect final  
installation, ie moved what was in extern into a new src folder, and  
move pubsubcore folder into src too. However, I still need to use  
package_dir:

setup(...
     packages     = [''],
     package_dir  = {'': 'src'},
     ...
)

How would you do it without package_dir?

Oliver





From tseaver at palladion.com  Tue Mar 25 17:56:03 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 25 Mar 2008 12:56:03 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
In-Reply-To: <20080325164235.4C9D13A4074@sparrow.telecommunity.com>
References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>	<20080325151819.199753A40FF@sparrow.telecommunity.com>	<20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca>
	<20080325164235.4C9D13A4074@sparrow.telecommunity.com>
Message-ID: <47E92EA3.4080809@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phillip J. Eby wrote:
> At 12:07 PM 3/25/2008 -0400, Oliver Schoenborn wrote:
>> Quoting "Phillip J. Eby" <pje at telecommunity.com>:
>>
>>> At 10:39 AM 3/25/2008 -0400, Oliver Schoenborn wrote:
>>>> I have a problem with "setup.py develop" installation.
>>>>
>>>> ...
>>> This use of package_dir is not compatible with 'develop'.  For
>>> 'develop' to work on a project, you may not have any non-empty keys in
>>> package_dir.  If you want to use develop with this project, you will
>>> have to alter your directory layout so that 'core' is under 'extern',
>>> or else move the 'extern' modules out of 'extern' and up to the
>>> top-level directory, and update (or get rid of) package_dir accordingly.
>> Thanks for your quick reply. That's a pitty, would it be difficult to
>> patch? E.g. from a list of all 'values' in the package_dir then add a
>> line to easy-install.pth:
>>
>> # pseudo code:
>> ...
>> dirs = [os.path.join(os.getcwd(), dd) for key, dd
>>         in package_dir.iteritems() if key.find('.') < 0]
>> for dd in dirs:
>>     easyinst_pth.write(dd+'\n')
>> ...
> 
> That won't work correctly with respect to linking the package's 
> .egg-info to its import path, and of course, uninstall still has to 
> be handled.  Implementing uninstall would be further complicated by 
> the possibility of package_dir having been changed between the 
> install and subsequent uninstall, or any intervening re-install.
> 
> In short, it'd be a serious pain, and I haven't see any use cases for 
> doing anything with package_dir besides defining a root location for 
> packages (e.g. a src/ or lib/ directory).  I have no idea why the 
> ability to do that was added to distutils in the first place, and if 
> I ever did a full distutils replacement, it'd likely be one of the 
> very first things lined up against the wall and shot.  ;-)

/me hums G & S chorus[1] in the background:

  He's got 'em on the list -- he's got 'em on the list;
  And they'll none of 'em be missed -- they'll none of 'em be missed.


[1] http://www.cs.rice.edu/~ssiyer/minstrels/poems/135.html



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH6S6j+gerLs4ltQ4RAhZLAKCDSxEmKnG4a0jZxH5ru0XXWqX96wCeKYYR
AYIMBgAVfvME63eRIPmeBKg=
=Ivo1
-----END PGP SIGNATURE-----


From pje at telecommunity.com  Tue Mar 25 18:22:28 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 13:22:28 -0400
Subject: [Distutils] setup.py develop un-egg-spected result
In-Reply-To: <20080325125203.iaa6yzdanon48swk@webmail.utoronto.ca>
References: <20080325103931.o6t2dvs7pxhpckw8@webmail.utoronto.ca>
	<20080325151819.199753A40FF@sparrow.telecommunity.com>
	<20080325120721.l7065o2nznwoo84s@webmail.utoronto.ca>
	<20080325164235.4C9D13A4074@sparrow.telecommunity.com>
	<20080325125203.iaa6yzdanon48swk@webmail.utoronto.ca>
Message-ID: <20080325172228.B1F843A4074@sparrow.telecommunity.com>

At 12:52 PM 3/25/2008 -0400, Oliver Schoenborn wrote:
>I still need to use package_dir:
>
>setup(...
>     packages     = [''],
>     package_dir  = {'': 'src'},
>     ...
>)

That's fine - that layout is fully supported by the 'develop' command.

>How would you do it without package_dir?

If we were replacing the distutils, I'd simply have a "source root" 
option, rather than a dictionary with a magic key.


From me at lbruno.org  Tue Mar 25 18:45:51 2008
From: me at lbruno.org (Luis Bruno)
Date: Tue, 25 Mar 2008 17:45:51 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <20080321190226.4198D3A4074@sparrow.telecommunity.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E3D11F.4030503@online.de>	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
Message-ID: <47E93A4F.1090900@lbruno.org>

Phillip J. Eby escreveu:
> Of course, one possible solution is for both A and B to depend on a  
> "virtual package" that contains C, such that both A and B can install 
> it if it's not there, and list it in their dependencies.
No need. Is this is what you want?
http://www.debian.org/doc/debian-policy/ap-pkg-diversions.html

I've not-so-recently advocated actually building .debs/.rpm/.whatever 
instead of reinventing wheels in .eggs. Maybe that would be an 
worthwhile discussion.

From pje at telecommunity.com  Tue Mar 25 19:31:58 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 14:31:58 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <47E93A4F.1090900@lbruno.org>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<47E93A4F.1090900@lbruno.org>
Message-ID: <20080325183201.7441D3A4074@sparrow.telecommunity.com>

At 05:45 PM 3/25/2008 +0000, Luis Bruno wrote:
>Phillip J. Eby escreveu:
> > Of course, one possible solution is for both A and B to depend on a
> > "virtual package" that contains C, such that both A and B can install
> > it if it's not there, and list it in their dependencies.
>No need. Is this is what you want?
>http://www.debian.org/doc/debian-policy/ap-pkg-diversions.html

No.  Even aside from the fact that it is debian-specific, it is not 
intended for the use case of a symmetrically-shared and duplicated file.


From lxander.m at gmail.com  Tue Mar 25 20:11:13 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Tue, 25 Mar 2008 15:11:13 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <47E93A4F.1090900@lbruno.org>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<47E93A4F.1090900@lbruno.org>
Message-ID: <525f23e80803251211i3135b2afvcef27afcff6a2f8@mail.gmail.com>

On Tue, Mar 25, 2008 at 1:45 PM, Luis Bruno <me at lbruno.org> wrote:
> Phillip J. Eby escreveu:
> > Of course, one possible solution is for both A and B to depend on a
>  > "virtual package" that contains C, such that both A and B can install
>  > it if it's not there, and list it in their dependencies.
>  No need. Is this is what you want?
>  http://www.debian.org/doc/debian-policy/ap-pkg-diversions.html
>
>  I've not-so-recently advocated actually building .debs/.rpm/.whatever
>  instead of reinventing wheels in .eggs. Maybe that would be an
>  worthwhile discussion.

I included this as an alternative in my stab at a rationale and
lead-in to a proposal. You can see Floris Bruynooghe's response
earlier in the thread, but basically it is an even harder problem to
create packages for every major platform automatically than it is to
create a simple database of installed packages that plays well with
every major package manager. How did you advocate solving this
problem? I agree that if every python package provided a
MSI/DEB/RPM/MPKG/etc., then there wouldn't be a need to install python
packages without a system-level installer. It just doesn't seem
practical to achieve that, so there remains, at the very least, the
need to provide a programmatic means to determine what versions of
what things are currently installed in your python environment. Well,
when I say "at the very least" I'm assuming that one can accept that a
non-negligible portion of the python community would like to maintain
and use a tool to help automate the task of adding or removing python
packages not available as system packages.

That said, my motivation for including at least briefly a list of
alternatives was to encourage others to flesh them out a bit more (or
porpose new ones) as maybe one of them really is a better! I
personally like the switchable sub-environment idea even though it
only side steps the issues instead of solving them.

From pje at telecommunity.com  Tue Mar 25 20:38:44 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 25 Mar 2008 15:38:44 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
 beyond
In-Reply-To: <525f23e80803251211i3135b2afvcef27afcff6a2f8@mail.gmail.com
 >
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<47E93A4F.1090900@lbruno.org>
	<525f23e80803251211i3135b2afvcef27afcff6a2f8@mail.gmail.com>
Message-ID: <20080325193844.DD7CE3A4074@sparrow.telecommunity.com>

At 03:11 PM 3/25/2008 -0400, Alexander Michael wrote:
>I agree that if every python package provided a
>MSI/DEB/RPM/MPKG/etc., then there wouldn't be a need to install python
>packages without a system-level installer.

Even if they did, there would still be a need.  System-level packages 
aren't generally retargetable to different locations, so there's no 
way to manage development sandboxes with them, or private application 
environments.  Oh, and let's not forget shared hosting.

No, system packages aren't even close to being the answer, for 
package authors *or* users.


From lxander.m at gmail.com  Wed Mar 26 11:33:38 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Wed, 26 Mar 2008 06:33:38 -0400
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <20080325003310.GA27433@laurie.devork>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E40726.1090209@egenix.com>
	<20080321212127.791B63A4074@sparrow.telecommunity.com>
	<47E4330C.5070601@egenix.com>
	<20080321224110.657C63A40A5@sparrow.telecommunity.com>
	<47E46176.4090205@v.loewis.de>
	<525f23e80803220621u4e826023u975048a45b76a079@mail.gmail.com>
	<47E51179.5040601@v.loewis.de>
	<525f23e80803241357r63b091c1r5b5f46c9734b13ce@mail.gmail.com>
	<20080325003310.GA27433@laurie.devork>
Message-ID: <525f23e80803260333w76693a53ke5ac910d02083c7@mail.gmail.com>

On Mon, Mar 24, 2008 at 8:33 PM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
>  Something that this requirement should probably explicitly describe:
>  does this database only concern modules, i.e. installed in a directory
>  that's on sys.path?  My understanding is that it does.  However I'm
>  unsure as to how that can be combined with supporting applications
>  that require certain modules.  Would every tool need to keep track of
>  the apps it installed separately to make sure that, when it's used to
>  uninstall a module, it doesn't render an app that it installed
>  useless?

Has the question of "what does it mean to be installed?" been
discussed yet? To me it seems that "python setup.py install"
constitutes installing something, but what about "python  setup.py
install --prefix=~/python-packages" or the various schemes
(PYTHONPATH, .pth files) to modify sys.path?

A possible way to accommodate the possibly distributed nature of a
python installation would be for every directory in sys.path to get
its own database so that a first-class environment can be constructed
from sys.path manipulations. But what about packages that have simply
been placed on sys.path without using distutils? Is it even
conceivable that they could be registered?

One way to be flexible here is to construct the database by scanning
the sys.path for packages and modules with PKG-INFO (ignoring the
specifics of how they present it for now), caching in each sys.path
directory the database of its own local contents. This way, you could
rebuild the database to reflect the current state (including any
manually copied packages) when and if desired.

From chris at simplistix.co.uk  Wed Mar 26 19:06:29 2008
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 26 Mar 2008 18:06:29 +0000
Subject: [Distutils] metadata/versions
In-Reply-To: <20080325152522.9AA853A4074@sparrow.telecommunity.com>
References: <34851.86.154.1.176.1206129466.squirrel@secure.simplistix.co.uk>
	<525f23e80803221152p21597de3sf54b8a306d7f43e9@mail.gmail.com>
	<47E908AC.7030709@simplistix.co.uk>
	<525f23e80803250754n2dc593el741241a5abedfeb@mail.gmail.com>
	<20080325152522.9AA853A4074@sparrow.telecommunity.com>
Message-ID: <47EA90A5.2010508@simplistix.co.uk>

Phillip J. Eby wrote:
> to download them.  Using a requires/provides model for dependency 
> resolution would have required new PyPI infrastructure to be built out, 

I find it shocking that PyPI doesn't have either a human or machine 
readable list of dependencies for each version of each package.
I mean really, even as a human, how am I supposed to look at a package 
and PyPI and figure out what else it needs and if it's going to be 
compatible with my current set of installed packages?

> PyPI and the requires/provides PEPs also appear to carry an implicit 
> assumption that these requires/provides are leveled on project+version, 
> without any distinction by Python version and target platform.  

True enough. I guess requirements becomes:

- the current python version
   (although, in the absence of a version-specific package, I'd imagine a
    non-version-specific package should suffice)
- the current architecture
   (although, in the absence of an architecture-specific package, I'd
    imagine a architecture-independent package should suffice)

Still, I wonder if the strategy should be inverted. What I usually end 
up doing by hand is:

- if the package has extension modules:

   - if I'm on unix, download the source install, run setup.py install

   - if I'm on windows, download the python-specific installer. If none
     is available, bin the package

- if the package has no extension modules, download the source install 
and run setup.py install

Of course, there's also a great deal of "wtf version of packagex do I 
currenlty have installed anyway?!" as well :-(

> Setuptools OTOH, assumes that specific dependencies are a property of a 
> *binary* distribution, not a source distribution.  (i.e., the PEPs put 
> the dependency info in a source package's PKG-INFO...  where it's pretty 
> much useless for any purpose).

Yeah, something like PKG-INFO should appear somewhere standard in 
whatever ends up in site-packages or similar...

> I've previously posted some of these comments about the 
> requires/provides PEPs on Python-Dev, Distutils-SIG, and Catalog-SIG, 
> with no response from anyone.

You're making the fatal assumption that people who have views about 
these things appear on these SIGs :-(
I've cared for many years but only found out about the distutils-sig 
relatively recently. Dunno what catalog-sig is and I don't think I 
belong anywhere near python-dev ;-)

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk

From skip at pobox.com  Wed Mar 26 20:28:35 2008
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 26 Mar 2008 14:28:35 -0500
Subject: [Distutils] CRLF in zipped packages?
Message-ID: <18410.41955.734921.523476@montanaro-dyndns-org.local>


I pulled processing 0.50 from PyPI today in the form of a zip archive.  I
unzipped it on a Solaris machine using "unzip -a" but it thought all the
files were binary (and zipinfo confirms this), so it didn't convert the CRLF
line terminators to LF.  Is this a side effect of the way the zip archive
was constructed?  I don't see anything conclusive which tells me the zip
archive was built with distutils, but you folks are the packaging experts so
I thought I'd ask here first.  Is there something I can do simpler than a
shell for loop and tr -d '\015' command to get rid of the annoying ^M
characters?

Thanks,

Skip Montanaro

From regebro at gmail.com  Wed Mar 26 21:20:25 2008
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 26 Mar 2008 21:20:25 +0100
Subject: [Distutils] [Python-Dev] how to easily consume just the parts
	of eggs that are good for you
In-Reply-To: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com>
References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com>
Message-ID: <319e029f0803261320r60f1e8f5mfe679e8c1e111299@mail.gmail.com>

Has somebody made a list of the problems with eggs? Because I use them
all the time and hasn't encountered any problems whatsoever, myself...
:) So I am a bit surprised at the various discussions about them.

From zooko at zooko.com  Wed Mar 26 20:18:54 2008
From: zooko at zooko.com (zooko)
Date: Wed, 26 Mar 2008 12:18:54 -0700
Subject: [Distutils] how to easily consume just the parts of eggs that are
	good for you
Message-ID: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com>

Folks:

Here is a simple proposal:  make the standard Python "import"  
mechanism notice eggs on the PYTHONPATH and insert them (into the  
*same* location) on the sys.path.

This eliminates the #1 problem with eggs -- that they don't easily  
work when installing them into places other than your site-packages  
and that if you allow any of them to be installed on your system then  
they take precedence over your non-egg packages even you explicitly  
put those other packages earlier in your PYTHONPATH.  (That latter  
behavior is very disagreeable to more than a few prorgammers.)

This also preserves most of the value of eggs for many use cases.

This is backward-compatible with most current use cases that rely on  
eggs.

This is very likely forward-compatible with new schemes that are  
currently being cooked up and will be deployed in the future.

Regards,

Zooko


From zooko at zooko.com  Thu Mar 27 02:09:45 2008
From: zooko at zooko.com (zooko)
Date: Wed, 26 Mar 2008 18:09:45 -0700
Subject: [Distutils] setuptools bug report: _get_unpatched() is way too
	clever (and also returns the wrong module)
Message-ID: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com>

Bug report #1:

_get_unpatched() returns a module from the system site-packages  
instead of the module from the PYTHONPATH which ought to have  
precedence, leading to setuptools-0.6c8 invoking setuptools-0.6a9 in  
the following stack trace:

PYTHONPATH="/home/buildslave/slave-tahoe/dapper/build/support/lib/ 
python2.4/site-packages" python ./setup.py develop --prefix="/home/ 
buildslave/slave-tahoe/dapper/build/support"
...
Running Nevow-0.9.18/setup.py -q bdist_egg --dist-dir /tmp/ 
easy_install-rk83KQ/Nevow-0.9.18/egg-dist-tmp-nJ8GSy
Traceback (most recent call last):
...
   File "/home/buildslave/slave-tahoe/dapper/build/ 
setuptools-0.6c8.egg/setuptools/command/develop.py", line 102, in  
install_for_development
   File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/ 
easy_install.py", line 518, in process_distribution

Full details:

https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/1427/ 
steps/compile/logs/stdio


Bug report #2:

_get_unpatched() is way too clever for me to spend time trying to  
understand and debug right now:

def _get_unpatched(cls):
     """Protect against re-patching the distutils if reloaded

     Also ensures that no other distutils extension monkeypatched the  
distutils
     first.
     """

What the heck?  I don't want to think about that, so instead we're  
delisting Ubuntu Dapper as a supported platform, since it is the only  
one of our supported platforms on which we have this problem.


Generalized complaint:

There are a lot of things that are too clever in setuptools/ 
easy_install/eggs.  Cleverness makes it work in more situations,  
which is good, but it also means that it is harder to know when it  
will fail or to understand why it failed.

A simpler system that satisfied 90% of the use cases while failing in  
more obvious ways would be appreciated.  (Provided, of course, that  
those 90% of use cases include the ones that I really care about.)

See my earlier post which proposes such a simplification:

http://mail.python.org/pipermail/python-dev/2008-March/078243.html


Bug report #3:

setuptools/easy_install/eggs needs a bug tracker.  (Also, of course,  
a buildbot and a rich suite of automated unit tests.)  Some of this  
message may have been inappropriate for distutils-sig, but what else  
am I going to do with this issue before I delist Dapper and forget  
all about it?

I would be willing to host a trac instance to serve as a wiki and  
issue tracker for setuptools, and a buildbot.  (But I don't want to  
have anything to do with subversion -- I much prefer darcs.)


Regards,

Zooko

From pje at telecommunity.com  Thu Mar 27 02:56:26 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Mar 2008 21:56:26 -0400
Subject: [Distutils] setuptools bug report: _get_unpatched() is way too
 clever (and also returns the wrong module)
In-Reply-To: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com>
References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com>
Message-ID: <20080327015802.D36773A40AE@sparrow.telecommunity.com>

At 06:09 PM 3/26/2008 -0700, zooko wrote:
>Bug report #1:
>...
>PYTHONPATH="/home/buildslave/slave-tahoe/dapper/build/support/lib/
>python2.4/site-packages" python ./setup.py develop --prefix="/home/
>buildslave/slave-tahoe/dapper/build/support"
>...
>Running Nevow-0.9.18/setup.py -q bdist_egg --dist-dir /tmp/
>easy_install-rk83KQ/Nevow-0.9.18/egg-dist-tmp-nJ8GSy
>Traceback (most recent call last):
>...
>    File "/home/buildslave/slave-tahoe/dapper/build/
>setuptools-0.6c8.egg/setuptools/command/develop.py", line 102, in
>install_for_development
>    File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/
>easy_install.py", line 518, in process_distribution
>
>Full details:
>
>https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/1427/
>steps/compile/logs/stdio

That URL requires login credentials that I don't have.  And I need 
the rest of the traceback to make any sense of this.


>Bug report #2:
>
>_get_unpatched() is way too clever for me to spend time trying to
>understand and debug right now:
>...
>What the heck?  I don't want to think about that, so instead we're
>delisting Ubuntu Dapper as a supported platform, since it is the only
>one of our supported platforms on which we have this problem.

Looking at what limited information you've given me, I doubt the 
problem has anything to do with this function, whose operation is 
ridiculously simple - it finds the first base class that wasn't 
defined by setuptools, and *was* defined by the distutils.  Since the 
distutils don't define an easy_install *or* develop command, neither 
one uses _get_unpatched and there should thus be no way that this is 
causing the weird import situation you have.

This is 99.9% unlikely to be related in *any* way to 
_get_unpatched(), and I'm only hedging the .1% because I'm cautious 
by nature.  :)  (Well, that, and there's not a lot of info to go on here.)


From zooko at zooko.com  Thu Mar 27 03:25:17 2008
From: zooko at zooko.com (zooko)
Date: Wed, 26 Mar 2008 19:25:17 -0700
Subject: [Distutils] setuptools bug report: _get_unpatched() is way too
	clever (and also returns the wrong module)
In-Reply-To: <20080327015802.D36773A40AE@sparrow.telecommunity.com>
References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com>
	<20080327015802.D36773A40AE@sparrow.telecommunity.com>
Message-ID: <561C960C-4663-4E26-BB4F-14FFBE6618F5@zooko.com>

On Mar 26, 2008, at 6:56 PM, Phillip J. Eby wrote:
>>
>> https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/ 
>> 1427/steps/compile/logs/stdio
>
> That URL requires login credentials that I don't have.  And I need  
> the rest of the traceback to make any sense of this.

My apologies -- I meant:

http://allmydata.org/buildbot/builders/dapper/builds/1427/steps/ 
compile/logs/stdio


> Looking at what limited information you've given me, I doubt the  
> problem has anything to do with this function, whose operation is  
> ridiculously simple - it finds the first base class that wasn't  
> defined by setuptools, and *was* defined by the distutils.  Since  
> the distutils don't define an easy_install *or* develop command,  
> neither one uses _get_unpatched and there should thus be no way  
> that this is causing the weird import situation you have.
>
> This is 99.9% unlikely to be related in *any* way to _get_unpatched 
> (), and I'm only hedging the .1% because I'm cautious by  
> nature.  :)  (Well, that, and there's not a lot of info to go on  
> here.)

Okay, then I admit that _get_unpatched()'s cleverness may not be the  
cause of this bug, but it was certainly the cause of Brian Warner and  
me ceasing to debug this, since we needed to grok _get_unpatched() in  
order to determine that it was not the cause of the problem.


My offer to host an issue tracker for setuptools/easy_install/ 
pkg_resources/eggs is still open.  It could be a good way to organize  
and remember knowledge.


Regards,

Zooko


From chrism at plope.com  Thu Mar 27 03:34:14 2008
From: chrism at plope.com (Chris McDonough)
Date: Wed, 26 Mar 2008 22:34:14 -0400
Subject: [Distutils] [Python-Dev] how to easily consume just the parts
 of eggs that are	good for you
In-Reply-To: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com>
References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com>
Message-ID: <47EB07A6.6040800@plope.com>

zooko wrote:
> Folks:
> 
> Here is a simple proposal:  make the standard Python "import"  
> mechanism notice eggs on the PYTHONPATH and insert them (into the  
> *same* location) on the sys.path.
> 
> This eliminates the #1 problem with eggs -- that they don't easily  
> work when installing them into places other than your site-packages  
> and that if you allow any of them to be installed on your system then  
> they take precedence over your non-egg packages even you explicitly  
> put those other packages earlier in your PYTHONPATH.  (That latter  
> behavior is very disagreeable to more than a few prorgammers.)

Sorry if I'm out of the loop and there's some subtlety here that I'm 
disregarding, but it doesn't appear that either of the issues you mention is a 
actually problem with eggs.  These are instead problems with how eggs get 
installed by easy_install (which uses a .pth file to extend sys.path).  It's 
reasonable to put eggs on the PYTHONPATH manually (e.g. 
sys.path.append('/path/to/some.egg')) instead of using easy_install to install them.

I don't think there would be any benefit to changing Python's import machinery 
to deal with them; they are essentially just directories (or zipfiles) that 
contain packages.

- C


From pje at telecommunity.com  Thu Mar 27 03:58:47 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 26 Mar 2008 22:58:47 -0400
Subject: [Distutils] setuptools bug report: _get_unpatched() is way too
 clever (and also returns the wrong module)
In-Reply-To: <561C960C-4663-4E26-BB4F-14FFBE6618F5@zooko.com>
References: <3219B2C8-AD42-4C74-8ECE-714EB66B63F8@zooko.com>
	<20080327015802.D36773A40AE@sparrow.telecommunity.com>
	<561C960C-4663-4E26-BB4F-14FFBE6618F5@zooko.com>
Message-ID: <20080327025845.0968B3A4062@sparrow.telecommunity.com>

At 07:25 PM 3/26/2008 -0700, zooko wrote:
>On Mar 26, 2008, at 6:56 PM, Phillip J. Eby wrote:
>>>
>>>https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/ 
>>>1427/steps/compile/logs/stdio
>>
>>That URL requires login credentials that I don't have.  And I need
>>the rest of the traceback to make any sense of this.
>
>My apologies -- I meant:
>
>http://allmydata.org/buildbot/builders/dapper/builds/1427/steps/ 
>compile/logs/stdio

Okay, looking at that, my guess is that the problem is due to the 
fact that setuptools 0.6c8 and its 'site.py' aren't installed until 
the "develop" command is run, and somehow causing the setuptools 
package to get split across two locations.  I would suggest that your 
makefile should ensure the desired version of setuptools is installed 
before anything else.  If that doesn't fix it, my next guess would be 
that Nevow's setup script is altering sys.path in some way, or that 
that or some other setup script tries to forcibly evict or reload 
setuptools.  That's about the only other way I can see for this to 
get split across two locations like that.

Are you using an ez_setup anywhere in there that monkeys with sys.modules?


>Okay, then I admit that _get_unpatched()'s cleverness may not be the
>cause of this bug, but it was certainly the cause of Brian Warner and
>me ceasing to debug this, since we needed to grok _get_unpatched() in
>order to determine that it was not the cause of the problem.

I don't really see how it relates in any way, since it's not listed 
in the traceback.  In any event, the traceback itself also appears 
unrelated to whatever problem Nevow's setup is having.


From ronaldoussoren at mac.com  Thu Mar 27 13:55:25 2008
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Thu, 27 Mar 2008 13:55:25 +0100
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6 and
	beyond
In-Reply-To: <525f23e80803251211i3135b2afvcef27afcff6a2f8@mail.gmail.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<47E93A4F.1090900@lbruno.org>
	<525f23e80803251211i3135b2afvcef27afcff6a2f8@mail.gmail.com>
Message-ID: <DA0237E6-F28B-4EC0-B188-1933E17B014A@mac.com>


On 25 Mar, 2008, at 20:11, Alexander Michael wrote:
> You can see Floris Bruynooghe's response
> earlier in the thread, but basically it is an even harder problem to
> create packages for every major platform automatically than it is to
> create a simple database of installed packages that plays well with
> every major package manager.

Why is that?  Looking in from the peanut gallery the major problem
with automaticly creating OS packages is political, in that some  
packagers
don't believe that developers could create valid OS packages.

The technical issues seem small compared to that.

Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2224 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080327/07d9b671/attachment.bin 

From python at dennis-kempin.de  Fri Mar 28 00:18:33 2008
From: python at dennis-kempin.de (Dennis Kempin)
Date: Thu, 27 Mar 2008 16:18:33 -0700
Subject: [Distutils] Multiple Egg files from one single setup.py
Message-ID: <200803271618.37056.python@dennis-kempin.de>

Hi folks,

i really like the setuptools project, it makes my life much easier, but 
recently i stumbled over the following problem:

My Project constist of 2 main egg distributions and a bunch of "plugin" eggs.

The first Problem ist: These projects are in the same source tree. like this:
src/project/master
src/project/plugins (namespace package)
src/project/plugins/pa
src/project/plugins/pb
src/project/plugins/pc

Well not a problem for setuptools since you can specify which packages to 
include into a distribution. 
But the problem is: when i call setup() multiple times in one setup.py file 
everything gets completly messed up. All egg files contain the whole project 
source.

My second solution was to create multiple setup.py files (setup-master.py, 
setup-pa.py, and so on) and call them via system(). But this resulted in some 
conflicts in the build directory. 

So well.. is it possible (and especially how is it possible) to create 
multiple egg files within one python instance (one setup.py file that is 
executed)?

And another question:
When i have all the plugin egg files packaged. I want to create a new egg file 
(project-dist) to distribute the master and all of its plugins (which are 
simply copied to a /etc/ folder).
Thus a single egg file that installs the master and copies some egg files to a 
specified folder.

thanks in advance and best regards,
Dennis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080327/b21b3d04/attachment.pgp 

From mail at dennis-kempin.de  Fri Mar 28 00:05:57 2008
From: mail at dennis-kempin.de (Dennis Kempin)
Date: Thu, 27 Mar 2008 16:05:57 -0700
Subject: [Distutils] Multiple Eggs in one setup file
Message-ID: <200803271606.00755.mail@dennis-kempin.de>

Hi folks,

i really like the setuptools project, it makes my life much easier, but 
recently i stumbled over the following problem:

My Project constist of 2 main egg distributions and a bunch of "plugin" eggs.

The first Problem ist: These projects are in the same source tree. like this:
src/project/master
src/project/plugins (namespace package)
src/project/plugins/pa
src/project/plugins/pb
src/project/plugins/pc

Well not a problem for setuptools since you can specify which packages to 
include into a distribution. 
But the problem is: when i call setup() multiple times in one setup.py file 
everything gets completly messed up. All egg files contain the whole project 
source.

My second solution was to create multiple setup.py files (setup-master.py, 
setup-pa.py, and so on) and call them via system(). But this resulted in some 
conflicts in the build directory. 

So well.. is it possible (and especially how is it possible) to create 
multiple egg files within one python instance (one setup.py file that is 
executed)?

And another question:
When i have all the plugin egg files packaged. I want to create a new egg file 
(project-dist) to distribute the master and all of its plugins (which are 
simply copied to a /etc/ folder).
Thus a single egg file that installs the master and copies some egg files to a 
specified folder.

thanks in advance and best regards,
Dennis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20080327/097575f0/attachment.pgp 

From floris.bruynooghe at gmail.com  Thu Mar 27 16:39:41 2008
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 27 Mar 2008 15:39:41 +0000
Subject: [Distutils] [Python-Dev] How we can get rid of eggs for 2.6	and
	beyond
In-Reply-To: <DA0237E6-F28B-4EC0-B188-1933E17B014A@mac.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>
	<47E3D11F.4030503@online.de>
	<18403.57501.22865.578079@montanaro-dyndns-org.local>
	<20080321190226.4198D3A4074@sparrow.telecommunity.com>
	<47E93A4F.1090900@lbruno.org>
	<525f23e80803251211i3135b2afvcef27afcff6a2f8@mail.gmail.com>
	<DA0237E6-F28B-4EC0-B188-1933E17B014A@mac.com>
Message-ID: <20080327153941.GB19256@laurie.devork>

On Thu, Mar 27, 2008 at 01:55:25PM +0100, Ronald Oussoren wrote:
>
> On 25 Mar, 2008, at 20:11, Alexander Michael wrote:
>> You can see Floris Bruynooghe's response
>> earlier in the thread, but basically it is an even harder problem to
>> create packages for every major platform automatically than it is to
>> create a simple database of installed packages that plays well with
>> every major package manager.
>
> Why is that?  Looking in from the peanut gallery the major problem
> with automaticly creating OS packages is political, in that some  
> packagers
> don't believe that developers could create valid OS packages.
>
> The technical issues seem small compared to that.

If all I describe in [1] seems political to you then here some
techinical difficulties:

Developing the tools required to create MSI databases and CAB files
for creating a MS windows install package on any platform python runs
on would be a huge project (msilib is implemented by calling the MS C
API).

Likewise it would be a massive job to create the toolchain in python
to create DEBs.  I don't know much about RPMs and all other systems
out there but if these two systems are any indication about the
complexity then supporting *every* OS out there is pretty hard.  Not
to mention some small distro that has their own format few people care
about.  Supporing everything in python itself as well as allowing
systems to use the python infrastructure is far easier.


Regards
Floris


[1] http://mail.python.org/pipermail/distutils-sig/2008-March/009112.html


> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ziade.tarek at gmail.com  Thu Mar 27 17:06:57 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 27 Mar 2008 17:06:57 +0100
Subject: [Distutils] Multiple Egg files from one single setup.py
In-Reply-To: <200803271618.37056.python@dennis-kempin.de>
References: <200803271618.37056.python@dennis-kempin.de>
Message-ID: <94bdd2610803270906i11952a6dyec6929c582ea1f6f@mail.gmail.com>

On Fri, Mar 28, 2008 at 12:18 AM, Dennis Kempin <python at dennis-kempin.de>
wrote:

> Hi folks,
>
> i really like the setuptools project, it makes my life much easier, but
> recently i stumbled over the following problem:
>
> My Project constist of 2 main egg distributions and a bunch of "plugin"
> eggs.
>
> The first Problem ist: These projects are in the same source tree. like
> this:
> src/project/master
> src/project/plugins (namespace package)
> src/project/plugins/pa
> src/project/plugins/pb
> src/project/plugins/pc
>
> Well not a problem for setuptools since you can specify which packages to
> include into a distribution.
> But the problem is: when i call setup() multiple times in one setup.pyfile
> everything gets completly messed up. All egg files contain the whole
> project
> source.


What I would do is to create several eggs with these names:

project.master
project.plugin.pa
project.plugin.pb
project.plugin.pc

Then organize the code like this on your box/repo:

src/project/packages
    project.master/
                 setup.py
                 etc...

    project.plugin.pa
               setup.py
               project
                     plugin
                          pa
                             the code
    etc..


This is how Zope 3 is organized for instance: a bunch of eggs, that share
the same namespaces
but that are separated in independant folders, since setuptools allow
several packages
to share the same top namespace.


>
> My second solution was to create multiple setup.py files (setup-master.py,
> setup-pa.py, and so on) and call them via system(). But this resulted in
> some
> conflicts in the build directory.
>
> So well.. is it possible (and especially how is it possible) to create
> multiple egg files within one python instance (one setup.py file that is
> executed)?


I am not sure you can, but if you do as mentioned below, you should be fine.


>
>
> And another question:
> When i have all the plugin egg files packaged. I want to create a new egg
> file
> (project-dist) to distribute the master and all of its plugins (which are
> simply copied to a /etc/ folder).
> Thus a single egg file that installs the master and copies some egg files
> to a
> specified folder.


If your application is based on a collection of eggs, you can add them in
the "install_requires"
meta-data of a "main" egg. This will fetch them and install them as
dependencies.
But each egg has to be available at PyPI

Another solution to wrap an egg-based application is to use a tool like
zc.buildout.

Now this is how I work and do with eggs, and how most people do in Zope
community,
but there might be other ways, I don't know.

Regards,
Tarek


-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20080327/b763323c/attachment.htm 

From jeff at drinktomi.com  Fri Mar 28 09:12:52 2008
From: jeff at drinktomi.com (Jeff Younker)
Date: Fri, 28 Mar 2008 01:12:52 -0700
Subject: [Distutils] How we can get rid of eggs for 2.6 and beyond
In-Reply-To: <47E82AA2.1020502@palladion.com>
References: <20080321134749.1C8993A40B0@sparrow.telecommunity.com>	<47E40726.1090209@egenix.com>	<20080321212127.791B63A4074@sparrow.telecommunity.com>	<47E4330C.5070601@egenix.com>	<47E4DE8E.6010809@holdenweb.com>	<79990c6b0803220646r50e9b455y5bd9b955814f334a@mail.gmail.com>
	<47E53435.4090104@v.loewis.de> <47E82AA2.1020502@palladion.com>
Message-ID: <E5A2DE54-7465-4321-AAC4-98A44E1CC9BB@drinktomi.com>

On Mar 24, 2008, at 3:26 PM, Tres Seaver wrote:

> Sharing the system python is hugely problematic on a unix box which
> actually *uses* python for its own tools:  the application is not  
> "safe"
> from additions / updates / removeals of the packages in
> /usr/lib/python2.x/site-packages done to support those system tools.
> The problem gets worse as multiple non-system applications install  
> files
> there:  e.g., the 'twisted' package on Debian boxes depends on an
> ancient version of 'zope.interface', which can't be used with any
> currently supported version of Zope.

This is why versioning would be an useful solution.  Each package
would use the dependent packages that it requires.  Foo 1.0 uses
Bar 2.3 and Baz 3.2 uses Foo 1.4.  An application can use both Foo 1.0
and Baz 3.2 without having to mediate between their requirements.
While nobody is really requesting versioning, it seems to be the
solution to many problems that plague us.

-jeff

From lxander.m at gmail.com  Fri Mar 28 16:02:19 2008
From: lxander.m at gmail.com (Alexander Michael)
Date: Fri, 28 Mar 2008 11:02:19 -0400
Subject: [Distutils] A Modest Proposal for "A Database of Installed Packages"
Message-ID: <525f23e80803280802r507322car646b6110799934b@mail.gmail.com>

I'll continue my fool hearty effort [1] to build a concrete proposal
for "a database of installed packages" by offering up a sketch of a
possible straw-man "solution". I realize that this is likely
oversimplified to a fault, but I hope it will help us move forward.
Apologies if the equivalent of this has been proposed and rejected
before. My proposal is basically to make PKG-INFO functional and
usable by:

* Fixing the technical issues with requirements (i.e. dependencies)
and naming as specified by PEP 314/345.

* Modifying distutils to install PKG-INFO alongside each module file
or package directory as a side-car file of the same name but with a
special extension (.pyi or whatever). These files would be the place
to include the optional list of installed files as well as the
optional md5sums, if desired by the installer. Files in the package
will be listed using relative paths, while far flung files (bin,
shared, etc) will get full paths so that there is full allowance for
relocating simple (nothing in bin or shared) modules and packages.
Although optional, "python setup.py install" will include the
installed file list by default.

That's it. The intent is to provide just enough information to allow
the development of tools to use it, for those that are interested,
while being minimally invasive to developers that are not interested
in such tools. To determine the current state of your python
environment, walk sys.path looking for modules and packages,
collecting PKG-INFO when available. No standard centralized database.
Some of us will choose to opt-in to a particular installation
management tool that might maintain a cache (centralized or
per-directory) for efficiency, but that would be considered a
performance optimization for that particular tool.

We can also bootstrap older python installations by creating an online
database (that can of course be downloaded by security conscious
individuals for offline querying) that maps
(module-file/package-directory name, md5sum) pairs to their respective
PKG-INFO contents (no list of installed files) which can be queried by
an automated sys.path walker to fill-in missing side-car files. Thus,
I can opt-in to this scheme for python 2.5 by installing a distutils
patch and meta-data side-car bootstrapper that does its best to
identify what's on my sys.path. It would be quite tractable to
maintain this for the python standard library and perhaps the official
installations of a few major OS versions. Such a database could even
be used for the community to provide metadata for packages that the
developer didn't (again, furthering an opt-in mentality). Of course,
even though it worked for CDDB, it would likely be too much to expect
this level of coverage through user submitted entries.

[1] http://mail.python.org/pipermail/distutils-sig/2008-March/009108.html

From mhammond at skippinet.com.au  Mon Mar 31 09:20:07 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Mon, 31 Mar 2008 18:20:07 +1100
Subject: [Distutils] FW: [issue2513] 64bit cross compilation on windows
Message-ID: <002401c892ff$a8254630$f86fd290$@com.au>

FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited!

Thanks,

Mark
-----Original Message-----
From: Mark Hammond [mailto:report at bugs.python.org] 
Sent: Sunday, 30 March 2008 6:01 PM
To: mhammond at users.sourceforge.net
Subject: [issue2513] 64bit cross compilation on windows


New submission from Mark Hammond <mhammond at users.sourceforge.net>:

I've taken the liberty of adding Trent, Christian and Martin to the nosy
list as I know they are actively, if reluctantly interested in this.

This patch allows the distutils to cross-compile on Windows.  It has
been tested on x86 and amd64 platforms, with both platforms successfully
able to natively and cross-compile extension modules and create binary
distributions.

To cross-compile, specify '--plat-name' to the build command (valid
values are 'win32', 'win-amd64' and 'win-ia64').  This option name was
chosen to be consistent with the bdist_dumb command.  I've included the
docs I added below (which are also in the patch), but note that as with
native compilation using distutils, it's not necessary to set any
environment variables or do anything else special with your environment
to make this work.

The patch also adds a x64 target for the 'bdist_wininst' target, which
it creates as distutils/command/wininst-9.0-amd64.exe.  This executable
is necessary even for bdist_wininst to work natively on x64, but is
still included here for simplicity.
    
To assist with testing, I've also added a distutils setup.py script to
the PC/example_nt directory.  This is capable of creating bdist_wininst
executables for both native and cross platforms; 'setup.py build
--platname=win-amd64 bdist_wininst' will create an amd64 installer on an
x86 machine.

The patch has not been tested with a Visual Studio environment without
cross-compile tools installed - it will obviously fail, but its not
clear how ugly this failure will be.

Below is the text I added to docs/distutils/builtdist.rst:

  Cross-compiling on Windows
  =====================

  Starting with Python 2.6, distutils is capable of cross-compiling
between Windows platforms.  In practice, this means that with the
correct tools installed, you can use a 32bit version of Windows to
create 64bit extensions and vice-versa.

  To build for an alternate platform, specify the :option:`--plat-name`
option to the build command.  Valid values are currently 'win32',
'win-amd64' and 'win-ia64'.  For example, on a 32bit version of Windows,
you could execute::

     python setup.py build --plat-name=win-amd64

  to build a 64bit version of your extension.  The Windows Installers
also support this option, so the command::

     python setup.py build --plat-name=win-amd64 bdist_wininst

  would create a 64bit installation executable on your 32bit version of
Windows.

  Note that by default, Visual Studio 2008 does not install 64bit
compilers or tools.  You may need to reexecute the Visual Studio setup
process and select these tools.

----------
assignee: mhammond
components: Distutils
files: windows-cross-compile.patch
keywords: 64bit, patch
messages: 64743
nosy: Trent.Nelson, ctheune, loewis, mhammond
severity: normal
status: open
title: 64bit cross compilation on windows
type: feature request
versions: Python 2.6
Added file: http://bugs.python.org/file9900/windows-cross-compile.patch

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2513>
__________________________________