From ziade.tarek at gmail.com  Thu Apr  1 16:35:32 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 1 Apr 2010 16:35:32 +0200
Subject: [Distutils] PEP 376 Status
In-Reply-To: <94bdd2611003250320i720ba93epe7a37d5cc2ffaf34@mail.gmail.com>
References: <94bdd2611003250320i720ba93epe7a37d5cc2ffaf34@mail.gmail.com>
Message-ID: <n2u94bdd2611004010735g7e12e1eeu9b8db35550f60a46@mail.gmail.com>

On Thu, Mar 25, 2010 at 11:20 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
[..]
> Then I propose to focus on PEP 376, and to go ahead and try to have it
> accepted before the first beta of Python 2.7.

I think we are through with this, I'll send a mail to python-dev later
today, proposing the PEP.
(minor typo/editing tasks are in progress but I think the PEP is now
in a good enough shape to be pushed in python-dev)

Then I propose to focus on the installation schemes details (pycon
sprint + sysconfig work etc..)
and build a new PEP out of it :)

Regards
Tarek
-- 
Tarek Ziad? | http://ziade.org

From matt at tplus1.com  Thu Apr  1 20:51:10 2010
From: matt at tplus1.com (Matthew Wilson)
Date: Thu, 1 Apr 2010 18:51:10 +0000 (UTC)
Subject: [Distutils] How to separate dependencies for using my package vs
	developing my package?
Message-ID: <hp2puu$ejv$1@dough.gmane.org>

I'm working on a package with two kinds of dependencies.

My package needs jinja2 and docutils to generate output.

Then I use nose, sphinx, mock, and a bunch of other testing tools while
I'm writing code for my package.

I don't see users of the app needing to run the test code, but maybe
they might want to generate the docs.

In pip, easy_install, whatever, is there some recommended way of
describing these separate kinds of dependencies?


Matt


From marius at pov.lt  Fri Apr  2 15:40:51 2010
From: marius at pov.lt (Marius Gedminas)
Date: Fri, 2 Apr 2010 16:40:51 +0300
Subject: [Distutils] zc.buildout: bootstrap.py script
In-Reply-To: <94bdd2611003301550s565f7200j87d08c6360c41242@mail.gmail.com>
References: <201003310008.50477.rene@fleschenberg.net>
	<94bdd2611003301550s565f7200j87d08c6360c41242@mail.gmail.com>
Message-ID: <20100402134051.GB25866@fridge.pov.lt>

On Wed, Mar 31, 2010 at 12:50:33AM +0200, Tarek Ziad? wrote:
> On Wed, Mar 31, 2010 at 12:08 AM, Ren? Fleschenberg
> <rene at fleschenberg.net> wrote:
> > Hi,
> >
> > The code in bootstrap.py doesn't use the if __name__ == "__main__" idiom.
> > This can lead to funny behaviour if it is imported for some reason (I
> > stumbled upon this in conjunction with nose).
> >
> > Is there a reason for not checking for __name__ == "__main__", or should
> > this be changed?
> 
> I don't see any goog reason not to have a __main__ section,
> 
> +1 for changing this

There's an open bug for it, with a patch attached (not that you need a
patch for something this trivial):
https://bugs.launchpad.net/zc.buildout/+bug/303432

I assume Jim is just busy.

I assume anyone with commit access to Zope SVN could change this and
release a zc.buildout point release, if Jim has no objections.  Jim?

Marius Gedminas
-- 
"What's the name of the new OO COBOL -- an equivalent of C++?"
"ADD 1 TO COBOL GIVING COBOL"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100402/5b669115/attachment.pgp>

From ziade.tarek at gmail.com  Sun Apr  4 23:51:18 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 4 Apr 2010 23:51:18 +0200
Subject: [Distutils] Distutils2 GSOC tasks
Message-ID: <u2v94bdd2611004041451of54225daye051b0087eedf55e@mail.gmail.com>

Hi,

I have started to list tasks for Distutils2 for a possible GSOC
project : http://wiki.python.org/moin/SummerOfCode/Distutils2.

In case you are working on some tasks on distutils2, let me know so we
can synchronize the effort.

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From ziade.tarek at gmail.com  Mon Apr  5 22:46:03 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 5 Apr 2010 22:46:03 +0200
Subject: [Distutils] Distribute 0.6.11 to be shipped
Message-ID: <o2z94bdd2611004051346r80ddbce3n98de80e8c5cc986e@mail.gmail.com>

Hello,

I am planning to release 0.6.11 sometimes at the end of week. If you
have some pending issues you want to have in this
release , speak up.

If you want to try out 0.6.11, and/or help with some feedback, you can
install the dev version like this:

$ easy_install -U distribute==dev

Regards,
Tarek

-- 
Tarek Ziad? | http://ziade.org

From sridharr at activestate.com  Tue Apr  6 00:26:06 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Mon, 5 Apr 2010 15:26:06 -0700
Subject: [Distutils] Distribute 0.6.11 to be shipped
In-Reply-To: <o2z94bdd2611004051346r80ddbce3n98de80e8c5cc986e@mail.gmail.com>
References: <o2z94bdd2611004051346r80ddbce3n98de80e8c5cc986e@mail.gmail.com>
Message-ID: <ECC3784A-DB99-4204-8E16-DD7235E5C7BE@activestate.com>


On 2010-04-05, at 1:46 PM, Tarek Ziad? wrote:

> Hello,
> 
> I am planning to release 0.6.11 sometimes at the end of week. If you
> have some pending issues you want to have in this
> release , speak up.
> 
> If you want to try out 0.6.11, and/or help with some feedback, you can
> install the dev version like this:
> 
> $ easy_install -U distribute==dev

This command will not work if you have Distribute also installed in ~/.local

  http://bitbucket.org/tarek/distribute/issue/95

-srid


From ziade.tarek at gmail.com  Tue Apr  6 00:58:05 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 6 Apr 2010 00:58:05 +0200
Subject: [Distutils] Distribute 0.6.11 to be shipped
In-Reply-To: <ECC3784A-DB99-4204-8E16-DD7235E5C7BE@activestate.com>
References: <o2z94bdd2611004051346r80ddbce3n98de80e8c5cc986e@mail.gmail.com>
	<ECC3784A-DB99-4204-8E16-DD7235E5C7BE@activestate.com>
Message-ID: <j2w94bdd2611004051558i24dc2e74yf40583af746d6479@mail.gmail.com>

On Tue, Apr 6, 2010 at 12:26 AM, Sridhar Ratnakumar
<sridharr at activestate.com> wrote:
>
> On 2010-04-05, at 1:46 PM, Tarek Ziad? wrote:
>
>> Hello,
>>
>> I am planning to release 0.6.11 sometimes at the end of week. If you
>> have some pending issues you want to have in this
>> release , speak up.
>>
>> If you want to try out 0.6.11, and/or help with some feedback, you can
>> install the dev version like this:
>>
>> $ easy_install -U distribute==dev
>
> This command will not work if you have Distribute also installed in ~/.local
>
> ?http://bitbucket.org/tarek/distribute/issue/95
>

Thanks for the feedback, I'll look at it this week

> -srid
>
>



-- 
Tarek Ziad? | http://ziade.org

From jim at zope.com  Tue Apr  6 17:22:24 2010
From: jim at zope.com (Jim Fulton)
Date: Tue, 6 Apr 2010 11:22:24 -0400
Subject: [Distutils] zc.buildout: bootstrap.py script
In-Reply-To: <20100402134051.GB25866@fridge.pov.lt>
References: <201003310008.50477.rene@fleschenberg.net>
	<94bdd2611003301550s565f7200j87d08c6360c41242@mail.gmail.com> 
	<20100402134051.GB25866@fridge.pov.lt>
Message-ID: <l2p1099b90b1004060822rb9a63a19j6adad1ba8d463ea1@mail.gmail.com>

On Fri, Apr 2, 2010 at 9:40 AM, Marius Gedminas <marius at pov.lt> wrote:
> On Wed, Mar 31, 2010 at 12:50:33AM +0200, Tarek Ziad? wrote:
>> On Wed, Mar 31, 2010 at 12:08 AM, Ren? Fleschenberg
>> <rene at fleschenberg.net> wrote:
>> > Hi,
>> >
>> > The code in bootstrap.py doesn't use the if __name__ == "__main__" idiom.
>> > This can lead to funny behaviour if it is imported for some reason (I
>> > stumbled upon this in conjunction with nose).
>> >
>> > Is there a reason for not checking for __name__ == "__main__", or should
>> > this be changed?
>>
>> I don't see any goog reason not to have a __main__ section,
>>
>> +1 for changing this
>
> There's an open bug for it, with a patch attached (not that you need a
> patch for something this trivial):
> https://bugs.launchpad.net/zc.buildout/+bug/303432
>
> I assume Jim is just busy.
>
> I assume anyone with commit access to Zope SVN could change this and
> release a zc.buildout point release, if Jim has no objections. ?Jim?

Nope. +1

Jim

-- 
Jim Fulton

From amit.pureenergy at gmail.com  Wed Apr  7 19:30:22 2010
From: amit.pureenergy at gmail.com (Amit Sethi)
Date: Wed, 7 Apr 2010 23:00:22 +0530
Subject: [Distutils] modules in sub-packages
Message-ID: <o2jda81a0a81004071030g74a4fc95gdb3372d33113b49a@mail.gmail.com>

I am trying to create  a module for the docutils . I wish to install
it docutils.writer.something
I have docutils installed at
/usr/local/lib/python2.6/dist-packages/docutils-0.6-py2.6.egg
on my ubuntu machine but when I tried to install it create a seperate
package called docutils and saves
the package in that ? How do I tell a package that this particular
package is a subpackage of a library
the setup.py of the package looks like :
 license='Public Domain',
 packages=['docutils.writers'],
 package_dir={'docutils.writers':'docbook'},
 scripts=['rst2docbook.py']

-- 
A-M-I-T S|S

From pje at telecommunity.com  Wed Apr  7 19:55:00 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 07 Apr 2010 13:55:00 -0400
Subject: [Distutils] modules in sub-packages
In-Reply-To: <o2jda81a0a81004071030g74a4fc95gdb3372d33113b49a@mail.gmail .com>
References: <o2jda81a0a81004071030g74a4fc95gdb3372d33113b49a@mail.gmail.com>
Message-ID: <20100407175506.025E83A4063@sparrow.telecommunity.com>

At 11:00 PM 4/7/2010 +0530, Amit Sethi wrote:
>I am trying to create  a module for the docutils . I wish to install
>it docutils.writer.something

Is that a documented, official way to extend docutils?  If not, and 
docutils.writer is not a namespace package, then you should not be doing that.


>I have docutils installed at
>/usr/local/lib/python2.6/dist-packages/docutils-0.6-py2.6.egg
>on my ubuntu machine but when I tried to install it create a seperate
>package called docutils and saves
>the package in that ? How do I tell a package that this particular
>package is a subpackage of a library
>the setup.py of the package looks like :
>  license='Public Domain',
>  packages=['docutils.writers'],
>  package_dir={'docutils.writers':'docbook'},
>  scripts=['rst2docbook.py']

The correct way to do this is to NOT list packages or package_dir; 
just use py_modules = 'docutils.writer.docbook', and have a 
docutils/writer/docbook.py file.  This will install it in a 
distutils-compatible way, but not a distribute-compatible or 
setuptools-compatible way.

Really, this is not a well-supported scenario in any case -- it is to 
be avoided if at all possible.  Installing your code into other 
people's packages is a bad way to extend them.  (Setuptools and 
Distribute offer better ways of providing extensibility, such as 
namespace packages and entry points.)



From kevin at bud.ca  Wed Apr  7 23:45:13 2010
From: kevin at bud.ca (Kevin Teague)
Date: Wed, 7 Apr 2010 14:45:13 -0700 (PDT)
Subject: [Distutils] How to separate dependencies for using my package
	vs developing my package?
In-Reply-To: <hp2puu$ejv$1@dough.gmane.org>
References: <hp2puu$ejv$1@dough.gmane.org>
Message-ID: <95d962f5-bc51-4cca-bde4-6a1f09f3f005@x20g2000yqb.googlegroups.com>

With pip or easy_install, there isn't a formal way to describe a
different working set from the main application. However, you can use
a note in an INSTALL.txt and optionally a pip requirements file
specifying specific versions. If there are package conflicts between
the main application dependencies in the support tools though, the
installs need to be in different locations.

With buildout, this use case can be formally described, and the
dependencies between the main application and the support tools are
kept distinct. Typically this might looks something like:

    [buildout]
    develop = .
    parts = myapp i18ndude releaser

    [myapp]
    recipe = zc.recipe.egg
    egg = myapp

    [i18ndude]
    recipe = zc.recipe.egg
    egg = i18ndude

    [releaser]
    recipe = zc.recipe.egg
    eggs = zest.releaser

Where the 'myapp' part would read the ./setup.py file for it's
dependencies, and the 'i18ndude' and 'zest.releaser' parts would read
the dependencies from the respective egg's install_requires fields.
Usually only tools which are specifically part of the workflow will
get put into a buildout.cfg inside the packages source tree. Then I'll
maintain a seperate buildout.cfg in my ~/bin/ directory for tools
which I use for developing, but don't want to enforce others to
install (generally these are tools such as boilerplate generation,
e.g. ZopeSkel):

    [buildout]
    parts = eggs
    bin-directory = .

    [eggs]
    recipe = zc.recipe.egg:scripts
    eggs = ZopeSkel
           PasteScript
           grokproject

If you have additional Python code which you want to specifically
support generating documentation for the project, you can do what Grok
does for managing it's documentation generation. Have a buildout.cfg
which specify's a documentation part (http://svn.zope.org/grok/trunk/
buildout.cfg?rev=109433&view=markup), but also specify that package
should be installed from a development version. Then embed the
development version of that package within the source tree of the main
package (in this case, there is a sub-directory named 'grokdocs'):

[buildout]
develop =
  .
  grokdocs

[docs]
recipe = zc.recipe.egg
eggs = grokdocs

Then in the 'grokdocs' subdirectory, the setup.py can be as
descriptive as it needs to be to set-up a specific documentation
publishing infrastructure.

In the Grok example, the 'grokdocs' package isn't published to PyPI,
so you can only generate the docs from an SVN checkout. If you want to
be able to generate docs from released versions only, then you need to
publish the package - although typically if you have a documentation
publishing process as part of your release process, then usually the
users can easily access docs that match a specific release and there
isn't any need to generate these yourself ...

From ziade.tarek at gmail.com  Thu Apr  8 12:07:22 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 8 Apr 2010 12:07:22 +0200
Subject: [Distutils] Tentative release schedule for distutils2
Message-ID: <z2i94bdd2611004080307y8ff24a1dv749427bb2654418a@mail.gmail.com>

Hello,

FYI, here's a tentative release schedule for Distutils2

1.0a1 - 2010-04-30  (Python 2.7b2)
1.0a2 - 2010-05-28  (Python 2.7rc1 and GSOC starts)
1.0b1 - 2010-06-25 (Python 2.7 final)
1.0b2 - 2010-07-17  (GSOC Mid-term)
1.0c1 - 2010-08-30  (GSOC Ends)
1.0 final - 2010-09-15

We will probably have new releases after 1.0 quite quickly, depending
on the feedback

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From techtonik at gmail.com  Sat Apr 10 17:19:15 2010
From: techtonik at gmail.com (anatoly techtonik)
Date: Sat, 10 Apr 2010 18:19:15 +0300
Subject: [Distutils] Fwd: [PythonInfo Wiki] Update of "SiteImprovements" by
	97.86.230.99
In-Reply-To: <20100410144805.19589.49957@ximinez.python.org>
References: <20100410144805.19589.49957@ximinez.python.org>
Message-ID: <n2ud34314101004100819u61dc3487i14c2f5b463ccba6d@mail.gmail.com>

This is probably for you to consider.
-- 
anatoly t.



---------- Forwarded message ----------
From: python.org wiki <wiki at python.org>
Date: Sat, Apr 10, 2010 at 5:48 PM
Subject: [PythonInfo Wiki] Update of "SiteImprovements" by 97.86.230.99
To: "python.org wiki" <wiki at python.org>


Dear Wiki user,

You have subscribed to a wiki page or wiki category on "PythonInfo
Wiki" for change notification.

The "SiteImprovements" page has been changed by 97.86.230.99:
http://wiki.python.org/moin/SiteImprovements?action=diff&rev1=54&rev2=55


?== misc ==

+ ?* Explain how to use .egg package archives. ?As a new user, I am
unable to find anything, but I do find .egg files in the package
index.
? * Merge '''pydotorg-www''' and '''pydotorg-redesign''' lists into one
? ? http://mail.python.org/pipermail/pydotorg-www/2010-February/000082.html
? * Fix webmaster at python.org response bug (per Issue:7929)

From setuptools at bugs.python.org  Sat Apr 10 21:27:54 2010
From: setuptools at bugs.python.org (Dave Abrahams)
Date: Sat, 10 Apr 2010 19:27:54 +0000
Subject: [Distutils] [issue106] Confusing docs on PiPI
In-Reply-To: <1270927674.82.0.662104390101.issue106@psf.upfronthosting.co.za>
Message-ID: <1270927674.82.0.662104390101.issue106@psf.upfronthosting.co.za>


New submission from Dave Abrahams <dave at boostpro.com>:

At http://pypi.python.org/pypi/setuptools
"Install setuptools using the provided .exe installer" is confounding. Unt
il you scroll to the bottom of the page you have no clue where this .exe is
"provided," and surprisingly there's no link in the opening paragraph!

----------
messages: 510
nosy: dabrahams
priority: bug
status: unread
title: Confusing docs on PiPI

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue106>
_______________________________________________

From luigi.paioro at gmail.com  Wed Apr  7 17:46:07 2010
From: luigi.paioro at gmail.com (Luigi)
Date: Wed, 7 Apr 2010 08:46:07 -0700 (PDT)
Subject: [Distutils] Passing user and password to easy_install
Message-ID: <773fef21-2b45-484a-a20c-64c8686f1c40@i25g2000yqm.googlegroups.com>

Dear All,

I'm using setuptool module to pack my software and I'd like to
distribute it via PyPI, making it installable exploiting easy_install.

I have a requirement, though, that I'm not sure is supported by
setuptools:

My software must be distributed to registered/authenticated/authorized
users only.

Well, registering the packages I might specify that the download URL
is a CGI, optionally with Basic Authentication accepting a default
user and password as described here:

http://peak.telecommunity.com/DevCenter/EasyInstall#id17

This would provides me with the possibility of tracking the number of
downloads, but always with a single public user/password. But I need
to know who is downloading my packages.

Is there a way to associate a user and password to easy_install
command? Such data would be nice if passed as Basic Authentication
data to the download URL...

Thanks in advance.

Luigi

From simon at ikanobori.jp  Wed Apr 14 14:59:00 2010
From: simon at ikanobori.jp (Simon de Vlieger)
Date: Wed, 14 Apr 2010 14:59:00 +0200
Subject: [Distutils] Passing user and password to easy_install
References: <C452E142-574C-41F0-BFDD-76EB3171B8E0@ikanobori.jp>
Message-ID: <A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>

Luigi,

I am not sure if using PyPi for private package distribution (what  
this equates to) would be
a particularly nice idea. This way the PyPi would contain packages  
which cannot be installed
by users of PyPi.

Alas, I don't know of any ways distutils can fulfill your request in  
its current form.

Regards,

Simon de Vlieger

On 7 apr 2010, at 17:46, Luigi wrote:

> Dear All,
>
> I'm using setuptool module to pack my software and I'd like to
> distribute it via PyPI, making it installable exploiting easy_install.
>
> I have a requirement, though, that I'm not sure is supported by
> setuptools:
>
> My software must be distributed to registered/authenticated/authorized
> users only.
>
> Well, registering the packages I might specify that the download URL
> is a CGI, optionally with Basic Authentication accepting a default
> user and password as described here:
>
> http://peak.telecommunity.com/DevCenter/EasyInstall#id17
>
> This would provides me with the possibility of tracking the number of
> downloads, but always with a single public user/password. But I need
> to know who is downloading my packages.
>
> Is there a way to associate a user and password to easy_install
> command? Such data would be nice if passed as Basic Authentication
> data to the download URL...
>
> Thanks in advance.
>
> Luigi
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100414/23850c7e/attachment.html>

From bradallen137 at gmail.com  Thu Apr 15 18:15:47 2010
From: bradallen137 at gmail.com (Brad Allen)
Date: Thu, 15 Apr 2010 11:15:47 -0500
Subject: [Distutils] Passing user and password to easy_install
In-Reply-To: <A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>
References: <C452E142-574C-41F0-BFDD-76EB3171B8E0@ikanobori.jp>
	<A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>
Message-ID: <z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>

On Wed, Apr 14, 2010 at 7:59 AM, Simon de Vlieger <simon at ikanobori.jp> wrote:
> Luigi,
>
> I am not sure if using PyPi for private package distribution (what this
> equates to) would be
> a particularly nice idea. This way the PyPi would contain packages which
> cannot be installed
> by users of PyPi.
>
> Alas, I don't know of any ways distutils can fulfill your request in its
> current form.

Luigi, you might want to consider setting up a private package server
instead of using PyPI. Where I work, we publish our eggs internally
via an Apache server, so we are able to make use of Apache's security
features (like https, authentication, and IP restriction). This works
fine with setuptools and buildout, and will probably also work in
future with the new 'distribute' tools.

From ziade.tarek at gmail.com  Thu Apr 15 18:25:07 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 15 Apr 2010 18:25:07 +0200
Subject: [Distutils] Passing user and password to easy_install
In-Reply-To: <z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>
References: <C452E142-574C-41F0-BFDD-76EB3171B8E0@ikanobori.jp>
	<A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>
	<z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>
Message-ID: <v2h94bdd2611004150925n89ede347of1184186501b0114@mail.gmail.com>

On Thu, Apr 15, 2010 at 6:15 PM, Brad Allen <bradallen137 at gmail.com> wrote:
> On Wed, Apr 14, 2010 at 7:59 AM, Simon de Vlieger <simon at ikanobori.jp> wrote:
>> Luigi,
>>
>> I am not sure if using PyPi for private package distribution (what this
>> equates to) would be
>> a particularly nice idea. This way the PyPi would contain packages which
>> cannot be installed
>> by users of PyPi.
>>
>> Alas, I don't know of any ways distutils can fulfill your request in its
>> current form.
>
> Luigi, you might want to consider setting up a private package server
> instead of using PyPI. Where I work, we publish our eggs internally
> via an Apache server, so we are able to make use of Apache's security
> features (like https, authentication, and IP restriction). This works
> fine with setuptools and buildout, and will probably also work in
> future with the new 'distribute' tools.

+1

You can also take a look at the alternative PyPI implementations,
that implement the server-side parts of the upload and register command.

I've changed Python 2.6's pypirc format so you can define in it
several PyPI servers,
and use the name of the target PyPI when you run register or upload
(see the doc)

I've backported it in a standalone project called "collective.dist" if
you use python < 2.6

For the server implementations, we have so far:

- EggBasket
- PloneSoftwareCenter (an extension for plone so you need to set up a plone)
- chishop
- ...

regards
Tarek


> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | http://ziade.org

From jim at zope.com  Thu Apr 15 22:42:51 2010
From: jim at zope.com (Jim Fulton)
Date: Thu, 15 Apr 2010 16:42:51 -0400
Subject: [Distutils] zc.buildout and "Not upgrading because not running
	a local buildout command"
In-Reply-To: <1011690799.20100324113817@gmail.com>
References: <1011690799.20100324113817@gmail.com>
Message-ID: <n2w1099b90b1004151342ue78f457bu7ddeb3b651e1c966@mail.gmail.com>

On Wed, Mar 24, 2010 at 6:38 AM, Adam GROSZER <agroszer at gmail.com> wrote:
> Hello,
>
> Having problems here with the above.
>
> The thing is that there should be two projects installed on the same
> server, but the one is a zope KGS 3.4 the other is a
> ztk/bluebream-ish.
> The two have different version requirements for setuptools and
> zc.buildout.
> The buildout.cfg comes from the web like this:
> buildout -t 2 -c http://mypypi.org/++projects++/proj/proj-stage-0.0.4.cfg
> (On top of that this is driven by keas.build, but that should not
> matter)
> Buildout (1.4.1) is installed via easy_install and setuptools (0.6c9).
> Therefore the message "Not upgrading because not running a local
> buildout command". The message is reasonable, but still makes problems
> in the way of making the one of the projects non-installable.
>
> Would not it be easy to create a bin/buildout script and restart with
> that?
>
> Or is there any other sane solution for this (that I missed)?

  buildout init
  bin/buildout -t 2 -c http://mypypi.org/++projects++/proj/proj-stage-0.0.4.cfg

Jim

-- 
Jim Fulton

From eposse at gmail.com  Sat Apr 17 03:50:55 2010
From: eposse at gmail.com (Ernesto Posse)
Date: Fri, 16 Apr 2010 21:50:55 -0400
Subject: [Distutils] bdist dist-dir
Message-ID: <x2o4ecfdb121004161850q3dd0f7a1y9052a6df67660045@mail.gmail.com>

Hi. I have a couple of questions regarding the installation directory
when using binary distributions. When I do

python setup.py bdist

the generated gztar ball contains the files prefixed with the
directory where they will be unpacked, which in my machine is
"./usr/local/lib/python2.6/dist-packages", which is fine if the user
has the same version of Python installed in the same directory as I
do, but in general this is not the case. So my questions are:

1) why is this the behaviour of bdist?
2) is there a way to specify "install wherever the user has Python
installed" with binary distros? the --relative option still seems to
assume certain prefixes.
3) if I used sdist instead, running python setup.py install would
install in the right place, but I want to distribute pre-compiled C
extensions. What is the most appropriate way of doing this? a bdist
seems the most natural, but it seems to assume certain target
directories. I suppose one could do an sdist and include the compiled
extensions as "data files" but that seems like a hack against the
"spirit" of distutils.
4) there is a build_rpm command. Why isn't there a build_deb? and are
there any plans to add this?

Thanks.

-- 
Ernesto Posse

Applied Formal Methods Group - Software Technology Lab
School of Computing
Queen's University - Kingston, Ontario, Canada

From reinout at vanrees.org  Tue Apr 20 15:44:35 2010
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 20 Apr 2010 15:44:35 +0200
Subject: [Distutils] Passing user and password to easy_install
In-Reply-To: <z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>
References: <C452E142-574C-41F0-BFDD-76EB3171B8E0@ikanobori.jp>	<A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>
	<z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>
Message-ID: <hqkb43$h37$1@dough.gmane.org>

On 04/15/2010 06:15 PM, Brad Allen wrote:
> On Wed, Apr 14, 2010 at 7:59 AM, Simon de Vlieger<simon at ikanobori.jp>  wrote:
>> Luigi,
>>
>> I am not sure if using PyPi for private package distribution (what this
>> equates to) would be
>> a particularly nice idea. This way the PyPi would contain packages which
>> cannot be installed
>> by users of PyPi.
>>
>> Alas, I don't know of any ways distutils can fulfill your request in its
>> current form.
>
> Luigi, you might want to consider setting up a private package server
> instead of using PyPI. Where I work, we publish our eggs internally
> via an Apache server, so we are able to make use of Apache's security
> features (like https, authentication, and IP restriction). This works
> fine with setuptools and buildout, and will probably also work in
> future with the new 'distribute' tools.

See for an example
http://reinout.vanrees.org/weblog/2010/02/25/sdistmaker-your-own-pypi.html

Replace 'sdistmaker' there with whatever method you choose for putting 
your own eggs online.

There's an apache snippet in there that shows an easy way to use a 
private egg server together with the "real pypi".


Reinout



From ziade.tarek at gmail.com  Tue Apr 20 16:03:27 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 20 Apr 2010 16:03:27 +0200
Subject: [Distutils] Passing user and password to easy_install
In-Reply-To: <hqkb43$h37$1@dough.gmane.org>
References: <C452E142-574C-41F0-BFDD-76EB3171B8E0@ikanobori.jp>
	<A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>
	<z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>
	<hqkb43$h37$1@dough.gmane.org>
Message-ID: <q2n94bdd2611004200703ic03f041cn45ea397211350a09@mail.gmail.com>

On Tue, Apr 20, 2010 at 3:44 PM, Reinout van Rees <reinout at vanrees.org> wrote:
> On 04/15/2010 06:15 PM, Brad Allen wrote:
>>
>> On Wed, Apr 14, 2010 at 7:59 AM, Simon de Vlieger<simon at ikanobori.jp>
>> ?wrote:
>>>
>>> Luigi,
>>>
>>> I am not sure if using PyPi for private package distribution (what this
>>> equates to) would be
>>> a particularly nice idea. This way the PyPi would contain packages which
>>> cannot be installed
>>> by users of PyPi.
>>>
>>> Alas, I don't know of any ways distutils can fulfill your request in its
>>> current form.
>>
>> Luigi, you might want to consider setting up a private package server
>> instead of using PyPI. Where I work, we publish our eggs internally
>> via an Apache server, so we are able to make use of Apache's security
>> features (like https, authentication, and IP restriction). This works
>> fine with setuptools and buildout, and will probably also work in
>> future with the new 'distribute' tools.
>
> See for an example
> http://reinout.vanrees.org/weblog/2010/02/25/sdistmaker-your-own-pypi.html
>
> Replace 'sdistmaker' there with whatever method you choose for putting your
> own eggs online.
>
> There's an apache snippet in there that shows an easy way to use a private
> egg server together with the "real pypi".

A good addition to the hithicker's guide could be to list all the
options to build a private PyPI,


>
>
> Reinout
>
>
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | http://ziade.org

From jbaker at zeomega.com  Tue Apr 20 20:00:45 2010
From: jbaker at zeomega.com (Jason Baker)
Date: Tue, 20 Apr 2010 13:00:45 -0500
Subject: [Distutils] tag-build vs tag_build in egg_info
Message-ID: <r2mfa34f1941004201100ra8c484ecn681a5b6ef319da60@mail.gmail.com>

The way my project is set up right now, we have a setup.cfg that specifies
.dev for tag_build.  This way, we can have the file tagged properly when
developing from source.  We also have a Hudson CI server that will run the
tests and then generate an sdist with the build number added (using the
command "python setup.py egg_info -b dev-$BUILD_NUMBER sdist").

Now, suppose I use the following setup.cfg in my project:

[egg_info]
tag_build=.dev

If I run the following commands, everything works as expected:

$ python setup.py egg_info -b .dev-123
$ tar -xzvf dist/jiva_interface-3.0.1.dev-123.tar.gz
$ cat jiva_interface-3.0.1.dev-123/setup.cfg

[egg_info]
tag_build = .dev-123
tag_date = 0
tag_svn_revision = 0

This is the appropriate behavior:  I want the tag_build option to be
overridden in the resulting setup.cfg file.  However, I made a mistake and
used tag-build instead of tag_build:

[egg_info]
tag-build=.dev

If I run the same commands as above, the sdist is named the same, the
egg_info version is the same, but the setup.cfg is different:

$ python setup.py egg_info -b .dev-123
$ tar -xzvf dist/jiva_interface-3.0.1.dev-123.tar.gz
$ cat jiva_interface-3.0.1.dev-123/setup.cfg

[egg_info]
tag_build = .dev-123
tag_date = 0
tag_svn_revision = 0
tag-build = .dev

As you can see, it's creating both a tag-build and tag_build option.  And
it's going by the tag-build option, which isn't what I want.  Is this a bug,
or is there some kind of subtlety to this option that I don't know about?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100420/2f5cd10e/attachment.html>

From luigi.paioro at gmail.com  Fri Apr 16 10:20:32 2010
From: luigi.paioro at gmail.com (Luigi GMail)
Date: Fri, 16 Apr 2010 10:20:32 +0200
Subject: [Distutils] Passing user and password to easy_install
In-Reply-To: <v2h94bdd2611004150925n89ede347of1184186501b0114@mail.gmail.com>
References: <C452E142-574C-41F0-BFDD-76EB3171B8E0@ikanobori.jp>	
	<A3B0F92F-9211-49F8-ABB7-70E0D59926AF@ikanobori.jp>	
	<z2j4957f1ef1004150915l78900b8eye042d970da1f9901@mail.gmail.com>
	<v2h94bdd2611004150925n89ede347of1184186501b0114@mail.gmail.com>
Message-ID: <4BC81DD0.7080403@gmail.com>

>> Luigi, you might want to consider setting up a private package server
>> >  instead of using PyPI. Where I work, we publish our eggs internally
>> >  via an Apache server, so we are able to make use of Apache's security
>> >  features (like https, authentication, and IP restriction). This works
>> >  fine with setuptools and buildout, and will probably also work in
>> >  future with the new 'distribute' tools.
> +1
>
> You can also take a look at the alternative PyPI implementations,
> that implement the server-side parts of the upload and register command.
>
> I've changed Python 2.6's pypirc format so you can define in it
> several PyPI servers,
> and use the name of the target PyPI when you run register or upload
> (see the doc)
>
> I've backported it in a standalone project called "collective.dist" if
> you use python<  2.6
>
> For the server implementations, we have so far:
>
> - EggBasket
> - PloneSoftwareCenter (an extension for plone so you need to set up a plone)
> - chishop
> - ...
>
> regards
> Tarek

Yes, actually I was considering to build up a dedicated PyPI 
implementation. Knowing that someone else already implemented something, 
it encourages me to follow this road. Thank you, Tarek, to indicate me 
some projects that allow to implement a PyPI server! Otherwise I had to 
do everything alone.

Actually, as it frequently happens, solving a problem changes the problem:

A PyPI server is supposed to host binary distributions (as .rpm, .egg, 
etc.) as well as source distribution. In the latter case easy_install 
downloads the source distribution, unpack it, and creates an egg file 
which is then installed. Alas my packages are not suitable to be 
distributed as eggs since they are quite complicated and they include 
some ANSI-C stuff that must be compiled and installed as libraries, 
executables, etc. under /bin (where python is) and /lib (above 
/pythonx.y/site-packages). For this reason setup.py files are quite 
customized, making an heavy use of distutils.ccompiler. Therefore 
instead of making an egg file, easy_install should simply run `python 
setup.py install`. Do you know whether this is possible? Maybe using a 
different tool instead of easy_install? Otherwise I have to make it on 
my own...

Thanks in advance.

Luigi

From manlio_perillo at libero.it  Mon Apr 19 20:37:03 2010
From: manlio_perillo at libero.it (Manlio Perillo)
Date: Mon, 19 Apr 2010 20:37:03 +0200
Subject: [Distutils] namespace packages
Message-ID: <4BCCA2CF.2080600@libero.it>

Hi.

I would like to use support to namespace packages in setuptools, however
I have some doubts.

* will this feature be supported for future setup tools?
* is it efficient to use?
* any reason why one should not use it?


Thanks  Manlio

From mbaiju at zeomega.com  Wed Apr 21 14:07:58 2010
From: mbaiju at zeomega.com (Baiju M)
Date: Wed, 21 Apr 2010 17:37:58 +0530
Subject: [Distutils] namespace packages
In-Reply-To: <4BCCA2CF.2080600@libero.it>
References: <4BCCA2CF.2080600@libero.it>
Message-ID: <i2sc911c70d1004210507kef2a2a5bo694ea3dd76fc6709@mail.gmail.com>

On Tue, Apr 20, 2010 at 12:07 AM, Manlio Perillo
<manlio_perillo at libero.it> wrote:
> Hi.
>
> I would like to use support to namespace packages in setuptools, however
> I have some doubts.
>
> * will this feature be supported for future setup tools?
> * is it efficient to use?
> * any reason why one should not use it?

FYI, there is a draft PEP for this feature in Python 3.2:
http://www.python.org/dev/peps/pep-0382/

Regards,
Baiju M

From barry at python.org  Wed Apr 21 14:14:03 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 21 Apr 2010 08:14:03 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <i2sc911c70d1004210507kef2a2a5bo694ea3dd76fc6709@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<i2sc911c70d1004210507kef2a2a5bo694ea3dd76fc6709@mail.gmail.com>
Message-ID: <20100421081403.007fe51e@heresy>

On Apr 21, 2010, at 05:37 PM, Baiju M wrote:

>On Tue, Apr 20, 2010 at 12:07 AM, Manlio Perillo
><manlio_perillo at libero.it> wrote:
>> Hi.
>>
>> I would like to use support to namespace packages in setuptools, however
>> I have some doubts.
>>
>> * will this feature be supported for future setup tools?
>> * is it efficient to use?
>> * any reason why one should not use it?
>
>FYI, there is a draft PEP for this feature in Python 3.2:
>http://www.python.org/dev/peps/pep-0382/

I am planning on looking into this soon.  Please contact me if you're
interested in working on this PEP.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100421/2051c655/attachment.pgp>

From ziade.tarek at gmail.com  Wed Apr 21 14:37:40 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 21 Apr 2010 14:37:40 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <20100421081403.007fe51e@heresy>
References: <4BCCA2CF.2080600@libero.it>
	<i2sc911c70d1004210507kef2a2a5bo694ea3dd76fc6709@mail.gmail.com>
	<20100421081403.007fe51e@heresy>
Message-ID: <u2p94bdd2611004210537n2ebc3c28m327b2db2a6f9d496@mail.gmail.com>

On Wed, Apr 21, 2010 at 2:14 PM, Barry Warsaw <barry at python.org> wrote:
> On Apr 21, 2010, at 05:37 PM, Baiju M wrote:
>
>>On Tue, Apr 20, 2010 at 12:07 AM, Manlio Perillo
>><manlio_perillo at libero.it> wrote:
>>> Hi.
>>>
>>> I would like to use support to namespace packages in setuptools, however
>>> I have some doubts.
>>>
>>> * will this feature be supported for future setup tools?
>>> * is it efficient to use?
>>> * any reason why one should not use it?
>>
>>FYI, there is a draft PEP for this feature in Python 3.2:
>>http://www.python.org/dev/peps/pep-0382/
>
> I am planning on looking into this soon. ?Please contact me if you're
> interested in working on this PEP.

Last time we've mentioned this pep on Python-dev, Martin said he would
do its implementation.

Therefore, I am not sure what's the state on his side,.. cc'ing him

Regards,
Tarek
-- 
Tarek Ziad? | http://ziade.org

From strawman at astraw.com  Wed Apr 21 16:30:28 2010
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 21 Apr 2010 07:30:28 -0700
Subject: [Distutils] namespace packages
In-Reply-To: <4BCCA2CF.2080600@libero.it>
References: <4BCCA2CF.2080600@libero.it>
Message-ID: <4BCF0C04.5080101@astraw.com>

Manlio Perillo wrote:
> Hi.
>
> I would like to use support to namespace packages in setuptools, however
> I have some doubts.
>
> * will this feature be supported for future setup tools?
> * is it efficient to use?
>   
I have heard that it is slow if you're operating on NFS, as it causes
lots more filesystem operations than non-namespace packages. This is why
we're not considering it for matplotlib.

> * any reason why one should not use it?
>
>   
Apart from the above, my colleague has had trouble getting it to work
with py2exe. (But apparently bbfreeze does work with namespace packages.)

-Andrew

From pje at telecommunity.com  Wed Apr 21 18:13:17 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 21 Apr 2010 12:13:17 -0400
Subject: [Distutils] Passing user and password to easy_install
Message-ID: <20100421161315.2165D3A4070@sparrow.telecommunity.com>

At 10:20 AM 4/16/2010 +0200, Luigi GMail wrote:
>>  Therefore instead of making an egg file, easy_install should 
>> simply run `python setup.py install`. Do you know whether this is 
>> possible? Maybe using a different tool instead of easy_install?

The 'pip' tool does this.


From martin at v.loewis.de  Wed Apr 21 20:45:52 2010
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 21 Apr 2010 20:45:52 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <u2p94bdd2611004210537n2ebc3c28m327b2db2a6f9d496@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>	
	<i2sc911c70d1004210507kef2a2a5bo694ea3dd76fc6709@mail.gmail.com>	
	<20100421081403.007fe51e@heresy>
	<u2p94bdd2611004210537n2ebc3c28m327b2db2a6f9d496@mail.gmail.com>
Message-ID: <4BCF47E0.4040101@v.loewis.de>

> Last time we've mentioned this pep on Python-dev, Martin said he would
> do its implementation.
> 
> Therefore, I am not sure what's the state on his side,.. cc'ing him

Unfortunately, I haven't made any progress - I still *plan* to do it
before the 3.2 betas, though. Contributions are welcome, of course.

Regards,
Martin

From barry at python.org  Wed Apr 21 21:05:12 2010
From: barry at python.org (Barry Warsaw)
Date: Wed, 21 Apr 2010 15:05:12 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <4BCF47E0.4040101@v.loewis.de>
References: <4BCCA2CF.2080600@libero.it>
	<i2sc911c70d1004210507kef2a2a5bo694ea3dd76fc6709@mail.gmail.com>
	<20100421081403.007fe51e@heresy>
	<u2p94bdd2611004210537n2ebc3c28m327b2db2a6f9d496@mail.gmail.com>
	<4BCF47E0.4040101@v.loewis.de>
Message-ID: <20100421150512.2a821bc7@heresy>

On Apr 21, 2010, at 08:45 PM, Martin v. L?wis wrote:

>> Last time we've mentioned this pep on Python-dev, Martin said he would
>> do its implementation.
>> 
>> Therefore, I am not sure what's the state on his side,.. cc'ing him
>
>Unfortunately, I haven't made any progress - I still *plan* to do it
>before the 3.2 betas, though. Contributions are welcome, of course.

Cool.  Let's coordinate here.  Eric's expressed interest too and we live close
to each other so we may sprint on it one evening in the next several weeks.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100421/3a0eefca/attachment.pgp>

From regebro at gmail.com  Wed Apr 21 21:07:41 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 21 Apr 2010 21:07:41 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <4BCCA2CF.2080600@libero.it>
References: <4BCCA2CF.2080600@libero.it>
Message-ID: <v2s319e029f1004211207n4db2e679h50d29774e705c06c@mail.gmail.com>

On Mon, Apr 19, 2010 at 20:37, Manlio Perillo <manlio_perillo at libero.it> wrote:
> Hi.
>
> I would like to use support to namespace packages in setuptools, however
> I have some doubts.
>
> * will this feature be supported for future setup tools?

All of Zope and Plone uses it all the time. Trust us, it won't
disappear ever. :)

> * is it efficient to use?

Well, obviously it will slow down package import, but not very much
and you only import packages once.

> * any reason why one should not use it?

Not that I can think of.

-- 
Lennart Regebro: http://regebro.wordpress.com/
Python 3 Porting: http://python-incompatibility.googlecode.com/
+33 661 58 14 64

From manlio.perillo at gmail.com  Wed Apr 21 21:26:21 2010
From: manlio.perillo at gmail.com (Manlio Perillo)
Date: Wed, 21 Apr 2010 21:26:21 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <v2s319e029f1004211207n4db2e679h50d29774e705c06c@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<v2s319e029f1004211207n4db2e679h50d29774e705c06c@mail.gmail.com>
Message-ID: <4BCF515D.5010000@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lennart Regebro ha scritto:
> On Mon, Apr 19, 2010 at 20:37, Manlio Perillo <manlio_perillo at libero.it> wrote:
>> Hi.
>>
>> I would like to use support to namespace packages in setuptools, however
>> I have some doubts.
>>
>> * will this feature be supported for future setup tools?
> 
> All of Zope and Plone uses it all the time. Trust us, it won't
> disappear ever. :)
> 

Well, I'm quite positive that it will not disappear, too. :)

But I do not want to use a feature that it is here for compatiblity
only, in a new project.

> [...]


Thanks  Manlio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvPUV0ACgkQscQJ24LbaURTjACZASrp83J6RFoR3zYFksZC2Uxb
eM4An2RGtQDNB7uTYCzssWtx4uZSGbdo
=5fbQ
-----END PGP SIGNATURE-----

From pje at telecommunity.com  Thu Apr 22 00:02:39 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 21 Apr 2010 18:02:39 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <4BCF515D.5010000@gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<v2s319e029f1004211207n4db2e679h50d29774e705c06c@mail.gmail.com>
	<4BCF515D.5010000@gmail.com>
Message-ID: <20100421220238.EB1B13A4070@sparrow.telecommunity.com>

At 09:26 PM 4/21/2010 +0200, Manlio Perillo wrote:
>But I do not want to use a feature that it is here for compatiblity
>only, in a new project.

Python itself has supported namespace packages through a stdlib 
utility since Python 2.3, and special import mechanism support has 
been proposed for addition in 3.2.  Zope may have developed the idea, 
and setuptools made it more accessible for people to use, but it's 
been a standard Python feature all along.

The new support in 3.x will also allow setuptools and its clones to 
drop some of their hackier bits of implementation, and hopefully make 
the adoption of namespace packages even more widespread.  While the 
Zen of Python says that flat is better than nested (i.e., "zope.*" 
vs. "org.zope.*" as would be done in Java), it also says that 
namespaces are a great idea, so let's have more of them.  ;-)


From cournape at gmail.com  Thu Apr 22 03:18:32 2010
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 22 Apr 2010 10:18:32 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <4BCCA2CF.2080600@libero.it>
References: <4BCCA2CF.2080600@libero.it>
Message-ID: <r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>

On Tue, Apr 20, 2010 at 3:37 AM, Manlio Perillo
<manlio_perillo at libero.it> wrote:

>
> * will this feature be supported for future setup tools?
> * is it efficient to use?

It is slower than "conventional" packages import. I still don't
understand the whole implementation well, but I think there is an
inherent added cost (i.e. any namespace implementation will cost
something), even though the implementation could be better (setuptools
namespace implementation depends on pkg_resources, which is a
complicated and quite slow piece of code).

Whether it matters entirely depends on what you are doing.

> * any reason why one should not use it?

If you already depend on setuptools in your package, I would say there
is not much reason. One problem with the setuptools implementation is
that several packages sharing the same namespace have files in common,
which is often a pain to package generally for native packages (be it
linux packages and otherwise). Removing this limitation is one of the
stated goal of PEP 382.

cheers,

David

From pje at telecommunity.com  Thu Apr 22 06:10:50 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 22 Apr 2010 00:10:50 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail .com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
Message-ID: <20100422041050.C2F7F3A4070@sparrow.telecommunity.com>

At 10:18 AM 4/22/2010 +0900, David Cournapeau wrote:
>One problem with the setuptools implementation is
>that several packages sharing the same namespace have files in common,

If that were actually true (it isn't), then it would be considered a 
bug in setuptools.

When you build a package for system packaging (using install --root 
or any of the bdist_* tools that use install --root internally), 
setuptools specifically excludes __init__.py* files from 
installation, and replaces them with uniquely-named '-nspkg.pth' 
files, similar function to PEP 382's package marker files, except 
with more complicated innards.


>Removing this limitation is one of the stated goal of PEP 382.

It's not a setuptools limitation, though.  As I said above, 
setuptools goes to ridiculous lengths to work around the problem; PEP 
382 will merely remove the need for setuptools to do the complex 
work-around, allowing it to drop -nspkg.pth files in favor of the PEP 
382-specified files.


>(setuptools namespace implementation depends on pkg_resources, which is a
>complicated and quite slow piece of code).

Slow at doing what, precisely, and slower compared to what alternative?


From cournape at gmail.com  Thu Apr 22 09:36:51 2010
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 22 Apr 2010 16:36:51 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
Message-ID: <z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>

On Thu, Apr 22, 2010 at 1:10 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 10:18 AM 4/22/2010 +0900, David Cournapeau wrote:
>>
>> One problem with the setuptools implementation is
>> that several packages sharing the same namespace have files in common,
>
> If that were actually true (it isn't), then it would be considered a bug in
> setuptools.

I am confused, then, by the setuptools doc:

Note, by the way, that your project's source tree must include the
namespace packages' __init__.py files (and the __init__.py  of any
parent packages), in a normal Python package layout. These __init__.py
files must contain the line:

and because that's in the rationale of Pep 382. If this is not true,
we need to update our own instructions because we advise to write such
__init__.py in scikits namespace in the scipy community.

>> (setuptools namespace implementation depends on pkg_resources, which is a
>> complicated and quite slow piece of code).
>
> Slow at doing what, precisely, and slower compared to what alternative?

Merely importing it is already quite slow, and the question was about
whether to use namespace at all, so the alternative of not using it is
obviously faster. As I mentioned, it may not matter, but that's
certainly one of  reason why I refuse to use pkg_resources myself,

cheers,

David

From cournape at gmail.com  Thu Apr 22 09:49:52 2010
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 22 Apr 2010 16:49:52 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
Message-ID: <l2p5b8d13221004220049k36173f05xa7742df046eeff04@mail.gmail.com>

On Thu, Apr 22, 2010 at 1:10 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 10:18 AM 4/22/2010 +0900, David Cournapeau wrote:
>>
>> One problem with the setuptools implementation is
>> that several packages sharing the same namespace have files in common,
>
> If that were actually true (it isn't), then it would be considered a bug in
> setuptools.
>
> When you build a package for system packaging (using install --root or any
> of the bdist_* tools that use install --root internally), setuptools
> specifically excludes __init__.py* files from installation, and replaces
> them with uniquely-named '-nspkg.pth' files, similar function to PEP 382's
> package marker files, except with more complicated innards.

I did not know about the --root option interaction with namespace
package, but I don't really see how that solve the problem when root
is not an option (when using stow for example).

cheers,

David

From pje at telecommunity.com  Thu Apr 22 18:10:38 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 22 Apr 2010 12:10:38 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail .com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
Message-ID: <20100422161038.45B403A4070@sparrow.telecommunity.com>

At 04:36 PM 4/22/2010 +0900, David Cournapeau wrote:
>On Thu, Apr 22, 2010 at 1:10 PM, P.J. Eby <pje at telecommunity.com> wrote:
> > At 10:18 AM 4/22/2010 +0900, David Cournapeau wrote:
> >>
> >> One problem with the setuptools implementation is
> >> that several packages sharing the same namespace have files in common,
> >
> > If that were actually true (it isn't), then it would be considered a bug in
> > setuptools.
>
>I am confused, then, by the setuptools doc:
>
>Note, by the way, that your project's source tree must include the
>namespace packages' __init__.py files (and the __init__.py  of any
>parent packages), in a normal Python package layout. These __init__.py
>files must contain the line:

If you read two paragraphs further in that doc, you'll see where it 
goes on to say:

"You must NOT include any other code and data in a namespace 
package's __init__.py. Even though it may appear to work during 
development, or when projects are installed as .egg files, it will 
not work when the projects are installed using "system" packaging 
tools -- in such cases the __init__.py files **will not be 
installed**, let alone executed. "  (emphasis added)


>and because that's in the rationale of Pep 382. If this is not true,
>we need to update our own instructions because we advise to write such
>__init__.py in scikits namespace in the scipy community.

You still need to write them, for egg-based installs.  But when 
packaged for *system* installation, they aren't going to be there.


> >> (setuptools namespace implementation depends on pkg_resources, which is a
> >> complicated and quite slow piece of code).
> >
> > Slow at doing what, precisely, and slower compared to what alternative?
>
>Merely importing it is already quite slow,

I'd be interested in knowing under what conditions that's the case 
(e.g. how many .egg zipfiles vs. how many .egg subdirs); it might be 
possible to improve it.


From pje at telecommunity.com  Thu Apr 22 18:14:03 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 22 Apr 2010 12:14:03 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <l2p5b8d13221004220049k36173f05xa7742df046eeff04@mail.gmail .com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<l2p5b8d13221004220049k36173f05xa7742df046eeff04@mail.gmail.com>
Message-ID: <20100422161403.6BEC13A40F5@sparrow.telecommunity.com>

At 04:49 PM 4/22/2010 +0900, David Cournapeau wrote:
>On Thu, Apr 22, 2010 at 1:10 PM, P.J. Eby <pje at telecommunity.com> wrote:
> > At 10:18 AM 4/22/2010 +0900, David Cournapeau wrote:
> >>
> >> One problem with the setuptools implementation is
> >> that several packages sharing the same namespace have files in common,
> >
> > If that were actually true (it isn't), then it would be considered a bug in
> > setuptools.
> >
> > When you build a package for system packaging (using install --root or any
> > of the bdist_* tools that use install --root internally), setuptools
> > specifically excludes __init__.py* files from installation, and replaces
> > them with uniquely-named '-nspkg.pth' files, similar function to PEP 382's
> > package marker files, except with more complicated innards.
>
>I did not know about the --root option interaction with namespace
>package, but I don't really see how that solve the problem when root
>is not an option (when using stow for example).

"setup.py install --single-version-externally-managed" does the same 
thing.  I just mentioned --root because --root is commonly used by 
system packaging tools, and --root automatically sets the install to 
be --single-version-externally-managed.


From jim at zope.com  Thu Apr 22 20:54:41 2010
From: jim at zope.com (Jim Fulton)
Date: Thu, 22 Apr 2010 14:54:41 -0400
Subject: [Distutils] setuptools install --dry-run broken?
Message-ID: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>

In researching an old ZODB bug about it's setup.py breaking the
install --dry-run option, I tried using the option with the current
ZODB and bobo setup.py scripts, using Python 2.6.5.

In both cases, I didn't get an error and the modules were installed in
site-packages.  While the ZODB setup.py is complicated, bobo's isn't.

Both of these setup.py files used setuptools, so
I assumed this was a bug in setuptools. Then I tried the distutils
example:

  http://docs.python.org/distutils/introduction.html#a-simple-example

running

  python2.6 setup.py install --dry-run

(or python2.6 setup.py install -n)

installed the package.

Am I missunderstanding something? Or is --dry-run an april fool's
joke? :)

Jim

--
Jim Fulton

From pje at telecommunity.com  Thu Apr 22 21:50:41 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 22 Apr 2010 15:50:41 -0400
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail .com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
Message-ID: <20100422195042.3CE233A4070@sparrow.telecommunity.com>

At 02:54 PM 4/22/2010 -0400, Jim Fulton wrote:
>I tried the distutils example:
>
>   http://docs.python.org/distutils/introduction.html#a-simple-example
>
>running
>
>   python2.6 setup.py install --dry-run
>
>(or python2.6 setup.py install -n)
>
>installed the package.
>
>Am I missunderstanding something? Or is --dry-run an april fool's
>joke? :)

It appears the issue is that this has always been broken in 
distutils.  What you need to do is:

    python2.6 setup.py --dry-run install

(Or -n).

The problem is that the 'install' command doesn't pass on its local 
--dry-run flag to any of the subcommands that do the real work.

Probably, the --dry-run option should be dropped from the individual 
commands altogether in distutils2, with only a global flag being allowed.


From jim at zope.com  Thu Apr 22 23:11:16 2010
From: jim at zope.com (Jim Fulton)
Date: Thu, 22 Apr 2010 17:11:16 -0400
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <20100422195042.3CE233A4070@sparrow.telecommunity.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com> 
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
Message-ID: <p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>

On Thu, Apr 22, 2010 at 3:50 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 02:54 PM 4/22/2010 -0400, Jim Fulton wrote:
>>
>> I tried the distutils example:
>>
>> ?http://docs.python.org/distutils/introduction.html#a-simple-example
>>
>> running
>>
>> ?python2.6 setup.py install --dry-run
>>
>> (or python2.6 setup.py install -n)
>>
>> installed the package.
>>
>> Am I missunderstanding something? Or is --dry-run an april fool's
>> joke? :)
>
> It appears the issue is that this has always been broken in distutils. ?What
> you need to do is:
>
> ? python2.6 setup.py --dry-run install
>
> (Or -n).
>
> The problem is that the 'install' command doesn't pass on its local
> --dry-run flag to any of the subcommands that do the real work.
>
> Probably, the --dry-run option should be dropped from the individual
> commands altogether in distutils2, with only a global flag being allowed.

That sounds good, although

    python2.6 setup.py --dry-run install

doesn't work either. With the "simple example":

  python2.6 setup.py --dry-run install
  running install
  running build
  running build_py
  running install_lib
  copying build/lib/foo.py -> /usr/local/python/2.6/lib/python2.6/site-packages
  error: file '/usr/local/python/2.6/lib/python2.6/site-packages/foo.py'
does not exist

Or maybe this constitutes "working" for distutils. ;)

Jim

-- 
Jim Fulton

From ziade.tarek at gmail.com  Thu Apr 22 23:19:16 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 22 Apr 2010 23:19:16 +0200
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
Message-ID: <t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>

On Thu, Apr 22, 2010 at 11:11 PM, Jim Fulton <jim at zope.com> wrote:
[..]
> That sounds good, although
>
> ? ?python2.6 setup.py --dry-run install
>
> doesn't work either. With the "simple example":
>
> ?python2.6 setup.py --dry-run install
> ?running install
> ?running build
> ?running build_py
> ?running install_lib
> ?copying build/lib/foo.py -> /usr/local/python/2.6/lib/python2.6/site-packages
> ?error: file '/usr/local/python/2.6/lib/python2.6/site-packages/foo.py'
> does not exist

I confirm this is not working (even under 2.5) -- please fill a bug in
bugs.python.org, I'll fix it.

> Or maybe this constitutes "working" for distutils. ;)

well, why do you want to do a dry run anyways ? make up your mind: do
you want it installed or not ? ;)

-- 
Tarek Ziad? | http://ziade.org

From jim at zope.com  Thu Apr 22 23:40:45 2010
From: jim at zope.com (Jim Fulton)
Date: Thu, 22 Apr 2010 17:40:45 -0400
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com> 
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
Message-ID: <n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>

On Thu, Apr 22, 2010 at 5:19 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Apr 22, 2010 at 11:11 PM, Jim Fulton <jim at zope.com> wrote:
> [..]
>> That sounds good, although
>>
>> ? ?python2.6 setup.py --dry-run install
>>
>> doesn't work either. With the "simple example":
>>
>> ?python2.6 setup.py --dry-run install
>> ?running install
>> ?running build
>> ?running build_py
>> ?running install_lib
>> ?copying build/lib/foo.py -> /usr/local/python/2.6/lib/python2.6/site-packages
>> ?error: file '/usr/local/python/2.6/lib/python2.6/site-packages/foo.py'
>> does not exist
>
> I confirm this is not working (even under 2.5) -- please fill a bug in
> bugs.python.org, I'll fix it.

http://bugs.python.org/issue8501

>
>> Or maybe this constitutes "working" for distutils. ;)
>
> well, why do you want to do a dry run anyways ?

I don't. I was researching an old ZODB issue in which someone
complained that ZODB's setup.py broke --dry-run. :)

> make up your mind: do
> you want it installed or not ? ;)

The documentation says that when --dry-run is used: "don't actually do
anything".  This is not the actual behavior.  I'd be happy to see the
option go away. I don't think anyone us depending on it and you have
better things to do.

Jim

--
Jim Fulton

From ziade.tarek at gmail.com  Thu Apr 22 23:46:45 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 22 Apr 2010 23:46:45 +0200
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
	<n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
Message-ID: <o2i94bdd2611004221446r2b1476fex665b13d67848caf5@mail.gmail.com>

On Thu, Apr 22, 2010 at 11:40 PM, Jim Fulton <jim at zope.com> wrote:
[..]
>> I confirm this is not working (even under 2.5) -- please fill a bug in
>> bugs.python.org, I'll fix it.
>
> http://bugs.python.org/issue8501
>

Thx

>>
>>> Or maybe this constitutes "working" for distutils. ;)
>>
>> well, why do you want to do a dry run anyways ?
>
> I don't. I was researching an old ZODB issue in which someone
> complained that ZODB's setup.py broke --dry-run. :)
>
>> make up your mind: do
>> you want it installed or not ? ;)
>
> The documentation says that when --dry-run is used: "don't actually do
> anything". ?This is not the actual behavior. ?I'd be happy to see the
> option go away. I don't think anyone us depending on it and you have
> better things to do.

Sounds reasonable, I never liked it anyways.

In any case, distutils is frozen, so I'll see if I can fix it in 2.6
onward, and remove the option for distutils2.

Tarek


-- 
Tarek Ziad? | http://ziade.org

From floris.bruynooghe at gmail.com  Fri Apr 23 00:19:49 2010
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 22 Apr 2010 23:19:49 +0100
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <o2i94bdd2611004221446r2b1476fex665b13d67848caf5@mail.gmail.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
	<n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
	<o2i94bdd2611004221446r2b1476fex665b13d67848caf5@mail.gmail.com>
Message-ID: <20100422221949.GA22445@laurie.devork>

On Thu, Apr 22, 2010 at 11:46:45PM +0200, Tarek Ziad? wrote:
> On Thu, Apr 22, 2010 at 11:40 PM, Jim Fulton <jim at zope.com> wrote:
> [..]
> >> I confirm this is not working (even under 2.5) -- please fill a bug in
> >> bugs.python.org, I'll fix it.
> >
> > http://bugs.python.org/issue8501
> >
> 
> Thx
> 
> >>
> >>> Or maybe this constitutes "working" for distutils. ;)
> >>
> >> well, why do you want to do a dry run anyways ?
> >
> > I don't. I was researching an old ZODB issue in which someone
> > complained that ZODB's setup.py broke --dry-run. :)
> >
> >> make up your mind: do
> >> you want it installed or not ? ;)
> >
> > The documentation says that when --dry-run is used: "don't actually do
> > anything". ?This is not the actual behavior. ?I'd be happy to see the
> > option go away. I don't think anyone us depending on it and you have
> > better things to do.
> 
> Sounds reasonable, I never liked it anyways.

I would think it's intended for paranoid people who want to see what
exactly what files will be copied where before doing a real install.
And I must admit that I like that functionality at times.

That said it's been broken since forever[0] and it's a real pain to
implement it properly and a high maintenance cost to keep it
functioning properly.  So maybe removing it is the sensible option.

Regards
Floris

[0] I have vague memories from years ago when I first played with
    distutils to use this and discover it still did some stuff and was
    useless in showing what actually would happen.  At that time I
    hadn't discovered bug trackers yet, in my defence I probably
    didn't have internet either.

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From greg.ewing at canterbury.ac.nz  Fri Apr 23 02:43:03 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 23 Apr 2010 12:43:03 +1200
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
	<n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
Message-ID: <4BD0ED17.2000403@canterbury.ac.nz>

Jim Fulton wrote:

> The documentation says that when --dry-run is used: "don't actually do
> anything".  This is not the actual behavior.  I'd be happy to see the
> option go away.

I wouldn't! I find it useful for testing a setup.py that
I'm about to distribute, to make sure it isn't going to
fall over due to some stupid bug as soon as someone tries
to use it.

-1 on removing this feature, +1 on fixing it.

--
Greg

From cournape at gmail.com  Fri Apr 23 03:16:54 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 23 Apr 2010 10:16:54 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <20100422161038.45B403A4070@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
Message-ID: <h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>

On Fri, Apr 23, 2010 at 1:10 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 04:36 PM 4/22/2010 +0900, David Cournapeau wrote:
>>
>> On Thu, Apr 22, 2010 at 1:10 PM, P.J. Eby <pje at telecommunity.com> wrote:
>> > At 10:18 AM 4/22/2010 +0900, David Cournapeau wrote:
>> >>
>> >> One problem with the setuptools implementation is
>> >> that several packages sharing the same namespace have files in common,
>> >
>> > If that were actually true (it isn't), then it would be considered a bug
>> > in
>> > setuptools.
>>
>> I am confused, then, by the setuptools doc:
>>
>> Note, by the way, that your project's source tree must include the
>> namespace packages' __init__.py files (and the __init__.py ?of any
>> parent packages), in a normal Python package layout. These __init__.py
>> files must contain the line:
>
> If you read two paragraphs further in that doc, you'll see where it goes on
> to say:
>
> "You must NOT include any other code and data in a namespace package's
> __init__.py. Even though it may appear to work during development, or when
> projects are installed as .egg files, it will not work when the projects are
> installed using "system" packaging tools -- in such cases the __init__.py
> files **will not be installed**, let alone executed. " ?(emphasis added)

Hm, you are right, the __init__.py is not installed for a trivial
namespace package with the right options. This is strange, because I
always install with --single-version-externally-managed, and always
had issues with shared __init__.py (which has only that one line to
declare the namespace). And I still see this behavior for the
namespace packages I am using, so I need to look further to understand
what cause this discrepancy.

> I'd be interested in knowing under what conditions that's the case (e.g. how
> many .egg zipfiles vs. how many .egg subdirs); it might be possible to
> improve it.

In my case, it is not even the issue of many eggs (I always install
things with --single-version-externally-managed and I forbid any code
to write into  easy_install.pth). Importing pkg_resources alone
(python -c "import pkg_resources") takes half a second on my netbook.
As a reference, importing pkg_resources is slower than importing
numpy.

cheers,

David

From ben+python at benfinney.id.au  Fri Apr 23 05:11:00 2010
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 23 Apr 2010 13:11:00 +1000
Subject: [Distutils] install --dry-run broken?
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
	<n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
	<4BD0ED17.2000403@canterbury.ac.nz>
Message-ID: <87eii6lv7f.fsf@benfinney.id.au>

Greg Ewing <greg.ewing at canterbury.ac.nz> writes:

> Jim Fulton wrote:
> > The documentation says that when --dry-run is used: "don't actually
> > do anything". This is not the actual behavior. I'd be happy to see
> > the option go away.
>
> I wouldn't! I find it useful for testing a setup.py that I'm about to
> distribute, to make sure it isn't going to fall over due to some
> stupid bug as soon as someone tries to use it.

Are you saying that the feature does work as described? If not, what
does it do that you find useful?

-- 
 \          ?Our products just aren't engineered for security.? ?Brian |
  `\             Valentine, senior vice-president of Microsoft Windows |
_o__)                                                      development |
Ben Finney


From cournape at gmail.com  Fri Apr 23 05:41:59 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 23 Apr 2010 12:41:59 +0900
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <20100422221949.GA22445@laurie.devork>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
	<n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
	<o2i94bdd2611004221446r2b1476fex665b13d67848caf5@mail.gmail.com>
	<20100422221949.GA22445@laurie.devork>
Message-ID: <h2v5b8d13221004222041gdc4ebd02vaf701a64a85c4031@mail.gmail.com>

On Fri, Apr 23, 2010 at 7:19 AM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:

>
> I would think it's intended for paranoid people who want to see what
> exactly what files will be copied where before doing a real install.
> And I must admit that I like that functionality at times.

The feature is very useful indeed, but the current implementation is
useless in distutils. Maybe it would be better to use something like
the FileList instance which is kept in the corresponding commands, and
parse this one.

> That said it's been broken since forever[0] and it's a real pain to
> implement it properly and a high maintenance cost to keep it
> functioning properly.

It is a pain in distutils, but that's easy to do it if you design the
install process correctly.

David

From greg.ewing at canterbury.ac.nz  Fri Apr 23 06:03:23 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 23 Apr 2010 16:03:23 +1200
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <87eii6lv7f.fsf@benfinney.id.au>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<t2m94bdd2611004221419m87e9c9b5ge862e7023bdfaba5@mail.gmail.com>
	<n2u1099b90b1004221440od3318090i7c1ca4975337e1b2@mail.gmail.com>
	<4BD0ED17.2000403@canterbury.ac.nz> <87eii6lv7f.fsf@benfinney.id.au>
Message-ID: <4BD11C0B.4060705@canterbury.ac.nz>

On 23/04/10 15:11, Ben Finney wrote:

> Are you saying that the feature does work as described?

All I can say is that I haven't noticed that it doesn't.
I've never actually bothered to check, because it never
occurred to me before that it might be broken. I'll
look into it and report back.

-- 
Greg

From pje at telecommunity.com  Fri Apr 23 06:58:14 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 00:58:14 -0400
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail .com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
Message-ID: <20100423045815.930CA3A4070@sparrow.telecommunity.com>

At 05:11 PM 4/22/2010 -0400, Jim Fulton wrote:
>Or maybe this constitutes "working" for distutils. ;)

It does, I'm afraid.  The problem is that the build is also a dry 
run, so the files aren't there for the later bits to actually copy.

It might be better to do a dry run mechanism based mainly on 
setuptools' sandboxing code -- the sandbox framework could be 
extended to allow faking all file operations that go through normal 
Python APIs.  (i.e. not external commands like compilers; those would 
of course fail unless separately faked.)

There are actually quite a lot of applications for such sandboxing 
within the distutils, that at one time I hoped to explore in later 
setuptools versions.  (e.g. for testing, uninstalls of arbitrary 
packages, etc.)


From pje at telecommunity.com  Fri Apr 23 07:03:25 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 01:03:25 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail .com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
Message-ID: <20100423050326.4AF373A4070@sparrow.telecommunity.com>

At 10:16 AM 4/23/2010 +0900, David Cournapeau wrote:
>In my case, it is not even the issue of many eggs (I always install
>things with --single-version-externally-managed and I forbid any code
>to write into  easy_install.pth). Importing pkg_resources alone
>(python -c "import pkg_resources") takes half a second on my netbook.

I find that weird, to say the least.  On my desktop just now, with a 
sys.path 79 entries long (including 41 .eggs), it's a "blink and you 
missed it" operation.  I'm curious what the difference might be.

(Running timeit -s 'import pkg_resources' 'reload(pkg_resources)' 
gives a timing result of 61.9 milliseconds for me.)


From ziade.tarek at gmail.com  Fri Apr 23 09:03:32 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 23 Apr 2010 09:03:32 +0200
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <20100423045815.930CA3A4070@sparrow.telecommunity.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<20100423045815.930CA3A4070@sparrow.telecommunity.com>
Message-ID: <s2y94bdd2611004230003gb1fb39ecw4d6c8afeb4edb72d@mail.gmail.com>

On Fri, Apr 23, 2010 at 6:58 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 05:11 PM 4/22/2010 -0400, Jim Fulton wrote:
>>
>> Or maybe this constitutes "working" for distutils. ;)
>
> It does, I'm afraid. ?The problem is that the build is also a dry run, so
> the files aren't there for the later bits to actually copy.
>
> It might be better to do a dry run mechanism based mainly on setuptools'
> sandboxing code -- the sandbox framework could be extended to allow faking
> all file operations that go through normal Python APIs. ?(i.e. not external
> commands like compilers; those would of course fail unless separately
> faked.)

Right, I think sandboxing would be a better solution, less intrusive
to the code,
and this would work for *any* distutils command out there.

Now my question is about setuptools' sandbox :  it seems to me that it was only
making sure a setup script wasn't trynig to write outside a given
directory, using DirectorySandbox.

1- How this tool could be used to record the writings ? Would we need
to subtype AbstractSandbox ?
2- If  1/ is doable, what about making this tool its own standalone
project ? that would be useful for many projects

Regards
Tarek
-- 
Tarek Ziad? | http://ziade.org

From cournape at gmail.com  Fri Apr 23 09:23:42 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 23 Apr 2010 16:23:42 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <20100423050326.4AF373A4070@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
Message-ID: <y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>

On Fri, Apr 23, 2010 at 2:03 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 10:16 AM 4/23/2010 +0900, David Cournapeau wrote:
>>
>> In my case, it is not even the issue of many eggs (I always install
>> things with --single-version-externally-managed and I forbid any code
>> to write into ?easy_install.pth). Importing pkg_resources alone
>> (python -c "import pkg_resources") takes half a second on my netbook.
>
> I find that weird, to say the least. ?On my desktop just now, with a
> sys.path 79 entries long (including 41 .eggs), it's a "blink and you missed
> it" operation. ?I'm curious what the difference might be.
>
> (Running timeit -s 'import pkg_resources' 'reload(pkg_resources)' gives a
> timing result of 61.9 milliseconds for me.)

I should re-emphasize that the half-second number was on a netbook,
which is a very weak machine on every account (CPU, memory size and
disk capabilities). But using pkg_resources for console_scripts in the
package I am working on made a big difference (more time in spent in
importing pkg_resources than everything else). Since we are talking
about import times, I guess the issue is the same as for namespace
packages. I have noticed this slow behavior on every machine I have
ever had my hands on, be it mine or someone else, on linux, windows or
mac os x.

My (limited) understanding of pkg_resources is that is that it scales
linearly with the number of packages it is aware of, and that it needs
to scan a few directories for every package. Importing pkg_resources
causes many more syscalls than relatively big packages (~ 1000 for
python -c "", 3000 for importing one of numpy/wx/gtk, 6000 for
pkg_resources). Assuming those are unavoidable (and the current
namespace implementation in setuptools requires it, right ?), I don't
see a way to reduce that cost significantly,

cheers,

David

From ziade.tarek at gmail.com  Fri Apr 23 09:51:20 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 23 Apr 2010 09:51:20 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
Message-ID: <i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>

On Fri, Apr 23, 2010 at 9:23 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Apr 23, 2010 at 2:03 PM, P.J. Eby <pje at telecommunity.com> wrote:
>> At 10:16 AM 4/23/2010 +0900, David Cournapeau wrote:
>>>
>>> In my case, it is not even the issue of many eggs (I always install
>>> things with --single-version-externally-managed and I forbid any code
>>> to write into ?easy_install.pth). Importing pkg_resources alone
>>> (python -c "import pkg_resources") takes half a second on my netbook.
>>
>> I find that weird, to say the least. ?On my desktop just now, with a
>> sys.path 79 entries long (including 41 .eggs), it's a "blink and you missed
>> it" operation. ?I'm curious what the difference might be.
>>
>> (Running timeit -s 'import pkg_resources' 'reload(pkg_resources)' gives a
>> timing result of 61.9 milliseconds for me.)
>
> I should re-emphasize that the half-second number was on a netbook,
> which is a very weak machine on every account (CPU, memory size and
> disk capabilities). But using pkg_resources for console_scripts in the
> package I am working on made a big difference (more time in spent in
> importing pkg_resources than everything else). Since we are talking
> about import times, I guess the issue is the same as for namespace
> packages. I have noticed this slow behavior on every machine I have
> ever had my hands on, be it mine or someone else, on linux, windows or
> mac os x.
>
> My (limited) understanding of pkg_resources is that is that it scales
> linearly with the number of packages it is aware of, and that it needs
> to scan a few directories for every package. Importing pkg_resources
> causes many more syscalls than relatively big packages (~ 1000 for
> python -c "", 3000 for importing one of numpy/wx/gtk, 6000 for
> pkg_resources). Assuming those are unavoidable (and the current
> namespace implementation in setuptools requires it, right ?), I don't
> see a way to reduce that cost significantly,

There's a memory cache though, that probably makes it faster already.

Now if we had a way to know that a directory tree hasn't changed on
the system, a
persistent cache will dramatically increase the work. Unfortunately I think
this is impossible unless we watch them (and yet,  this would be quite
hard to implement).

We can probably have a persistent cache for zip files though, because we can
avoid to brows their content again if the zip file wasn't changed.

For regular directories, I haven't profiled it, but the bottleneck is
probably find_on_path(), the function that gets called for every
directory in sys.path to look for .eggs.

Now since the code mostly deals with strings work besides the I/O,
maybe it could be reimplemented in C.

I'd be very interested in speeding up this process, as we will have
something similar in pkg_util once PEP 376 is accepted,

Tarek

>
> cheers,
>
> David
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | http://ziade.org

From cournape at gmail.com  Fri Apr 23 10:30:50 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 23 Apr 2010 17:30:50 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
Message-ID: <m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>

On Fri, Apr 23, 2010 at 4:51 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Fri, Apr 23, 2010 at 9:23 AM, David Cournapeau <cournape at gmail.com> wrote:
>> On Fri, Apr 23, 2010 at 2:03 PM, P.J. Eby <pje at telecommunity.com> wrote:
>>> At 10:16 AM 4/23/2010 +0900, David Cournapeau wrote:
>>>>
>>>> In my case, it is not even the issue of many eggs (I always install
>>>> things with --single-version-externally-managed and I forbid any code
>>>> to write into ?easy_install.pth). Importing pkg_resources alone
>>>> (python -c "import pkg_resources") takes half a second on my netbook.
>>>
>>> I find that weird, to say the least. ?On my desktop just now, with a
>>> sys.path 79 entries long (including 41 .eggs), it's a "blink and you missed
>>> it" operation. ?I'm curious what the difference might be.
>>>
>>> (Running timeit -s 'import pkg_resources' 'reload(pkg_resources)' gives a
>>> timing result of 61.9 milliseconds for me.)
>>
>> I should re-emphasize that the half-second number was on a netbook,
>> which is a very weak machine on every account (CPU, memory size and
>> disk capabilities). But using pkg_resources for console_scripts in the
>> package I am working on made a big difference (more time in spent in
>> importing pkg_resources than everything else). Since we are talking
>> about import times, I guess the issue is the same as for namespace
>> packages. I have noticed this slow behavior on every machine I have
>> ever had my hands on, be it mine or someone else, on linux, windows or
>> mac os x.
>>
>> My (limited) understanding of pkg_resources is that is that it scales
>> linearly with the number of packages it is aware of, and that it needs
>> to scan a few directories for every package. Importing pkg_resources
>> causes many more syscalls than relatively big packages (~ 1000 for
>> python -c "", 3000 for importing one of numpy/wx/gtk, 6000 for
>> pkg_resources). Assuming those are unavoidable (and the current
>> namespace implementation in setuptools requires it, right ?), I don't
>> see a way to reduce that cost significantly,
>
> There's a memory cache though, that probably makes it faster already.

Sure - I should have mentioned all those approximate numbers are given
for the hot cache. In my experience, unless some package do
CPU-intensive stuff at import time, the cost is mostly a function of
syscall numbers (the correlation between syscall and import time is
pretty strong) and regex. The profile_import tool (available from the
bzr project) is quite useful to detect those cases.

> Now if we had a way to know that a directory tree hasn't changed on
> the system, a
> persistent cache will dramatically increase the work. Unfortunately I think
> this is impossible unless we watch them (and yet, ?this would be quite
> hard to implement).

This all sounds complicated. pkg_resources is already complicated
enough (the other big reason why I don't use it in any of my
packages). Without a clear specification of what pkg_resources or its
successor is doing, caching and C-based implementation sound like
premature optimization to me. For example, pkg_resources does a lot of
things which seem quite orthogonal to me. While scanning every package
may be needed for namespace package support the "setuptools" way, this
is almost never useful for locating resources at runtime, but you pay
the price in both cases.

> For regular directories, I haven't profiled it, but the bottleneck is
> probably find_on_path(), the function that gets called for every
> directory in sys.path to look for .eggs.
>
> Now since the code mostly deals with strings work besides the I/O,
> maybe it could be reimplemented in C.

cheers,

David

From ziade.tarek at gmail.com  Fri Apr 23 10:54:59 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 23 Apr 2010 10:54:59 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
	<m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
Message-ID: <s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>

On Fri, Apr 23, 2010 at 10:30 AM, David Cournapeau <cournape at gmail.com> wrote:
[..]
> This all sounds complicated. pkg_resources is already complicated
> enough (the other big reason why I don't use it in any of my
> packages). Without a clear specification of what pkg_resources or its
> successor is doing, caching and C-based implementation sound like
> premature optimization to me. For example, pkg_resources does a lot of
> things which seem quite orthogonal to me. While scanning every package
> may be needed for namespace package support the "setuptools" way, this
> is almost never useful for locating resources at runtime, but you pay
> the price in both cases.

I am not sure what you are defining as "complicated". While pkg_resources
is hard to read and it's a project on its own with many other
features, the use case
we are talking about here is dead simple:

  scan all sys.path entries to look for .egg and .egg-info files/directories.

It's setuptools' "live" installed packages database index. It has to
be recalculated
at every run because sys.path and the directories it points may change.

That's the basic feature it needs for a broad range of features, and
pkg_util will need
it as well in a near future in the stdlib. It's the de-facto way to
find out what's installed (see PEP 376)

We have an implementation
(http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py) that is being
moved
to distutils2, and that will probably land in the stdlib if PEP 376 is accepted.

Now, about the C part, what I am saying is that I don't know if we
will be able to reduce
the numbers of I/O to read the content of directories if we use C, but
I do know that string comparisons
will be faster in C.

Hey, I barely did any C since college, I am going to give it a shot
and see if it goes faster :)
I am just scared that this can be quite complex when I take a look at
posixpath.c.. any C expert here that want to help ?

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From cournape at gmail.com  Fri Apr 23 12:01:15 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 23 Apr 2010 19:01:15 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
	<m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
	<s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>
Message-ID: <h2i5b8d13221004230301x9df501f7sbaa1f83d38f60d2f@mail.gmail.com>

On Fri, Apr 23, 2010 at 5:54 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> I am not sure what you are defining as "complicated". While pkg_resources
> is hard to read and it's a project on its own with many other
> features, the use case
> we are talking about here is dead simple:
>
> ?scan all sys.path entries to look for .egg and .egg-info files/directories.

My knowledge may be lacking here, but doen't pkg_resources need to
scan things beyond egg-info (to get namespace_package.txt, presumably)
?

Scanning egg/egg-info is easy, but that does not explain most
additional syscalls caused by pkg_resources import.

> We have an implementation
> (http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py) that is being
> moved
> to distutils2, and that will probably land in the stdlib if PEP 376 is accepted.

pkgutil.py implements everything needed for setuptools namespace
package ? From reading the discussions around PEP 382, it seemed much
more complicated.

> Hey, I barely did any C since college, I am going to give it a shot
> and see if it goes faster :)

If pkgutils.py is indeed all that is needed to support setuptools
namespace package, I am willing to look at the code to see if it can
be sped up. I really doubt that C should be used - if it takes so much
time for a couple of dozens packages, it is much more likely a design
problem than an implementation problem.

cheers,

David

From ziade.tarek at gmail.com  Fri Apr 23 12:16:11 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 23 Apr 2010 12:16:11 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <h2i5b8d13221004230301x9df501f7sbaa1f83d38f60d2f@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
	<m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
	<s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>
	<h2i5b8d13221004230301x9df501f7sbaa1f83d38f60d2f@mail.gmail.com>
Message-ID: <r2s94bdd2611004230316hc8c8a6c5v681cb209ccc1e110@mail.gmail.com>

On Fri, Apr 23, 2010 at 12:01 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Apr 23, 2010 at 5:54 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>> I am not sure what you are defining as "complicated". While pkg_resources
>> is hard to read and it's a project on its own with many other
>> features, the use case
>> we are talking about here is dead simple:
>>
>> ?scan all sys.path entries to look for .egg and .egg-info files/directories.
>
> My knowledge may be lacking here, but doen't pkg_resources need to
> scan things beyond egg-info (to get namespace_package.txt, presumably)
> ?

That's a file located in egg-info, it reads.

>
> Scanning egg/egg-info is easy, but that does not explain most
> additional syscalls caused by pkg_resources import.

Well it scans directories and open files so you have roughly  (N * F) + P calls
where N is the number of packages, F the average files open per package,
and P the number of entries in sys.path

>
>> We have an implementation
>> (http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py) that is being
>> moved
>> to distutils2, and that will probably land in the stdlib if PEP 376 is accepted.
>
> pkgutil.py implements everything needed for setuptools namespace
> package ? From reading the discussions around PEP 382, it seemed much
> more complicated.

No sorry, I am not talking about the namespace packages in particular anymore,
I am talking about the low-level feature of scanning the metadata of
projects found is sys.path.

That's the basis of all stuff.

>
>> Hey, I barely did any C since college, I am going to give it a shot
>> and see if it goes faster :)
>
> If pkgutils.py is indeed all that is needed to support setuptools
> namespace package, I am willing to look at the code to see if it can
> be sped up.

For any feature that needs to scan the metadata of installed packages,
unless there's a central database, we will have to loop over directories.

Now if we consider that everything loaded in sys.path has to be scaned,
you can't have a central database, thus you need to read the dirs.

> I really doubt that C should be used - if it takes so much
> time for a couple of dozens packages, it is much more likely a design
> problem than an implementation problem.

What's "so much time" ? That's pretty vague. Again, knowing the code,
all it does is I/O and string comparisons, so my guess is that C would
help here.

But guessing is not the right thing to do for optimization, we need facts.
So if you come back with some profiling information on your use case
where it seems so slow,
we can see who is causing this and have a much more efficient
discussion I guess :)

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org

From cournape at gmail.com  Fri Apr 23 13:38:13 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 23 Apr 2010 20:38:13 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <r2s94bdd2611004230316hc8c8a6c5v681cb209ccc1e110@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
	<m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
	<s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>
	<h2i5b8d13221004230301x9df501f7sbaa1f83d38f60d2f@mail.gmail.com>
	<r2s94bdd2611004230316hc8c8a6c5v681cb209ccc1e110@mail.gmail.com>
Message-ID: <l2i5b8d13221004230438m32b469f5hde126f7471438838@mail.gmail.com>

On Fri, Apr 23, 2010 at 7:16 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> But guessing is not the right thing to do for optimization, we need facts.
> So if you come back with some profiling information on your use case
> where it seems so slow

Here is what cProfile -s cum gives me for a script which only import
pkg_resources:

      42208 function calls (41959 primitive calls) in 0.421 CPU seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.421    0.421 <string>:1(<module>)
        1    0.000    0.000    0.421    0.421 {execfile}
        1    0.003    0.003    0.421    0.421 yo.py:1(<module>)
        1    0.008    0.008    0.418    0.418 pkg_resources.py:14(<module>)
        1    0.001    0.001    0.232    0.232 pkg_resources.py:673(subscribe)
       93    0.001    0.000    0.231    0.002 pkg_resources.py:2623(<lambda>)
       93    0.003    0.000    0.230    0.002 pkg_resources.py:2166(activate)
      257    0.005    0.000    0.192    0.001 pkg_resources.py:1743(_handle_ns)
      253    0.008    0.000    0.160    0.001 pkgutil.py:176(find_module)
      275    0.024    0.000    0.151    0.001 posixpath.py:351(realpath)
       36    0.003    0.000    0.138    0.004 pkg_resources.py:428(add_entry)
       94    0.002    0.000    0.137    0.001 {map}
       93    0.003    0.000    0.133    0.001
pkg_resources.py:1795(fixup_namespace_packages)
      230    0.022    0.000    0.110    0.000
pkg_resources.py:1686(find_on_path)
        1    0.000    0.000    0.078    0.078 pkg_resources.py:414(__init__)
    30/23    0.001    0.000    0.064    0.003
pkg_resources.py:1764(declare_namespace)
     1518    0.036    0.000    0.059    0.000 posixpath.py:59(join)
      196    0.008    0.000    0.044    0.000
pkg_resources.py:2068(from_location)
     1206    0.014    0.000    0.043    0.000 posixpath.py:129(islink)
      402    0.004    0.000    0.042    0.000 re.py:229(_compile)
       13    0.000    0.000    0.036    0.003 sre_compile.py:501(compile)
       12    0.000    0.000    0.035    0.003 re.py:188(compile)
      196    0.006    0.000    0.027    0.000 pkg_resources.py:2054(__init__)
      275    0.003    0.000    0.026    0.000 posixpath.py:337(abspath)
      275    0.012    0.000    0.020    0.000 posixpath.py:308(normpath)
     1738    0.005    0.000    0.020    0.000
pkg_resources.py:1831(_normalize_cached)
      196    0.008    0.000    0.019    0.000 pkg_resources.py:506(add)
       13    0.000    0.000    0.019    0.001 sre_parse.py:669(parse)
     1206    0.019    0.000    0.019    0.000 {posix.lstat}
    36/13    0.001    0.000    0.018    0.001 sre_parse.py:307(_parse_sub)
       93    0.010    0.000    0.018    0.000 pkg_resources.py:2257(insert_on)
    44/13    0.006    0.000    0.018    0.001 sre_parse.py:385(_parse)
     5839    0.017    0.000    0.017    0.000 {method 'endswith' of
'str' objects}
     5173    0.017    0.000    0.017    0.000 {method 'startswith' of
'str' objects}
      390    0.004    0.000    0.016    0.000 re.py:144(sub)
       13    0.000    0.000    0.016    0.001 sre_compile.py:486(_code)
       22    0.000    0.000    0.015    0.001
pkg_resources.py:1827(normalize_path)
      253    0.014    0.000    0.014    0.000 {imp.find_module}
       99    0.001    0.000    0.013    0.000
pkg_resources.py:2537(_find_adapter)
      196    0.001    0.000    0.011    0.000 pkg_resources.py:1108(safe_name)
    85/13    0.005    0.000    0.011    0.001 sre_compile.py:38(_compile)
      228    0.003    0.000    0.011    0.000 genericpath.py:38(isdir)
       99    0.010    0.000    0.011    0.000 pkg_resources.py:2530(_get_mro)
      116    0.002    0.000    0.011    0.000
pkg_resources.py:2161(_get_metadata)
     1196    0.007    0.000    0.011    0.000 stat.py:55(S_ISLNK)
      194    0.002    0.000    0.010    0.000
pkg_resources.py:1116(safe_version)
      568    0.006    0.000    0.009    0.000 pkg_resources.py:2098(key)
       63    0.001    0.000    0.008    0.000
pkg_resources.py:1805(file_ns_handler)

let me know if you need more info,

cheers,

David

From ziade.tarek at gmail.com  Fri Apr 23 13:57:18 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 23 Apr 2010 13:57:18 +0200
Subject: [Distutils] namespace packages
In-Reply-To: <l2i5b8d13221004230438m32b469f5hde126f7471438838@mail.gmail.com>
References: <4BCCA2CF.2080600@libero.it>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
	<m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
	<s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>
	<h2i5b8d13221004230301x9df501f7sbaa1f83d38f60d2f@mail.gmail.com>
	<r2s94bdd2611004230316hc8c8a6c5v681cb209ccc1e110@mail.gmail.com>
	<l2i5b8d13221004230438m32b469f5hde126f7471438838@mail.gmail.com>
Message-ID: <i2j94bdd2611004230457p4e2a7fb8s418b826fba7b5598@mail.gmail.com>

On Fri, Apr 23, 2010 at 1:38 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Apr 23, 2010 at 7:16 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>> But guessing is not the right thing to do for optimization, we need facts.
>> So if you come back with some profiling information on your use case
>> where it seems so slow
>
> Here is what cProfile -s cum gives me for a script which only import
> pkg_resources:
[..]

> let me know if you need more info,

A pstats file with the same content ? so I can play with it.

And your sys.path, and the pkg_resources version


>
> cheers,
>
> David
>



-- 
Tarek Ziad? | http://ziade.org

From manlio_perillo at libero.it  Fri Apr 23 12:19:42 2010
From: manlio_perillo at libero.it (Manlio Perillo)
Date: Fri, 23 Apr 2010 12:19:42 +0200
Subject: [Distutils] test resources with setup tools
Message-ID: <4BD1743E.2090608@libero.it>

Hi.

In a project test suite I need some external resources, that must be
downloaded from internet.

Is this directly supported by setuptools, or it is better if I write an
additional script that does the job?


Thanks  Manlio

From barry at python.org  Fri Apr 23 14:42:20 2010
From: barry at python.org (Barry Warsaw)
Date: Fri, 23 Apr 2010 08:42:20 -0400
Subject: [Distutils] Disabling 2to3 fixers in setup.py w/Distribute?
Message-ID: <20100423084220.76c0c191@heresy>

Does distribute's 2to3 support allow you to disable specific fixers?

I ask because I just reported bug 8505:

    http://bugs.python.org/issue8505

I can work around it in the code, but it might also make sense to just disable
the fix_future.py fixer, and I'd like to know if it's possible in my package's
setup.py.

Thanks,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100423/b487a653/attachment.pgp>

From pje at telecommunity.com  Fri Apr 23 18:32:25 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 12:32:25 -0400
Subject: [Distutils] namespace packages
In-Reply-To: <y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail .com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
Message-ID: <20100423163226.C67C33A4070@sparrow.telecommunity.com>

At 04:23 PM 4/23/2010 +0900, David Cournapeau wrote:
>  Importing pkg_resources
>causes many more syscalls than relatively big packages (~ 1000 for
>python -c "", 3000 for importing one of numpy/wx/gtk, 6000 for
>pkg_resources). Assuming those are unavoidable (and the current
>namespace implementation in setuptools requires it, right ?), I don't
>see a way to reduce that cost significantly,

If you don't mind trying a simple test for me, would you patch your 
pkg_resources to comment out this loop:

             for pkg in self._get_metadata('namespace_packages.txt'):
                 if pkg in sys.modules: declare_namespace(pkg)

It's in the 'activate()' method of Distribution, and it's targeted 
for removal in setuptools 0.7 anyway...  I suspect you will see a 
huge reduction in stat calls, and the startup time should drop to 
being proportional to the number of non-.egg entries on sys.path, 
rather than being proportional the total number of packages installed 
in such directories.  (For .egg files and directories on sys.path, 
there should be no system calls at all with the above removed.)

This change is not backward compatible with some older packages (from 
years ago) that were not declaring their namespace packages 
correctly, but it has been announced for some time (with warnings) 
that such packages will not work with setuptools 0.7.

(By the way, in case you're thinking this change would only affect 
namespace packages, and you don't have any, what's happening is that 
the _get_metadata() call forces a check for the *existence* of 
namespace_packages.txt in every .egg-info or .egg/EGG-INFO on your 
path, whether the file actually exists or not.  In the case of zipped 
eggs, this check is just looking in a dictionary; for actual 
files/directories, this is a stat call.)


From pje at telecommunity.com  Fri Apr 23 18:44:51 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 12:44:51 -0400
Subject: [Distutils] test resources with setup tools
In-Reply-To: <4BD1743E.2090608@libero.it>
References: <4BD1743E.2090608@libero.it>
Message-ID: <20100423164452.37E743A40F5@sparrow.telecommunity.com>

At 12:19 PM 4/23/2010 +0200, Manlio Perillo wrote:
>Hi.
>
>In a project test suite I need some external resources, that must be
>downloaded from internet.
>
>Is this directly supported by setuptools, or it is better if I write an
>additional script that does the job?

If the resources can be accessed in the form of an .egg file (or 
directory) added to sys.path, setuptools can handle it.  Just add 
'tests_require=requirementslist' to your setup().


From pje at telecommunity.com  Fri Apr 23 18:46:00 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 12:46:00 -0400
Subject: [Distutils] install --dry-run broken?
Message-ID: <20100423164600.C832C3A4108@sparrow.telecommunity.com>

At 09:03 AM 4/23/2010 +0200, Tarek Ziad? wrote:
>On Fri, Apr 23, 2010 at 6:58 AM, P.J. Eby <pje at telecommunity.com> wrote:
> > At 05:11 PM 4/22/2010 -0400, Jim Fulton wrote:
> >>
> >> Or maybe this constitutes "working" for distutils. ;)
> >
> > It does, I'm afraid.  The problem is that the build is also a dry run, so
> > the files aren't there for the later bits to actually copy.
> >
> > It might be better to do a dry run mechanism based mainly on setuptools'
> > sandboxing code -- the sandbox framework could be extended to allow faking
> > all file operations that go through normal Python APIs.  (i.e. not external
> > commands like compilers; those would of course fail unless separately
> > faked.)
>
>Right, I think sandboxing would be a better solution, less intrusive
>to the code,
>and this would work for *any* distutils command out there.
>
>Now my question is about setuptools' sandbox :  it seems to me that 
>it was only
>making sure a setup script wasn't trynig to write outside a given
>directory, using DirectorySandbox.
>
>1- How this tool could be used to record the writings ? Would we need
>to subtype AbstractSandbox ?
>2- If  1/ is doable, what about making this tool its own standalone
>project ? that would be useful for many projects

Yes, you'd need to make a subclass with an alternative policy 
mechanism.  One of the simplest ways might be to remap all the paths 
to a temporary subdirectory tree, except maybe for read-only access 
to files that don't yet exist in that tree.  (The advantage of using 
a temporary tree would be that non-Python code could potentially be 
pointed to real files.)

I'd be fine with making it available standalone.  The only reason 
it's bundled in setuptools is that setuptools needs to install from a 
single file; otherwise it would've had its own PyPI project from the beginning.


From pje at telecommunity.com  Fri Apr 23 18:46:14 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 12:46:14 -0400
Subject: [Distutils] namespace packages
Message-ID: <20100423164615.0DF2C3A4108@sparrow.telecommunity.com>

At 12:16 PM 4/23/2010 +0200, Tarek Ziad? wrote:
>On Fri, Apr 23, 2010 at 12:01 PM, David Cournapeau <cournape at gmail.com> wrote:
> > On Fri, Apr 23, 2010 at 5:54 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> >
> >> I am not sure what you are defining as "complicated". While pkg_resources
> >> is hard to read and it's a project on its own with many other
> >> features, the use case
> >> we are talking about here is dead simple:
> >>
> >>  scan all sys.path entries to look for .egg and .egg-info 
> files/directories.
> >
> > My knowledge may be lacking here, but doen't pkg_resources need to
> > scan things beyond egg-info (to get namespace_package.txt, presumably)
> > ?
>
>That's a file located in egg-info, it reads.

And it's planned to be dropped in 0.7, because it's only needed to 
support packages that declared their namespaces in setup.py but not 
in the relevant __init__.py files.  Setuptools has been warning about 
this coming change for a few years now.  ;-)

With that change, essentially all of pkg_resources' additional disk 
access overhead should disappear.


> > Scanning egg/egg-info is easy, but that does not explain most
> > additional syscalls caused by pkg_resources import.
>
>Well it scans directories and open files so you have roughly  (N * 
>F) + P calls
>where N is the number of packages, F the average files open per package,
>and P the number of entries in sys.path

Yes -- and F should drop to *zero* in setuptools 0.7.  (Also, P is to 
some extent always mitigated by the fact that Python's own import 
machinery is already forcing directory loads for those sys.path entries.)


>For any feature that needs to scan the metadata of installed packages,
>unless there's a central database, we will have to loop over directories.
>
>Now if we consider that everything loaded in sys.path has to be scaned,
>you can't have a central database, thus you need to read the dirs.

But that's not a fixed startup time overhead; it'll just happen when 
you ask for it.


From pje at telecommunity.com  Fri Apr 23 18:46:23 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 12:46:23 -0400
Subject: [Distutils] namespace packages
Message-ID: <20100423164624.4624C3A4108@sparrow.telecommunity.com>

At 08:38 PM 4/23/2010 +0900, David Cournapeau wrote:
>       275    0.024    0.000    0.151    0.001 posixpath.py:351(realpath)

Ouch.  So, over one third of the execution time is spent translating 
symlinks?  That seems...  excessive.


From pje at telecommunity.com  Fri Apr 23 19:07:06 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 23 Apr 2010 13:07:06 -0400
Subject: [Distutils] distribute 0.6.10 and convert_2to3_doctests
In-Reply-To: <20100130190857.015a4382@freewill.wooz.org>
References: <20100106181718.2a506964@freewill>
	<319e029f1001070309kba96eb8ha8031d2dfd285844@mail.gmail.com>
	<F059AED4-FD82-44D6-A128-5D80DE2A335C@python.org>
	<319e029f1001070534j44556099x89d5029aac9d67e9@mail.gmail.com>
	<319e029f1001260853x4697afdo252e45f5938028a9@mail.gmail.com>
	<20100129160521.65eebd10@freewill.wooz.org>
	<94bdd2611001291327m7acccf87j91ee5a43da3c42f9@mail.gmail.com>
	<20100129165426.12de7aba@freewill.wooz.org>
	<94bdd2611001291403t370eff57w6c93a8347c739146@mail.gmail.com>
	<20100130190857.015a4382@freewill.wooz.org>
Message-ID: <20100423170707.B5D9B3A4070@sparrow.telecommunity.com>

At 07:08 PM 1/30/2010 -0500, Barry Warsaw wrote:
>On Jan 29, 2010, at 11:03 PM, Tarek Ziad?? wrote:
>
> >Yes, that's how Jinja does already for example, using Setuptools's
> >pkg_resources :
> >
> >__version__ = __import__('pkg_resources').get_distribution('Jinja2').version
>
>And that's different yet again from what PJE suggests.

*shrug* A mere stylistic difference.  My spelling's just a shorter 
way to do the same thing, while at the same time asserting the 
package's requirements.  But there's nothing *wrong* with the above 
way of doing it.

Personally, I think that querying a package's version is generally a 
wrongheaded idea in the first place.  If you need a specific version, 
asserting this via your project's metadata or a require() line in 
your script is the way to go.  If you are querying API existence, 
hasattr() is the way to go.

That kind of leaves working around bugs in a specific version or two 
as the only sensible use case for a version check...  in which case, 
you're still better off just querying the installed version rather 
than asking the package for a __version__.


>   This is screaming for
>a blessed API to be pushed into the stdlib.
>
>(BTW, why use __import__() there?)

To make it a one-liner, I would guess, in the case where you're not 
doing anything else with pkg_resources.


From regebro at gmail.com  Fri Apr 23 21:45:24 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 23 Apr 2010 21:45:24 +0200
Subject: [Distutils] Disabling 2to3 fixers in setup.py w/Distribute?
In-Reply-To: <20100423084220.76c0c191@heresy>
References: <20100423084220.76c0c191@heresy>
Message-ID: <v2t319e029f1004231245h43e93935k1ad3d5b0946c8c4f@mail.gmail.com>

On Fri, Apr 23, 2010 at 14:42, Barry Warsaw <barry at python.org> wrote:
> Does distribute's 2to3 support allow you to disable specific fixers?

Yeah, you can set the build commands fixer_names attribute to a list
of all the fixers you want. That means you get no default, and
use_2to3_fixers is ignored. Setting this is most neatly done by
subclassing build_py to your own custom build command with the
attribute set as a class attribute.
-- 
Lennart Regebro: http://regebro.wordpress.com/
Python 3 Porting: http://python-incompatibility.googlecode.com/
+33 661 58 14 64

From cournape at gmail.com  Sat Apr 24 07:45:29 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 24 Apr 2010 14:45:29 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <20100423163226.C67C33A4070@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<20100423163226.C67C33A4070@sparrow.telecommunity.com>
Message-ID: <x2p5b8d13221004232245va4c70709lb08665e49513423a@mail.gmail.com>

On Sat, Apr 24, 2010 at 1:32 AM, P.J. Eby <pje at telecommunity.com> wrote:

>
> If you don't mind trying a simple test for me, would you patch your
> pkg_resources to comment out this loop:
>
> ? ? ? ? ? ?for pkg in self._get_metadata('namespace_packages.txt'):
> ? ? ? ? ? ? ? ?if pkg in sys.modules: declare_namespace(pkg)

That looks much better. It is roughly half the time (450 ms -> 250
ms). I had a simple test set with a directory containing N empty
*.egg-info directory, and the import time was proportional to N, now
it does not matter anymore.


> This change is not backward compatible with some older packages (from years
> ago) that were not declaring their namespace packages correctly, but it has
> been announced for some time (with warnings) that such packages will not
> work with setuptools 0.7.
>
> (By the way, in case you're thinking this change would only affect namespace
> packages, and you don't have any, what's happening is that the
> _get_metadata() call forces a check for the *existence* of
> namespace_packages.txt in every .egg-info or .egg/EGG-INFO on your path,
> whether the file actually exists or not. ?In the case of zipped eggs, this
> check is just looking in a dictionary; for actual files/directories, this is
> a stat call.)

Yes, that's exactly what I was seeing in the strace output. Is there a
design document or something else decribing how the namespace
mechanism works for setuptools ? I would like to support namespace
package in my own packaging project, but it is not clear to me what
needs to be done on my side of things.

David

From cournape at gmail.com  Sat Apr 24 07:48:20 2010
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 24 Apr 2010 14:48:20 +0900
Subject: [Distutils] namespace packages
In-Reply-To: <20100423164323.841D93A4070@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<i2k94bdd2611004230051sa07b561bk25c31548f3646bcb@mail.gmail.com>
	<m2t5b8d13221004230130ie4b769acv168d5b748ff80696@mail.gmail.com>
	<s2p94bdd2611004230154r9eaada54sb509ddf3bc9d2855@mail.gmail.com>
	<h2i5b8d13221004230301x9df501f7sbaa1f83d38f60d2f@mail.gmail.com>
	<r2s94bdd2611004230316hc8c8a6c5v681cb209ccc1e110@mail.gmail.com>
	<l2i5b8d13221004230438m32b469f5hde126f7471438838@mail.gmail.com>
	<20100423164323.841D93A4070@sparrow.telecommunity.com>
Message-ID: <t2r5b8d13221004232248l8667c9afu523f869a58b0b299@mail.gmail.com>

On Sat, Apr 24, 2010 at 1:43 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 08:38 PM 4/23/2010 +0900, David Cournapeau wrote:
>>
>> ? ? ?275 ? ?0.024 ? ?0.000 ? ?0.151 ? ?0.001 posixpath.py:351(realpath)
>
> Ouch. ?So, over one third of the execution time is spent translating
> symlinks? ?That seems... ?excessive.

Yes, before making the modification you suggested, I memoized those
calls (normalize_path), and it was already a significant win (like 25
% less time in import, but still proportional to the number of
*.egg-info). Even now it seems to be worthwhile (but only a few %
win).

David

From pje at telecommunity.com  Sun Apr 25 03:34:51 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 24 Apr 2010 21:34:51 -0400
Subject: [Distutils] namespace packages (and PEP 382)
In-Reply-To: <x2p5b8d13221004232245va4c70709lb08665e49513423a@mail.gmail .com>
References: <4BCCA2CF.2080600@libero.it>
	<r2k5b8d13221004211818z93e68db5ie8016d4b8c316c5a@mail.gmail.com>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com>
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com>
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<20100423163226.C67C33A4070@sparrow.telecommunity.com>
	<x2p5b8d13221004232245va4c70709lb08665e49513423a@mail.gmail.com>
Message-ID: <20100425013502.7FFDA3A40F5@sparrow.telecommunity.com>

At 02:45 PM 4/24/2010 +0900, David Cournapeau wrote:
>Is there a
>design document or something else decribing how the namespace
>mechanism works for setuptools ? I would like to support namespace
>package in my own packaging project, but it is not clear to me what
>needs to be done on my side of things.

Namespace packages in *Python* are the result of adding entries to a 
package's '__path__' attribute.  By default, the list only has one 
entry - if you make it have one for each directory (or zipfile 
subdirectory) that contains modules for that package, then presto, 
you have a namespace package.

How you get those entries in __path__ is the only real difference 
between pkgutil, pkg_resources, and the upcoming PEP 382.  (Well, 
there's also the nspkg.pth hack, which I'd just as soon do away with 
altogether.)

I'm thinking that in 0.7 I may add a PEP 382 emulation module 
(separate from pkg_resources) for older versions of Python, as it 
would allow some odd bits to be dispensed with in pkg_resources.

For example, declare_namespace could simply add a '*' to the __path__ 
of the right package(s), and the nspkg.pth files could just make sure 
the PEP 382 emulation was loaded, without needing to set up fake 
package modules.  Other pkg_resources APIs that rely on an internal 
list of namespace packages could just look in sys.namespace_packages, 
and so on.

One thing that's not entirely clear at the moment in PEP 382, though, 
is that it doesn't really specify how the other sys.path entries are 
found and mapped -- presumably it's something like 
os.path.join(syspathentry, *pkgname.split('.')) -- but the handling 
for such path entries isn't really described.  In a Python version 
where NullImporter exists (2.5+), though, the cost of extra 
nonexistent __path__ entries is negligible, so I suppose filtering 
the entries for directory existence and the presence of other .pth 
files isn't necessary.

The PEP should probably be clearer on this point, though - i.e., both 
on what exactly goes in __path__, as well as whether .pth files are 
read in the other paths, and whether .pth files are needed on any 
other paths.  (I also wonder if other .pth contents besides '*' 
really need to be supported anyway; I expect it would depend on 
whether anybody is using the .pkg files allowed by extend_path()...)


From regebro at gmail.com  Sun Apr 25 10:07:26 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Sun, 25 Apr 2010 10:07:26 +0200
Subject: [Distutils] namespace packages (and PEP 382)
In-Reply-To: <20100425013502.7FFDA3A40F5@sparrow.telecommunity.com>
References: <4BCCA2CF.2080600@libero.it>
	<20100422041050.C2F7F3A4070@sparrow.telecommunity.com> 
	<z2x5b8d13221004220036nfaff20d8n4ecea84298f4dc4a@mail.gmail.com> 
	<20100422161038.45B403A4070@sparrow.telecommunity.com>
	<h2g5b8d13221004221816j3b6af88eoead95b2fa2b2710e@mail.gmail.com>
	<20100423050326.4AF373A4070@sparrow.telecommunity.com>
	<y2l5b8d13221004230023q93e303ceh890c3c481b8cf175@mail.gmail.com>
	<20100423163226.C67C33A4070@sparrow.telecommunity.com>
	<x2p5b8d13221004232245va4c70709lb08665e49513423a@mail.gmail.com>
	<20100425013502.7FFDA3A40F5@sparrow.telecommunity.com>
Message-ID: <q2j319e029f1004250107lbcb74e14odcd8984bb2b1319b@mail.gmail.com>

On Sun, Apr 25, 2010 at 03:34, P.J. Eby <pje at telecommunity.com> wrote:
> One thing that's not entirely clear at the moment in PEP 382, though, is
> that it doesn't really specify how the other sys.path entries are found and
> mapped -- presumably it's something like os.path.join(syspathentry,
> *pkgname.split('.')) -- but the handling for such path entries isn't really
> described. ?In a Python version where NullImporter exists (2.5+), though,
> the cost of extra nonexistent __path__ entries is negligible, so I suppose
> filtering the entries for directory existence and the presence of other .pth
> files isn't necessary.

PEP 382 also could benefit from some examples. As it is now, you
really have to understand the details of both namespace packages and
.pth files to understand it.
-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From greg.ewing at canterbury.ac.nz  Mon Apr 26 02:09:59 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 26 Apr 2010 12:09:59 +1200
Subject: [Distutils] install --dry-run broken?
In-Reply-To: <20100423045815.930CA3A4070@sparrow.telecommunity.com>
References: <o2t1099b90b1004221154s25e78a2fx249fa5aca395c11d@mail.gmail.com>
	<20100422195042.3CE233A4070@sparrow.telecommunity.com>
	<p2z1099b90b1004221411hcb38857dt42665b02c1625cd4@mail.gmail.com>
	<20100423045815.930CA3A4070@sparrow.telecommunity.com>
Message-ID: <4BD4D9D7.8060701@canterbury.ac.nz>

P.J. Eby wrote:
> The problem is that the build is also a dry run, 
> so the files aren't there for the later bits to actually copy.

That's not a problem for me, because I'm not trying to
find out what will get copied. I only use --dry-run to
exercise my setup.py code to make sure it doesn't crash.

Please don't take this functionality away just because
of some other functionality that it doesn't provide.

-- 
Greg

From manlio.perillo at gmail.com  Mon Apr 26 18:53:20 2010
From: manlio.perillo at gmail.com (Manlio Perillo)
Date: Mon, 26 Apr 2010 18:53:20 +0200
Subject: [Distutils] test resources with setup tools
In-Reply-To: <20100423164452.37E743A40F5@sparrow.telecommunity.com>
References: <4BD1743E.2090608@libero.it>
	<20100423164452.37E743A40F5@sparrow.telecommunity.com>
Message-ID: <4BD5C500.2020907@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

P.J. Eby ha scritto:
> At 12:19 PM 4/23/2010 +0200, Manlio Perillo wrote:
>> Hi.
>>
>> In a project test suite I need some external resources, that must be
>> downloaded from internet.
>>
>> Is this directly supported by setuptools, or it is better if I write an
>> additional script that does the job?
> 
> If the resources can be accessed in the form of an .egg file (or
> directory) added to sys.path, setuptools can handle it.  

The resources are archives that must be downloaded from internet,
decompressed and processed.

Right now I'm using a shell script that must be manually executed.
The data is copied to a test/resources directory.

The test directory contains the test suite and it is in the top level
directory of the Python project. It is not installed on the system by
the setup.py script.

Test functions access the the data using __file__.

I was just wondering if there is a better method.


Thanks  Manlio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvVxQAACgkQscQJ24LbaUQdXACfT9gQHDOto/utLXs6Xuc8qt8N
eYsAnRlw35TUAZIWiWSNpUEJT0AQjmkh
=jJg4
-----END PGP SIGNATURE-----

From mbaiju at zeomega.com  Mon Apr 26 19:35:44 2010
From: mbaiju at zeomega.com (Baiju M)
Date: Mon, 26 Apr 2010 23:05:44 +0530
Subject: [Distutils] test resources with setup tools
In-Reply-To: <4BD5C500.2020907@gmail.com>
References: <4BD1743E.2090608@libero.it>
	<20100423164452.37E743A40F5@sparrow.telecommunity.com>
	<4BD5C500.2020907@gmail.com>
Message-ID: <i2gc911c70d1004261035z7cd9c157r98a37f39995ae00b@mail.gmail.com>

On Mon, Apr 26, 2010 at 10:23 PM, Manlio Perillo
<manlio.perillo at gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> P.J. Eby ha scritto:
>> At 12:19 PM 4/23/2010 +0200, Manlio Perillo wrote:
>>> Hi.
>>>
>>> In a project test suite I need some external resources, that must be
>>> downloaded from internet.
>>>
>>> Is this directly supported by setuptools, or it is better if I write an
>>> additional script that does the job?
>>
>> If the resources can be accessed in the form of an .egg file (or
>> directory) added to sys.path, setuptools can handle it.
>
> The resources are archives that must be downloaded from internet,
> decompressed and processed.
>
> Right now I'm using a shell script that must be manually executed.
> The data is copied to a test/resources directory.
>
> The test directory contains the test suite and it is in the top level
> directory of the Python project. It is not installed on the system by
> the setup.py script.
>
> Test functions access the the data using __file__.
>
> I was just wondering if there is a better method.

You can use Buildout something like this:

1. Get bootstrap script

   wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

2. Create a buildout conf with appropriate parts

[buildout]
parts = resource
           extract_resource
           test

[resource]
recipe = hexagonit.recipe.download
url = http://example.com/resource.tar.bz2
location = ${buildout:directory}/downloads
download-only = true

[extract_resource]
recipe = iw.recipe.cmd
on_install = true
on_update = true
shell = bash
cmds =
    cd ${buildout:directory};tar jxvf
${buildout:directory}/downloads/resource.tar.bz2

[test]
recipe = zc.recipe.testrunner
eggs = mypkg
defaults = ['-v', '-k']

3. Run bootstrap.py

python bootstrap.py

4. Run buildout

./bin/buildout

5. Run test

./bin/test

So many assumptions are made here about the requirements.
You can change the recipes and parts as you required.
There are nearly 200 recipes and extensions available in PyPI,
creating one yourself is also very easy:
http://pypi.python.org/pypi?:action=browse&c=512

Basically Buildout provides an isolated-repeatable environment.

Regards,
Baiju M

From pje at telecommunity.com  Mon Apr 26 19:54:45 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 26 Apr 2010 13:54:45 -0400
Subject: [Distutils] test resources with setup tools
In-Reply-To: <4BD5C500.2020907@gmail.com>
References: <4BD1743E.2090608@libero.it>
	<20100423164452.37E743A40F5@sparrow.telecommunity.com>
	<4BD5C500.2020907@gmail.com>
Message-ID: <20100426175444.2B8F03A4070@sparrow.telecommunity.com>

At 06:53 PM 4/26/2010 +0200, Manlio Perillo wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>P.J. Eby ha scritto:
> > At 12:19 PM 4/23/2010 +0200, Manlio Perillo wrote:
> >> Hi.
> >>
> >> In a project test suite I need some external resources, that must be
> >> downloaded from internet.
> >>
> >> Is this directly supported by setuptools, or it is better if I write an
> >> additional script that does the job?
> >
> > If the resources can be accessed in the form of an .egg file (or
> > directory) added to sys.path, setuptools can handle it.
>
>The resources are archives that must be downloaded from internet,
>decompressed and processed.
>
>Right now I'm using a shell script that must be manually executed.
>The data is copied to a test/resources directory.
>
>The test directory contains the test suite and it is in the top level
>directory of the Python project. It is not installed on the system by
>the setup.py script.
>
>Test functions access the the data using __file__.
>
>I was just wondering if there is a better method.


If you put those resources as package data files or metadata files in 
another PyPI project, so that they can be installed in .egg form, 
then setuptools can do it for you, as I described above.


From manlio_perillo at libero.it  Mon Apr 26 20:52:42 2010
From: manlio_perillo at libero.it (Manlio Perillo)
Date: Mon, 26 Apr 2010 20:52:42 +0200
Subject: [Distutils] test resources with setup tools
In-Reply-To: <20100426175444.2B8F03A4070@sparrow.telecommunity.com>
References: <4BD1743E.2090608@libero.it>
	<20100423164452.37E743A40F5@sparrow.telecommunity.com>
	<4BD5C500.2020907@gmail.com>
	<20100426175444.2B8F03A4070@sparrow.telecommunity.com>
Message-ID: <4BD5E0FA.6070700@libero.it>

P.J. Eby ha scritto:
> [...]
>> Right now I'm using a shell script that must be manually executed.
>> The data is copied to a test/resources directory.
>>
>> The test directory contains the test suite and it is in the top level
>> directory of the Python project. It is not installed on the system by
>> the setup.py script.
>>
>> Test functions access the the data using __file__.
>>
>> I was just wondering if there is a better method.
> 
> 
> If you put those resources as package data files or metadata files in
> another PyPI project, so that they can be installed in .egg form, then
> setuptools can do it for you, as I described above.
> 

This seems the best solution, thanks.

However, I would like to put the data in a *sub* project, that is know
only to my project setup.  Unfortunately this seems to not be possible.

Creating a "public" project for this does not feel right, to me.


I have some other questions.
Supposing that in the other PyPI project I still want to use a shell
script to download and process the data, is it correct to execute the
script in the setup.py script, before the setup function is called?

Since the shell script depends on some executables to be available on
the system (wget, tar, bzip2, imagemagick), is there a standard method
to check for existence of system executables in the setup?



Thanks  Manlio

From janssen at parc.com  Mon Apr 26 21:33:30 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 26 Apr 2010 12:33:30 PDT
Subject: [Distutils] VBScript puzzle
In-Reply-To: <b8ad139e1003231357t1310d253q8ba0e53434dee835@mail.gmail.com>
References: <b8ad139e1003231357t1310d253q8ba0e53434dee835@mail.gmail.com>
Message-ID: <3166.1272310410@parc.com>

Jess Austin <jess.austin at gmail.com> wrote:

> On Mon, 22 Mar 2010 at 21:25:55,  "Martin v. L?wis" <martin at v.loewis.de> wrote:
> > As this might be an interesting puzzle to solve, I'd like to pose it to
> > the entire distutils-sig readership. So here is the problem again:
> >
> > As a batch file, my attempt at having a batch file that is also a
> > Python script was to write it as
> >
> > rem="""
> > %1 %0
> > exit
> > """
> > <python code here>
> >
> > This should have been run with argv[1] being the path to the Python
> > interpreter. As a batch file, the first line is a comment, the second
> > line runs Python, anbd the third line terminates the script (so it
> > doesn't consider the following lines). As a Python script, the first
> > block is a variable assignment, followed by the regular Python script.
> >
> > Now (as Installer won't run batch files) we need the same as VBScript
> > (or, failing that, as JScript). The script would be computed at the time
> > of MSI generation.
> 
> I couldn't get this to work as VBScript (that requires too many line
> continuation characters), but I think the following JScript should
> work.  The call to WShell.Run() pops another window, so I included a
> time.sleep() call to make that slow enough to notice.  Fortunately
> JScript and python both will happily ignore integer and string
> literals at the beginning of the file, and JScript comments look like
> python floor division.  I ran this at the command line like so:
> 
> 
> C:\>cscript bdist.js "Python26\python.exe"
> 
> 
> Here's the script:
> 
> 
> 4 // 3; '''
> ;
> if (WScript.Arguments.Count() < 1)
>   WScript.Echo("usage: " + WScript.ScriptName + " <path to python>");
> else
>   WScript.CreateObject("WScript.Shell").Run(WScript.Arguments.Item(0) + " " +
>                                             WScript.ScriptFullName, 1, true); /*
> '''
> # start of python script
> from time import sleep
> print("hello from python")
> sleep(5)
> # end of python script */

Note that JScript run under the MSI context (custom action type 5)
doesn't have access to the WScript object.  It's restricted to
the set of objects accessible under the installer.

http://msdn.microsoft.com/en-us/library/aa367810(VS.85).aspx

Might be able to invoke "wscript.exe" to run the script, making it a
type 50 custom action.  But apparently that ability is frequently
disabled by a group policy.

Bill

From pje at telecommunity.com  Mon Apr 26 22:58:10 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 26 Apr 2010 16:58:10 -0400
Subject: [Distutils] test resources with setup tools
In-Reply-To: <4BD5E0E3.5080309@libero.it>
References: <4BD1743E.2090608@libero.it>
	<20100423164452.37E743A40F5@sparrow.telecommunity.com>
	<4BD5C500.2020907@gmail.com>
	<20100426175444.2B8F03A4070@sparrow.telecommunity.com>
	<4BD5E0E3.5080309@libero.it>
Message-ID: <20100426205805.8323F3A4070@sparrow.telecommunity.com>

At 08:52 PM 4/26/2010 +0200, Manlio Perillo wrote:
>However, I would like to put the data in a *sub* project, that is know
>only to my project setup.  Unfortunately this seems to not be possible.
>
>Creating a "public" project for this does not feel right, to me.

In that case, why not just ship the data with the source distribution?


>Supposing that in the other PyPI project I still want to use a shell
>script to download and process the data, is it correct to execute the
>script in the setup.py script, before the setup function is called?

No.  Otherwise, this code will run even if somebody tries to do 
setup.py --help or build, say, a source distribution.  The only safe 
way to extend a setup script's behavior is by subclassing distutils 
commnads to perform the actions.


>Since the shell script depends on some executables to be available on
>the system (wget, tar, bzip2, imagemagick), is there a standard method
>to check for existence of system executables in the setup?

Distutils has some functions for this sort of thing; I think they're 
in distutils.util.


From janssen at parc.com  Mon Apr 26 23:04:12 2010
From: janssen at parc.com (Bill Janssen)
Date: Mon, 26 Apr 2010 14:04:12 PDT
Subject: [Distutils] VBScript puzzle
In-Reply-To: <b8ad139e1003231357t1310d253q8ba0e53434dee835@mail.gmail.com>
References: <b8ad139e1003231357t1310d253q8ba0e53434dee835@mail.gmail.com>
Message-ID: <4309.1272315852@parc.com>

Jess Austin <jess.austin at gmail.com> wrote:

> C:\>cscript bdist.js "Python26\python.exe"
> 
> 
> Here's the script:
> 
> 
> 4 // 3; '''
> ;
> if (WScript.Arguments.Count() < 1)
>   WScript.Echo("usage: " + WScript.ScriptName + " <path to python>");
> else
>   WScript.CreateObject("WScript.Shell").Run(WScript.Arguments.Item(0) + " " +
>                                             WScript.ScriptFullName, 1, true); /*
> '''
> # start of python script
> from time import sleep
> print("hello from python")
> sleep(5)
> # end of python script */

The killer here is WScript.ScriptFullName.  The MSI framework does not
want to give that to you.  The only way to do this (I think) is to
include the full Python program as a literal in the JScript program.  It
can then create a temporary file in InstallTemp, write the Python to it,
find Python in the registry, and invoke Python on the Python script.

To make it more general, you could use a property for the location of
Python, and use another custom action to find and/or set that property.

What seems to work is to base64-encode the Python script andadd it as a
literal to the JScript.  Then the JScript saves it to a file, and
invokes Python on it with
"import base64; exec(base64.decodestring(open(TEMPFILE).read().strip()))":

Here's my JScript:

var pythonscript_base64 = 'cHJpbnQgJ2hlbGxvLCB3b3JsZCEn\n';
var WshShell = new ActiveXObject ("WScript.Shell");
var filesystem = new ActiveXObject("Scripting.FileSystemObject");
var logfile = filesystem.CreateTextFile("C:\\install-script.log");
var installtemp = Session.Property("TempFolder");
logfile.WriteLine("TempFolder is " + installtemp);
var scriptfile = filesystem.CreateTextFile(installtemp + "iscript.py");
scriptfile.Write(pythonscript_base64);
scriptfile.Close();

var pythonDir = WshShell.RegRead ("HKEY_LOCAL_MACHINE\\Software\\Python\\PythonCore\\2.6\\InstallPath\\");
var pythonExe = pythonDir + "pythonw.exe";
if (! filesystem.FileExists(pythonExe)) {
  logfile.WriteLine("Python not found!");
  exit(1);              // can we really just use exit?
 } else {
  logfile.WriteLine("python is " + pythonExe);
 };
var cmd = pythonExe + ' -c "import base64; exec(base64.decodestring(open(' + "r'" + installtemp + "iscript.py').read()))" + '"';
logfile.WriteLine("cmd is " + cmd);
var oExec = WshShell.Exec(cmd);
while (oExec.Status == 0) {
  while (!oExec.StdOut.AtEndOfStream) {
    logfile.WriteLine(oExec.StdOut.ReadLine());
  }
  while (!oExec.StdErr.AtEndOfStream) {
    logfile.WriteLine(oExec.StdErr.ReadLine());
  }
  // WScript.Sleep(100);
} /*
'''


Bill

From manlio.perillo at gmail.com  Mon Apr 26 23:59:35 2010
From: manlio.perillo at gmail.com (Manlio Perillo)
Date: Mon, 26 Apr 2010 23:59:35 +0200
Subject: [Distutils] test resources with setup tools
In-Reply-To: <20100426205805.8323F3A4070@sparrow.telecommunity.com>
References: <4BD1743E.2090608@libero.it>
	<20100423164452.37E743A40F5@sparrow.telecommunity.com>
	<4BD5C500.2020907@gmail.com>
	<20100426175444.2B8F03A4070@sparrow.telecommunity.com>
	<4BD5E0E3.5080309@libero.it>
	<20100426205805.8323F3A4070@sparrow.telecommunity.com>
Message-ID: <4BD60CC7.4010702@gmail.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

P.J. Eby ha scritto:
> At 08:52 PM 4/26/2010 +0200, Manlio Perillo wrote:
>> However, I would like to put the data in a *sub* project, that is know
>> only to my project setup.  Unfortunately this seems to not be possible.
>>
>> Creating a "public" project for this does not feel right, to me.
> 
> In that case, why not just ship the data with the source distribution?
> 

Because the shell script will download about 24 MB, of images and fonts!
And the data is used only by the test suite.

> 
>> Supposing that in the other PyPI project I still want to use a shell
>> script to download and process the data, is it correct to execute the
>> script in the setup.py script, before the setup function is called?
> 
> No.  Otherwise, this code will run even if somebody tries to do setup.py
> --help or build, say, a source distribution.  The only safe way to
> extend a setup script's behavior is by subclassing distutils commnads to
> perform the actions.
> 

Ok.

> 
>> Since the shell script depends on some executables to be available on
>> the system (wget, tar, bzip2, imagemagick), is there a standard method
>> to check for existence of system executables in the setup?
> 
> Distutils has some functions for this sort of thing; I think they're in
> distutils.util.
> 
> 


Thanks  Manlio
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvWDMcACgkQscQJ24LbaURDDACeNh/Li/kigx+JM7/vONqgyzqk
wRMAn2N2bZprlnUtJ8I8bM7Z3ZqcyO9v
=Ffwu
-----END PGP SIGNATURE-----

From cp.hennessy at openapp.ie  Tue Apr 27 17:45:14 2010
From: cp.hennessy at openapp.ie (CP Hennessy)
Date: Tue, 27 Apr 2010 16:45:14 +0100
Subject: [Distutils] buildout usage : a setup steps needs access to an
	installed package
Message-ID: <201004271645.14603.cp.hennessy@openapp.ie>

Hi,
 I'm relatively new to buildout so this may be a stupid question :)

I'm trying to install nltk (Natural Language ToolKit) but it depends on 
PyYAML. So PyYAML is successfully installed as an egg before nltk. 

However nltk needs PyYAML during it's installation, but nltk does not have 
access to the PyYAML egg. 

How do I configure the buildout to give access to the PyYAML egg to the setup 
of nltk.

Thanks,
CPH

From jim at zope.com  Tue Apr 27 19:13:33 2010
From: jim at zope.com (Jim Fulton)
Date: Tue, 27 Apr 2010 13:13:33 -0400
Subject: [Distutils] buildout usage : a setup steps needs access to an
	installed package
In-Reply-To: <201004271645.14603.cp.hennessy@openapp.ie>
References: <201004271645.14603.cp.hennessy@openapp.ie>
Message-ID: <h2n1099b90b1004271013k8dd4c78h6ab54c07f1bc6002@mail.gmail.com>

On Tue, Apr 27, 2010 at 11:45 AM, CP Hennessy <cp.hennessy at openapp.ie> wrote:
> Hi,
> ?I'm relatively new to buildout so this may be a stupid question :)
>
> I'm trying to install nltk (Natural Language ToolKit) but it depends on
> PyYAML. So PyYAML is successfully installed as an egg before nltk.
>
> However nltk needs PyYAML during it's installation, but nltk does not have
> access to the PyYAML egg.
>
> How do I configure the buildout to give access to the PyYAML egg to the setup
> of nltk.

Does the setup.py include a setting for setup_requires that names
PyYAML.  I'm told that providing that option will make buildout handle
this situation.

(I've been meaning to add support for setup_requires to buildout, but
haven't yet, but I'm told it works anyway. Perhaps setuptools is
taking care of it for me. If so, thanks setuptools! :)

Jim

--
Jim Fulton

From kevin at bud.ca  Tue Apr 27 19:27:35 2010
From: kevin at bud.ca (Kevin Teague)
Date: Tue, 27 Apr 2010 10:27:35 -0700
Subject: [Distutils] buildout usage : a setup steps needs access to an
	installed package
In-Reply-To: <201004271645.14603.cp.hennessy@openapp.ie>
References: <201004271645.14603.cp.hennessy@openapp.ie>
Message-ID: <p2ye990419a1004271027t3a7cae06yc0bcfa8e69c813f8@mail.gmail.com>

"Thou shalt not put thine metadata inside thine data."

The nltk setup.py imports nltk which in turn imports PyYAML, so you can't
install nltk and it's dependencies until nltk and it's dependencies are
installed. Ideally you should ask the nltk maintainers to not import the
thing which they are trying to install, and move their metadata out of the
nltk package and into setup.py. FWIW, this package won't install w/ pip
either.

I wasn't aware of the 'setup_requires' option - so I just gave that a try
with a modified nltk package w/ buildout by making the PyYAML setup
dependency explicit, but no dice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100427/8c361ae8/attachment.html>

From sridharr at activestate.com  Tue Apr 27 21:47:52 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Tue, 27 Apr 2010 12:47:52 -0700
Subject: [Distutils] PEP 345 and build_requires?
Message-ID: <0F258A2A-B599-4E68-8076-4901674DD60A@activestate.com>

http://goo.gl/zPtX (Google Cache of PEP 345)

PEP 345 introduces `Requires-Dist` metadata field that can be used to define *install-time* requirements. Has anyone thought about having a field that can be used to define *build-time* requirement (similar to setuptools' setup_requires?)? Eg: 

$ grep Build-Requires-Dist lxml-2.2.6/METADATA
Build-Requires-Dist: Cython
Build-Requires-Dist: Scons
$

Or is the only hope to put them all in install_requires/Requires-Dist and have the packagers/installers to resolve and install the dependencies *before* actually attempting to build that module?

-srid


From warren.weckesser at enthought.com  Tue Apr 27 19:18:22 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 27 Apr 2010 12:18:22 -0500
Subject: [Distutils] Problem with --uninstall option to setup.py develop
Message-ID: <4BD71C5E.8030306@enthought.com>

I am using version "0.6c9-s1" of setuptools and Python 2.6.4.

I am trying to use the --uninstall option of "setup.py develop", but 
even with a simple example, I get the following error:

-----
none:devtest warren$ python setup.py develop --uninstall
running develop
Traceback (most recent call last):
 File "setup.py", line 8, in <module>
   author="WW",
 File 
"/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/distutils/core.py", 
line 152, in setup
   dist.run_commands()
 File 
"/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/distutils/dist.py", 
line 975, in run_commands
   self.run_command(cmd)
 File 
"/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/distutils/dist.py", 
line 995, in run_command
   cmd_obj.run()
 File 
"/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/setuptools/command/develop.py", 
line 26, in run
   self._run_egg_info_script(self.distribution.location,
AttributeError: Distribution instance has no attribute 'location'
none:devtest warren$
-----

My files look like this:

devtest/
   setup.py
   qwerty/
       __init__.py
       qwerty.py

Running "setup.py develop" works fine.

Here's setup.py:

-----
from setuptools import setup, find_packages

setup(
   name="qwerty",
   packages=find_packages(),
   version="0.1.0",
   author="WW",
)
-----

Is this a known problem, maybe fixed in a later version?


Regards,

Warren




From warren.weckesser at enthought.com  Tue Apr 27 21:35:30 2010
From: warren.weckesser at enthought.com (Warren Weckesser)
Date: Tue, 27 Apr 2010 14:35:30 -0500
Subject: [Distutils] Problem with --uninstall option to setup.py develop
In-Reply-To: <4BD71C5E.8030306@enthought.com>
References: <4BD71C5E.8030306@enthought.com>
Message-ID: <4BD73C82.4010501@enthought.com>

Update:  the version of setuptools that I was using was relatively old, 
and had local patches.  With distribute version 0.6.10, the --uninstall 
option works.

Warren


Warren Weckesser wrote:
> I am using version "0.6c9-s1" of setuptools and Python 2.6.4.
>
> I am trying to use the --uninstall option of "setup.py develop", but 
> even with a simple example, I get the following error:
>
> -----
> none:devtest warren$ python setup.py develop --uninstall
> running develop
> Traceback (most recent call last):
> File "setup.py", line 8, in <module>
>   author="WW",
> File 
> "/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/distutils/core.py", 
> line 152, in setup
>   dist.run_commands()
> File 
> "/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/distutils/dist.py", 
> line 975, in run_commands
>   self.run_command(cmd)
> File 
> "/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/distutils/dist.py", 
> line 995, in run_command
>   cmd_obj.run()
> File 
> "/Library/Frameworks/Python.framework/Versions/6.1/lib/python2.6/site-packages/setuptools/command/develop.py", 
> line 26, in run
>   self._run_egg_info_script(self.distribution.location,
> AttributeError: Distribution instance has no attribute 'location'
> none:devtest warren$
> -----
>
> My files look like this:
>
> devtest/
>   setup.py
>   qwerty/
>       __init__.py
>       qwerty.py
>
> Running "setup.py develop" works fine.
>
> Here's setup.py:
>
> -----
> from setuptools import setup, find_packages
>
> setup(
>   name="qwerty",
>   packages=find_packages(),
>   version="0.1.0",
>   author="WW",
> )
> -----
>
> Is this a known problem, maybe fixed in a later version?
>
>
> Regards,
>
> Warren
>
>
>
>


From pje at telecommunity.com  Wed Apr 28 00:30:23 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Tue, 27 Apr 2010 18:30:23 -0400
Subject: [Distutils] Problem with --uninstall option to setup.py develop
In-Reply-To: <4BD71C5E.8030306@enthought.com>
References: <4BD71C5E.8030306@enthought.com>
Message-ID: <20100427223020.07CC23A407B@sparrow.telecommunity.com>

At 12:18 PM 4/27/2010 -0500, Warren Weckesser wrote:
>I am using version "0.6c9-s1" of setuptools and Python 2.6.4.

That's not an official release, and is probably out of date in any 
case; try updating to 0.6c11.


From chris at simplistix.co.uk  Wed Apr 28 15:14:13 2010
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 28 Apr 2010 14:14:13 +0100
Subject: [Distutils] building with static dependencies
Message-ID: <4BD834A5.8070504@simplistix.co.uk>

Hi All,

I'd like to build some packages into eggs such that I can easilly 
install them with virtual_env or buildout.

The packages in question are numpy and scipy, at the moment.

Are there parameters I can pass to "python setup.py build" or "python 
setup.py bdist_egg" that will cause the egg to be built will all its 
dependencies statically linked and included within the egg?

(ie: so I can build once on linux and once on windows, put the binary 
eggs in our local index and then not have to worry about the build 
proceses and convoluted dependencies on other machines of a similar 
architecture)

cheers,

Chris


From jim at zope.com  Wed Apr 28 18:09:54 2010
From: jim at zope.com (Jim Fulton)
Date: Wed, 28 Apr 2010 12:09:54 -0400
Subject: [Distutils] buildout usage : a setup steps needs access to an
	installed package
In-Reply-To: <p2ye990419a1004271027t3a7cae06yc0bcfa8e69c813f8@mail.gmail.com>
References: <201004271645.14603.cp.hennessy@openapp.ie>
	<p2ye990419a1004271027t3a7cae06yc0bcfa8e69c813f8@mail.gmail.com>
Message-ID: <w2h1099b90b1004280909x5d004f99q980941c29f9dfaac@mail.gmail.com>

On Tue, Apr 27, 2010 at 1:27 PM, Kevin Teague <kevin at bud.ca> wrote:
> "Thou shalt not put thine metadata inside thine data."
>
> The nltk setup.py imports nltk which in turn imports PyYAML, so you can't
> install nltk and it's dependencies until nltk and it's dependencies are
> installed. Ideally you should ask the nltk maintainers to not import the
> thing which they are trying to install, and move their metadata out of the
> nltk package and into setup.py. FWIW, this package won't install w/ pip
> either.
>
> I wasn't aware of the 'setup_requires' option - so I just gave that a try
> with a modified nltk package w/ buildout by making the PyYAML setup
> dependency explicit, but no dice.

Dang.  Well, I guess trying to make that work in buildout is till on
my to-do list.

Jim

-- 
Jim Fulton

From sridharr at activestate.com  Wed Apr 28 18:55:25 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Wed, 28 Apr 2010 09:55:25 -0700
Subject: [Distutils] building with static dependencies
In-Reply-To: <4BD834A5.8070504@simplistix.co.uk>
References: <4BD834A5.8070504@simplistix.co.uk>
Message-ID: <3D45B04E-A8C3-4575-98D4-BC4DC36E0C1F@activestate.com>

Similar to "STATICDEPS=True python setup.py bdist_egg" in lxml?

For numpy you only need to use the pre-built ATLAS libraries (see scipy installation document) on windows 32-bit (for 64-bit, you will need the proprietary MKL to build atlas), and build atlas yourself on linux. In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.

To statically link libgfortran: on Mac, take a look at the libgfotran.a copying hack in pavement.py (scipy trunk); on Linux, I haven't yet figured out a way to reliably link libgfortran statically to numpy.

In PyPM, we are standardizing on the 'STATIC=True' environment variable for configuring packages to build with static linking. Eg: we patched mysql-python to test this environment variable and then build statically against mysql (after automatically downloading/building it). 

You will have to patch setup.py yourself in order to make the packages build statically (eg: see buildlibxml.py in lxml source)

-srid

On 2010-04-28, at 6:14 AM, Chris Withers wrote:

> Hi All,
> 
> I'd like to build some packages into eggs such that I can easilly install them with virtual_env or buildout.
> 
> The packages in question are numpy and scipy, at the moment.
> 
> Are there parameters I can pass to "python setup.py build" or "python setup.py bdist_egg" that will cause the egg to be built will all its dependencies statically linked and included within the egg?
> 
> (ie: so I can build once on linux and once on windows, put the binary eggs in our local index and then not have to worry about the build proceses and convoluted dependencies on other machines of a similar architecture)
> 
> cheers,
> 
> Chris
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


From jbaker at zeomega.com  Thu Apr 29 20:05:41 2010
From: jbaker at zeomega.com (Jason Baker)
Date: Thu, 29 Apr 2010 13:05:41 -0500
Subject: [Distutils] How can I silence setuptools/distutils output in a
	custom command?
Message-ID: <s2gfa34f1941004291105ra4effb34l9cc9e48746e5ce5b@mail.gmail.com>

I've written a simple setuptools command that will read options from a
setup.cfg.  The code is here:
http://github.com/jasonbaker/rdopts/blob/master/rdopts.py

I'd like to be able to use the output of this command to pass to other
programs, but that is a bit difficult when distutils is writing "running
rdopts" out every time I use it.  The solution I found is to just call
distutils.log.set_threshold(ERROR) at import time, but that just feels like
a recipe for something to go wrong.  Are there any better ways to do this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100429/5f079119/attachment.html>

From pje at telecommunity.com  Thu Apr 29 20:27:51 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 29 Apr 2010 14:27:51 -0400
Subject: [Distutils] How can I silence setuptools/distutils output in a
 custom command?
In-Reply-To: <s2gfa34f1941004291105ra4effb34l9cc9e48746e5ce5b@mail.gmail .com>
References: <s2gfa34f1941004291105ra4effb34l9cc9e48746e5ce5b@mail.gmail.com>
Message-ID: <20100429182751.94C3A3A4070@sparrow.telecommunity.com>

At 01:05 PM 4/29/2010 -0500, Jason Baker wrote:
>I've written a simple setuptools command that will read options from 
>a setup.cfg.  The code is 
>here: 
><http://github.com/jasonbaker/rdopts/blob/master/rdopts.py>http://github.com/jasonbaker/rdopts/blob/master/rdopts.py
>
>I'd like to be able to use the output of this command to pass to 
>other programs, but that is a bit difficult when distutils is 
>writing "running rdopts" out every time I use it.  The solution I 
>found is to just call distutils.log.set_threshold(ERROR) at import 
>time, but that just feels like a recipe for something to go 
>wrong.  Are there any better ways to do this?

python setup.py -q rdopts


From jbaker at zeomega.com  Thu Apr 29 20:58:19 2010
From: jbaker at zeomega.com (Jason Baker)
Date: Thu, 29 Apr 2010 13:58:19 -0500
Subject: [Distutils] How can I silence setuptools/distutils output in a
	custom command?
In-Reply-To: <20100429182751.94C3A3A4070@sparrow.telecommunity.com>
References: <s2gfa34f1941004291105ra4effb34l9cc9e48746e5ce5b@mail.gmail.com> 
	<20100429182751.94C3A3A4070@sparrow.telecommunity.com>
Message-ID: <r2hfa34f1941004291158nc40117bcvee800c27f7c125fd@mail.gmail.com>

On Thu, Apr 29, 2010 at 1:27 PM, P.J. Eby <pje at telecommunity.com> wrote:

> At 01:05 PM 4/29/2010 -0500, Jason Baker wrote:
>
>> I've written a simple setuptools command that will read options from a
>> setup.cfg.  The code is here: <
>> http://github.com/jasonbaker/rdopts/blob/master/rdopts.py>
>> http://github.com/jasonbaker/rdopts/blob/master/rdopts.py
>>
>>
>> I'd like to be able to use the output of this command to pass to other
>> programs, but that is a bit difficult when distutils is writing "running
>> rdopts" out every time I use it.  The solution I found is to just call
>> distutils.log.set_threshold(ERROR) at import time, but that just feels like
>> a recipe for something to go wrong.  Are there any better ways to do this?
>>
>
> python setup.py -q rdopts
>

Ah, I see.  I was doing "python setup.py rdopts -q".  Is there any way to
make -q enabled by default?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100429/2bcc7b55/attachment.html>

From pje at telecommunity.com  Thu Apr 29 23:21:28 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 29 Apr 2010 17:21:28 -0400
Subject: [Distutils] How can I silence setuptools/distutils output in a
 custom command?
In-Reply-To: <r2hfa34f1941004291158nc40117bcvee800c27f7c125fd@mail.gmail .com>
References: <s2gfa34f1941004291105ra4effb34l9cc9e48746e5ce5b@mail.gmail.com>
	<20100429182751.94C3A3A4070@sparrow.telecommunity.com>
	<r2hfa34f1941004291158nc40117bcvee800c27f7c125fd@mail.gmail.com>
Message-ID: <20100429212127.07B1A3A4116@sparrow.telecommunity.com>

At 01:58 PM 4/29/2010 -0500, Jason Baker wrote:
>On Thu, Apr 29, 2010 at 1:27 PM, P.J. Eby 
><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
>At 01:05 PM 4/29/2010 -0500, Jason Baker wrote:
>I've written a simple setuptools command that will read options from 
>a setup.cfg.  The code is here: 
><<http://github.com/jasonbaker/rdopts/blob/master/rdopts.py>http://github.com/jasonbaker/rdopts/blob/master/rdopts.py>http://github.com/jasonbaker/rdopts/blob/master/rdopts.py
>
>
>I'd like to be able to use the output of this command to pass to 
>other programs, but that is a bit difficult when distutils is 
>writing "running rdopts" out every time I use it.  The solution I 
>found is to just call distutils.log.set_threshold(ERROR) at import 
>time, but that just feels like a recipe for something to go 
>wrong.  Are there any better ways to do this?
>
>
>python setup.py -q rdopts
>
>
>Ah, I see.  I was doing "python setup.py rdopts -q".  Is there any 
>way to make -q enabled by default?

Apart from using the alias command (or other config file settings), I 
don't know of any.



From SridharR at activestate.com  Fri Apr 30 02:21:56 2010
From: SridharR at activestate.com (Sridhar Ratnakumar)
Date: Thu, 29 Apr 2010 17:21:56 -0700
Subject: [Distutils] Buildout for Python 3
Message-ID: <F228F4CE-F2C2-47F3-9671-45CDE12A1595@activestate.com>

Hi,

I have begun porting PyPM along with some dependent modules (cmdln, platinfo) to Python 3. Since I use buildout, I wonder if anyone has already taken up the work of porting it to Python 3. If so, would you need any help in completing it? If no one is currently doing it, I'd be willing to take a stab at working on it myself provided there is some assurance of it getting merged with zc.buildout trunk. Otherwise, I may have to look at alternative setups (virtualenv+pip?).

-srid


From setuptools at bugs.python.org  Fri Apr 30 03:18:58 2010
From: setuptools at bugs.python.org (Patrick Ting)
Date: Fri, 30 Apr 2010 01:18:58 +0000
Subject: [Distutils] [issue107] bdist_egg --bdist-dir param not working
	quite as expected
In-Reply-To: <1272590338.83.0.399619968852.issue107@psf.upfronthosting.co.za>
Message-ID: <1272590338.83.0.399619968852.issue107@psf.upfronthosting.co.za>


New submission from Patrick Ting <pcting at thewhitepixel.com>:

So when specifying a build directory through the --bdist-dir param to anything than the default directory:
1. (as expected) a build directory is created, the pyc files are generated in this directory, and then the directory is deleted.
2. (bug) build/ directory in the cwd is always created with the original .py files copied into it.

I'm running python v2.6.4 and setuptools v0.6c9 on ubuntu 9.10 amd64

----------
messages: 515
nosy: pcting
priority: bug
status: unread
title: bdist_egg --bdist-dir param not working quite as expected

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue107>
_______________________________________________

From gary.poster at canonical.com  Fri Apr 30 04:28:33 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Thu, 29 Apr 2010 22:28:33 -0400
Subject: [Distutils] Beta release of zc.buildout 1.5.0
Message-ID: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>

I have made a beta release of zc.buildout 1.5.0. 

Among other changes, this release includes the ability to use zc.buildout with a system Python, if you use the new z3c.recipe.scripts instead of zc.recipe.egg for generating scripts.

Changelog:

http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29

In my experience, working with a system ("non-clean") Python can be fragile in many ways.  However, this release addresses the problems I have encountered so far.

Your feedback is welcome.  I hope to make a final release sometime next week.

Gary

From tseaver at palladion.com  Fri Apr 30 05:53:34 2010
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 29 Apr 2010 23:53:34 -0400
Subject: [Distutils] Buildout for Python 3
In-Reply-To: <F228F4CE-F2C2-47F3-9671-45CDE12A1595@activestate.com>
References: <F228F4CE-F2C2-47F3-9671-45CDE12A1595@activestate.com>
Message-ID: <hrdk7u$e8a$1@dough.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sridhar Ratnakumar wrote:

> I have begun porting PyPM along with some dependent modules (cmdln,
> platinfo) to Python 3. Since I use buildout, I wonder if anyone has
> already taken up the work of porting it to Python 3. If so, would you
> need any help in completing it? If no one is currently doing it, I'd
> be willing to take a stab at working on it myself provided there is
> some assurance of it getting merged with zc.buildout trunk.
> Otherwise, I may have to look at alternative setups
> (virtualenv+pip?).

There is definitely interest, and a little momentum as well:  Lennart
Regebro has just finished disentangling the zope.testing{,.testrunner}
snarl, moving the testrunner out into a new package (zope.testrunner),
and making it possible to run tests with it under Python3.  He also
updated the buildout recipe to use the new testrunner package.  All
these fixes should make it more feasible to test zc.buildout as the
porting occurs.

Oone known issue:  as with the testrunner, buildout is problematic for
porting using the 2to3 style because it has a dependency on itself, and
in particular on the bundled-but-separate zc.recipe.egg. ;(


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvaVD4ACgkQ+gerLs4ltQ5Y4QCgzXztQBoZKtGOGLpHagSGg0ds
gmMAn0kbkd6Rq74XTJhetUZ+3kAQfvzT
=lPKC
-----END PGP SIGNATURE-----


From chrism at plope.com  Fri Apr 30 08:16:24 2010
From: chrism at plope.com (Chris McDonough)
Date: Fri, 30 Apr 2010 02:16:24 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
Message-ID: <1272608184.6129.58.camel@thinko>

I have a relatively simple buildout that is a directory which is also a
virtualenv (the buildout directory was prepped after checkout with
Python 2.4 "virtualenv --no-site-packages ." and the bootstrap was run
using the resulting "bin/python").

Just now, I had to rerun bin/buildout in that existing directory, and I
got this:

[chrism at thinko deformsite]$ bin/buildout 
Getting distribution for 'zc.buildout'.
Got zc.buildout 1.5.0b1.
Upgraded:
  zc.buildout version 1.5.0b1;
restarting.
Generated script '/home/chrism/projects/deformsite/bin/buildout'.
Develop: '/home/chrism/projects/deformsite/src/deformsite'
Develop: '/home/chrism/projects/deformsite/src/deform'
Develop: '/home/chrism/projects/deformsite/src/colander'
Getting distribution for 'zc.recipe.egg'.
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
    from setuptools.extension import Extension, Library
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
    from distutils.core import Extension as _Extension
  File
"/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
line 3, in ?
    import warnings 
ImportError: No module named warnings
An error occured when trying to install zc.recipe.egg 1.2.3b1. Look
above this message for any errors that were output by easy_install.
While:
  Installing.
  Getting section deformsite.
  Initializing section deformsite.
  Installing recipe zc.recipe.egg.
  Getting distribution for 'zc.recipe.egg'.
Error: Couldn't install: zc.recipe.egg 1.2.3b1

I had built a new virtualenv'ed buildout only a few hours before it the
new zc.buildout was released with the most recent releases of
everything.  I had seen a new release of zc.buildout went up, so I
figured I just needed to kinda start from scratch.  So subsequently I
blew everything away and did what I usually do when I want to start
over:

chrism at thinko projects]$ svn co $REPOZE_SVN/buildouts/deformsite
<snip>
Checked out revision 9276.
[chrism at thinko projects]$ cd deformsite
[chrism at thinko deformsite]$ virtualenv --no-site-packages .
New python executable in ./bin/python
Installing setuptools.............done.
[chrism at thinko deformsite]$ bin/python bootstrap.py 
Creating directory '/home/chrism/projects/deformsite/parts'.
Creating directory '/home/chrism/projects/deformsite/eggs'.
Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
Getting distribution for 'setuptools'.
Got setuptools 0.6c11.

But when I went to run it, I got this:

Generated script '/home/chrism/projects/deformsite/bin/buildout'.
[chrism at thinko deformsite]$ bin/buildout 
Traceback (most recent call last):
  File "bin/buildout", line 14, in ?
    import site # imports custom buildout-generated site.py
  File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
661, in ?
    main()
  File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
628, in main
    virtual_install_main_packages()
  File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
546, in virtual_install_main_packages
    f = open(os.path.join(os.path.dirname(__file__), 'orig-prefix.txt'))
IOError: [Errno 2] No such file or directory:
'/home/chrism/projects/deformsite/parts/buildout/orig-prefix.txt'

If you were paying attention, you'd notice that I made the buildout dir
into a virtualenv above, which I've been doing for a long time now to
get actual isolation.  So at that point, I tried to pin zc.buildout and
zc.recipe.egg to an older version by adding the following to my
buildout.cfg:

versions = versions
[versions]
zc.buildout = 1.4.3
zc.recipe.egg = 1.2.2

But when I then did the same dance that blew up above with the version
pins, it now blew up differently:

[chrism at thinko projects]$ cd deformsite
[chrism at thinko deformsite]$ virtualenv --no-site-packages .
New python executable in ./bin/python
Installing setuptools.............done.
[chrism at thinko deformsite]$ bin/python bootstrap.py 
Creating directory '/home/chrism/projects/deformsite/parts'.
Creating directory '/home/chrism/projects/deformsite/eggs'.
Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
Getting distribution for 'setuptools'.
Got setuptools 0.6c11.
Getting distribution for 'zc.buildout==1.4.3'.
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
    from setuptools.extension import Extension, Library
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
    from distutils.core import Extension as _Extension
  File
"/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
line 3, in ?
    import warnings 
ImportError: No module named warnings
An error occured when trying to install zc.buildout 1.4.3. Look above
this message for any errors that were output by easy_install.
While:
  Bootstrapping.
  Getting distribution for 'zc.buildout==1.4.3'.
Error: Couldn't install: zc.buildout 1.4.3

I pdb stepped through this, and it dies when it reexecutes Python via
os.spawnle of sys.executable (not on the initial execution of "python
bootstrap.py"):

[chrism at thinko deformsite]$ bin/python bootstrap.py 
> /home/chrism/projects/deformsite/bootstrap.py(44)?()
-> assert os.spawnle(
(Pdb) l
 39  	
 40  	ws = pkg_resources.working_set
 41  	
 42  	import pdb; pdb.set_trace()
 43  	
 44  ->	assert os.spawnle(
 45  	    os.P_WAIT, sys.executable, sys.executable,
 46  	    '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
 47  	    dict(os.environ,
 48  	         PYTHONPATH=
 49  	         
(Pdb) p sys.executable, cmd, tmpeggs
('/home/chrism/projects/deformsite/bin/python', 'from
setuptools.command.easy_install import main; main()', '/tmp/tmpEduLSX')
(Pdb) p ws.find(pkg_resources.Requirement.parse('setuptools')).location
'/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg'
(Pdb) 

But sure enough, when I run it with a "real" Python instead of the
virtualenv it works.

Likely, the recent developers buildout of don't use virtualenv, so they
won't have seen this behavior.  I don't have the fortitude to debug
buildout right now, and the isolation of a virtualenv keeps us sane.
Ideally, I could change the buildout.cfg somehow to ignore the new
buildout release entirely to get back to a state where buildout could be
driven with a virtualenv-Python.  Any hints?


- C





From chrism at plope.com  Fri Apr 30 08:21:45 2010
From: chrism at plope.com (Chris McDonough)
Date: Fri, 30 Apr 2010 02:21:45 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
Message-ID: <1272608505.6129.59.camel@thinko>

I have a relatively simple buildout that is a directory which is also a
virtualenv (the buildout directory was prepped after checkout with
Python 2.4 "virtualenv --no-site-packages ." and the bootstrap was run
using the resulting "bin/python").

Just now, I had to rerun bin/buildout in that existing directory, and I
got this:

[chrism at thinko deformsite]$ bin/buildout 
Getting distribution for 'zc.buildout'.
Got zc.buildout 1.5.0b1.
Upgraded:
  zc.buildout version 1.5.0b1;
restarting.
Generated script '/home/chrism/projects/deformsite/bin/buildout'.
Develop: '/home/chrism/projects/deformsite/src/deformsite'
Develop: '/home/chrism/projects/deformsite/src/deform'
Develop: '/home/chrism/projects/deformsite/src/colander'
Getting distribution for 'zc.recipe.egg'.
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
    from setuptools.extension import Extension, Library
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
    from distutils.core import Extension as _Extension
  File
"/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
line 3, in ?
    import warnings 
ImportError: No module named warnings
An error occured when trying to install zc.recipe.egg 1.2.3b1. Look
above this message for any errors that were output by easy_install.
While:
  Installing.
  Getting section deformsite.
  Initializing section deformsite.
  Installing recipe zc.recipe.egg.
  Getting distribution for 'zc.recipe.egg'.
Error: Couldn't install: zc.recipe.egg 1.2.3b1

I had built a new virtualenv'ed buildout only a few hours before it the
new zc.buildout was released with the most recent releases of
everything.  I had seen a new release of zc.buildout went up, so I
figured I just needed to kinda start from scratch.  So subsequently I
blew everything away and did what I usually do when I want to start
over:

chrism at thinko projects]$ svn co $REPOZE_SVN/buildouts/deformsite
<snip>
Checked out revision 9276.
[chrism at thinko projects]$ cd deformsite
[chrism at thinko deformsite]$ virtualenv --no-site-packages .
New python executable in ./bin/python
Installing setuptools.............done.
[chrism at thinko deformsite]$ bin/python bootstrap.py 
Creating directory '/home/chrism/projects/deformsite/parts'.
Creating directory '/home/chrism/projects/deformsite/eggs'.
Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
Getting distribution for 'setuptools'.
Got setuptools 0.6c11.

But when I went to run it, I got this:

Generated script '/home/chrism/projects/deformsite/bin/buildout'.
[chrism at thinko deformsite]$ bin/buildout 
Traceback (most recent call last):
  File "bin/buildout", line 14, in ?
    import site # imports custom buildout-generated site.py
  File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
661, in ?
    main()
  File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
628, in main
    virtual_install_main_packages()
  File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
546, in virtual_install_main_packages
    f = open(os.path.join(os.path.dirname(__file__), 'orig-prefix.txt'))
IOError: [Errno 2] No such file or directory:
'/home/chrism/projects/deformsite/parts/buildout/orig-prefix.txt'

If you were paying attention, you'd notice that I made the buildout dir
into a virtualenv above, which I've been doing for a long time now to
get actual isolation.  So at that point, I tried to pin zc.buildout and
zc.recipe.egg to an older version by adding the following to my
buildout.cfg:

versions = versions
[versions]
zc.buildout = 1.4.3
zc.recipe.egg = 1.2.2

But when I then did the same dance that blew up above with the version
pins, it now blew up differently:

[chrism at thinko projects]$ cd deformsite
[chrism at thinko deformsite]$ virtualenv --no-site-packages .
New python executable in ./bin/python
Installing setuptools.............done.
[chrism at thinko deformsite]$ bin/python bootstrap.py 
Creating directory '/home/chrism/projects/deformsite/parts'.
Creating directory '/home/chrism/projects/deformsite/eggs'.
Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
Getting distribution for 'setuptools'.
Got setuptools 0.6c11.
Getting distribution for 'zc.buildout==1.4.3'.
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
    from setuptools.extension import Extension, Library
  File
"/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
    from distutils.core import Extension as _Extension
  File
"/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
line 3, in ?
    import warnings 
ImportError: No module named warnings
An error occured when trying to install zc.buildout 1.4.3. Look above
this message for any errors that were output by easy_install.
While:
  Bootstrapping.
  Getting distribution for 'zc.buildout==1.4.3'.
Error: Couldn't install: zc.buildout 1.4.3

I pdb stepped through this, and it dies when it reexecutes Python via
os.spawnle of sys.executable (not on the initial execution of "python
bootstrap.py"):

[chrism at thinko deformsite]$ bin/python bootstrap.py 
> /home/chrism/projects/deformsite/bootstrap.py(44)?()
-> assert os.spawnle(
(Pdb) l
 39  	
 40  	ws = pkg_resources.working_set
 41  	
 42  	import pdb; pdb.set_trace()
 43  	
 44  ->	assert os.spawnle(
 45  	    os.P_WAIT, sys.executable, sys.executable,
 46  	    '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
 47  	    dict(os.environ,
 48  	         PYTHONPATH=
 49  	         
(Pdb) p sys.executable, cmd, tmpeggs
('/home/chrism/projects/deformsite/bin/python', 'from
setuptools.command.easy_install import main; main()', '/tmp/tmpEduLSX')
(Pdb) p ws.find(pkg_resources.Requirement.parse('setuptools')).location
'/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg'
(Pdb) 

But sure enough, when I run it with a "real" Python instead of the
virtualenv it works.

Likely, the recent developers buildout of don't use virtualenv, so they
won't have seen this behavior.  I don't have the fortitude to debug
buildout right now, and the isolation of a virtualenv keeps us sane.
Ideally, I could change the buildout.cfg somehow to ignore the new
buildout release entirely to get back to a state where buildout could be
driven with a virtualenv-Python.  Any hints?


- C






From rok.garbas at gmail.com  Fri Apr 30 08:55:31 2010
From: rok.garbas at gmail.com (Rok Garbas)
Date: Fri, 30 Apr 2010 08:55:31 +0200
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <1272608184.6129.58.camel@thinko>
References: <1272608184.6129.58.camel@thinko>
Message-ID: <t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>

run buildout with -v1.4.3 ... this fixed it for me.... after 1h ofcourse :P

2010/4/30 Chris McDonough <chrism at plope.com>:
> I have a relatively simple buildout that is a directory which is also a
> virtualenv (the buildout directory was prepped after checkout with
> Python 2.4 "virtualenv --no-site-packages ." and the bootstrap was run
> using the resulting "bin/python").
>
> Just now, I had to rerun bin/buildout in that existing directory, and I
> got this:
>
> [chrism at thinko deformsite]$ bin/buildout
> Getting distribution for 'zc.buildout'.
> Got zc.buildout 1.5.0b1.
> Upgraded:
> ?zc.buildout version 1.5.0b1;
> restarting.
> Generated script '/home/chrism/projects/deformsite/bin/buildout'.
> Develop: '/home/chrism/projects/deformsite/src/deformsite'
> Develop: '/home/chrism/projects/deformsite/src/deform'
> Develop: '/home/chrism/projects/deformsite/src/colander'
> Getting distribution for 'zc.recipe.egg'.
> Traceback (most recent call last):
> ?File "<string>", line 1, in ?
> ?File
> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
> ? ?from setuptools.extension import Extension, Library
> ?File
> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
> ? ?from distutils.core import Extension as _Extension
> ?File
> "/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
> line 3, in ?
> ? ?import warnings
> ImportError: No module named warnings
> An error occured when trying to install zc.recipe.egg 1.2.3b1. Look
> above this message for any errors that were output by easy_install.
> While:
> ?Installing.
> ?Getting section deformsite.
> ?Initializing section deformsite.
> ?Installing recipe zc.recipe.egg.
> ?Getting distribution for 'zc.recipe.egg'.
> Error: Couldn't install: zc.recipe.egg 1.2.3b1
>
> I had built a new virtualenv'ed buildout only a few hours before it the
> new zc.buildout was released with the most recent releases of
> everything. ?I had seen a new release of zc.buildout went up, so I
> figured I just needed to kinda start from scratch. ?So subsequently I
> blew everything away and did what I usually do when I want to start
> over:
>
> chrism at thinko projects]$ svn co $REPOZE_SVN/buildouts/deformsite
> <snip>
> Checked out revision 9276.
> [chrism at thinko projects]$ cd deformsite
> [chrism at thinko deformsite]$ virtualenv --no-site-packages .
> New python executable in ./bin/python
> Installing setuptools.............done.
> [chrism at thinko deformsite]$ bin/python bootstrap.py
> Creating directory '/home/chrism/projects/deformsite/parts'.
> Creating directory '/home/chrism/projects/deformsite/eggs'.
> Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
> Getting distribution for 'setuptools'.
> Got setuptools 0.6c11.
>
> But when I went to run it, I got this:
>
> Generated script '/home/chrism/projects/deformsite/bin/buildout'.
> [chrism at thinko deformsite]$ bin/buildout
> Traceback (most recent call last):
> ?File "bin/buildout", line 14, in ?
> ? ?import site # imports custom buildout-generated site.py
> ?File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
> 661, in ?
> ? ?main()
> ?File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
> 628, in main
> ? ?virtual_install_main_packages()
> ?File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
> 546, in virtual_install_main_packages
> ? ?f = open(os.path.join(os.path.dirname(__file__), 'orig-prefix.txt'))
> IOError: [Errno 2] No such file or directory:
> '/home/chrism/projects/deformsite/parts/buildout/orig-prefix.txt'
>
> If you were paying attention, you'd notice that I made the buildout dir
> into a virtualenv above, which I've been doing for a long time now to
> get actual isolation. ?So at that point, I tried to pin zc.buildout and
> zc.recipe.egg to an older version by adding the following to my
> buildout.cfg:
>
> versions = versions
> [versions]
> zc.buildout = 1.4.3
> zc.recipe.egg = 1.2.2
>
> But when I then did the same dance that blew up above with the version
> pins, it now blew up differently:
>
> [chrism at thinko projects]$ cd deformsite
> [chrism at thinko deformsite]$ virtualenv --no-site-packages .
> New python executable in ./bin/python
> Installing setuptools.............done.
> [chrism at thinko deformsite]$ bin/python bootstrap.py
> Creating directory '/home/chrism/projects/deformsite/parts'.
> Creating directory '/home/chrism/projects/deformsite/eggs'.
> Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
> Getting distribution for 'setuptools'.
> Got setuptools 0.6c11.
> Getting distribution for 'zc.buildout==1.4.3'.
> Traceback (most recent call last):
> ?File "<string>", line 1, in ?
> ?File
> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
> ? ?from setuptools.extension import Extension, Library
> ?File
> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
> ? ?from distutils.core import Extension as _Extension
> ?File
> "/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
> line 3, in ?
> ? ?import warnings
> ImportError: No module named warnings
> An error occured when trying to install zc.buildout 1.4.3. Look above
> this message for any errors that were output by easy_install.
> While:
> ?Bootstrapping.
> ?Getting distribution for 'zc.buildout==1.4.3'.
> Error: Couldn't install: zc.buildout 1.4.3
>
> I pdb stepped through this, and it dies when it reexecutes Python via
> os.spawnle of sys.executable (not on the initial execution of "python
> bootstrap.py"):
>
> [chrism at thinko deformsite]$ bin/python bootstrap.py
>> /home/chrism/projects/deformsite/bootstrap.py(44)?()
> -> assert os.spawnle(
> (Pdb) l
> ?39
> ?40 ? ? ws = pkg_resources.working_set
> ?41
> ?42 ? ? import pdb; pdb.set_trace()
> ?43
> ?44 ?-> assert os.spawnle(
> ?45 ? ? ? ? os.P_WAIT, sys.executable, sys.executable,
> ?46 ? ? ? ? '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
> ?47 ? ? ? ? dict(os.environ,
> ?48 ? ? ? ? ? ? ?PYTHONPATH=
> ?49
> (Pdb) p sys.executable, cmd, tmpeggs
> ('/home/chrism/projects/deformsite/bin/python', 'from
> setuptools.command.easy_install import main; main()', '/tmp/tmpEduLSX')
> (Pdb) p ws.find(pkg_resources.Requirement.parse('setuptools')).location
> '/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg'
> (Pdb)
>
> But sure enough, when I run it with a "real" Python instead of the
> virtualenv it works.
>
> Likely, the recent developers buildout of don't use virtualenv, so they
> won't have seen this behavior. ?I don't have the fortitude to debug
> buildout right now, and the isolation of a virtualenv keeps us sane.
> Ideally, I could change the buildout.cfg somehow to ignore the new
> buildout release entirely to get back to a state where buildout could be
> driven with a virtualenv-Python. ?Any hints?
>
>
> - C
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Rok Garbas, Python.Zope.Plone consulting
web: http://garbas.si
phone(si): +386 70 707 300
phone(es): +34 68 941 79 62
email: rok at garbas.si

From rok.garbas at gmail.com  Fri Apr 30 08:57:43 2010
From: rok.garbas at gmail.com (Rok Garbas)
Date: Fri, 30 Apr 2010 08:57:43 +0200
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>
References: <1272608184.6129.58.camel@thinko>
	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>
Message-ID: <u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>

i ment on bootstrap ..

python bootstrap.py -v1.4.3

2010/4/30 Rok Garbas <rok.garbas at gmail.com>:
> run buildout with -v1.4.3 ... this fixed it for me.... after 1h ofcourse :P
>
> 2010/4/30 Chris McDonough <chrism at plope.com>:
>> I have a relatively simple buildout that is a directory which is also a
>> virtualenv (the buildout directory was prepped after checkout with
>> Python 2.4 "virtualenv --no-site-packages ." and the bootstrap was run
>> using the resulting "bin/python").
>>
>> Just now, I had to rerun bin/buildout in that existing directory, and I
>> got this:
>>
>> [chrism at thinko deformsite]$ bin/buildout
>> Getting distribution for 'zc.buildout'.
>> Got zc.buildout 1.5.0b1.
>> Upgraded:
>> ?zc.buildout version 1.5.0b1;
>> restarting.
>> Generated script '/home/chrism/projects/deformsite/bin/buildout'.
>> Develop: '/home/chrism/projects/deformsite/src/deformsite'
>> Develop: '/home/chrism/projects/deformsite/src/deform'
>> Develop: '/home/chrism/projects/deformsite/src/colander'
>> Getting distribution for 'zc.recipe.egg'.
>> Traceback (most recent call last):
>> ?File "<string>", line 1, in ?
>> ?File
>> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
>> ? ?from setuptools.extension import Extension, Library
>> ?File
>> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
>> ? ?from distutils.core import Extension as _Extension
>> ?File
>> "/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
>> line 3, in ?
>> ? ?import warnings
>> ImportError: No module named warnings
>> An error occured when trying to install zc.recipe.egg 1.2.3b1. Look
>> above this message for any errors that were output by easy_install.
>> While:
>> ?Installing.
>> ?Getting section deformsite.
>> ?Initializing section deformsite.
>> ?Installing recipe zc.recipe.egg.
>> ?Getting distribution for 'zc.recipe.egg'.
>> Error: Couldn't install: zc.recipe.egg 1.2.3b1
>>
>> I had built a new virtualenv'ed buildout only a few hours before it the
>> new zc.buildout was released with the most recent releases of
>> everything. ?I had seen a new release of zc.buildout went up, so I
>> figured I just needed to kinda start from scratch. ?So subsequently I
>> blew everything away and did what I usually do when I want to start
>> over:
>>
>> chrism at thinko projects]$ svn co $REPOZE_SVN/buildouts/deformsite
>> <snip>
>> Checked out revision 9276.
>> [chrism at thinko projects]$ cd deformsite
>> [chrism at thinko deformsite]$ virtualenv --no-site-packages .
>> New python executable in ./bin/python
>> Installing setuptools.............done.
>> [chrism at thinko deformsite]$ bin/python bootstrap.py
>> Creating directory '/home/chrism/projects/deformsite/parts'.
>> Creating directory '/home/chrism/projects/deformsite/eggs'.
>> Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
>> Getting distribution for 'setuptools'.
>> Got setuptools 0.6c11.
>>
>> But when I went to run it, I got this:
>>
>> Generated script '/home/chrism/projects/deformsite/bin/buildout'.
>> [chrism at thinko deformsite]$ bin/buildout
>> Traceback (most recent call last):
>> ?File "bin/buildout", line 14, in ?
>> ? ?import site # imports custom buildout-generated site.py
>> ?File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
>> 661, in ?
>> ? ?main()
>> ?File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
>> 628, in main
>> ? ?virtual_install_main_packages()
>> ?File "/home/chrism/projects/deformsite/parts/buildout/site.py", line
>> 546, in virtual_install_main_packages
>> ? ?f = open(os.path.join(os.path.dirname(__file__), 'orig-prefix.txt'))
>> IOError: [Errno 2] No such file or directory:
>> '/home/chrism/projects/deformsite/parts/buildout/orig-prefix.txt'
>>
>> If you were paying attention, you'd notice that I made the buildout dir
>> into a virtualenv above, which I've been doing for a long time now to
>> get actual isolation. ?So at that point, I tried to pin zc.buildout and
>> zc.recipe.egg to an older version by adding the following to my
>> buildout.cfg:
>>
>> versions = versions
>> [versions]
>> zc.buildout = 1.4.3
>> zc.recipe.egg = 1.2.2
>>
>> But when I then did the same dance that blew up above with the version
>> pins, it now blew up differently:
>>
>> [chrism at thinko projects]$ cd deformsite
>> [chrism at thinko deformsite]$ virtualenv --no-site-packages .
>> New python executable in ./bin/python
>> Installing setuptools.............done.
>> [chrism at thinko deformsite]$ bin/python bootstrap.py
>> Creating directory '/home/chrism/projects/deformsite/parts'.
>> Creating directory '/home/chrism/projects/deformsite/eggs'.
>> Creating directory '/home/chrism/projects/deformsite/develop-eggs'.
>> Getting distribution for 'setuptools'.
>> Got setuptools 0.6c11.
>> Getting distribution for 'zc.buildout==1.4.3'.
>> Traceback (most recent call last):
>> ?File "<string>", line 1, in ?
>> ?File
>> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/__init__.py", line 2, in ?
>> ? ?from setuptools.extension import Extension, Library
>> ?File
>> "/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg/setuptools/extension.py", line 1, in ?
>> ? ?from distutils.core import Extension as _Extension
>> ?File
>> "/home/chrism/projects/deformsite/lib/python2.4/distutils/__init__.py",
>> line 3, in ?
>> ? ?import warnings
>> ImportError: No module named warnings
>> An error occured when trying to install zc.buildout 1.4.3. Look above
>> this message for any errors that were output by easy_install.
>> While:
>> ?Bootstrapping.
>> ?Getting distribution for 'zc.buildout==1.4.3'.
>> Error: Couldn't install: zc.buildout 1.4.3
>>
>> I pdb stepped through this, and it dies when it reexecutes Python via
>> os.spawnle of sys.executable (not on the initial execution of "python
>> bootstrap.py"):
>>
>> [chrism at thinko deformsite]$ bin/python bootstrap.py
>>> /home/chrism/projects/deformsite/bootstrap.py(44)?()
>> -> assert os.spawnle(
>> (Pdb) l
>> ?39
>> ?40 ? ? ws = pkg_resources.working_set
>> ?41
>> ?42 ? ? import pdb; pdb.set_trace()
>> ?43
>> ?44 ?-> assert os.spawnle(
>> ?45 ? ? ? ? os.P_WAIT, sys.executable, sys.executable,
>> ?46 ? ? ? ? '-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
>> ?47 ? ? ? ? dict(os.environ,
>> ?48 ? ? ? ? ? ? ?PYTHONPATH=
>> ?49
>> (Pdb) p sys.executable, cmd, tmpeggs
>> ('/home/chrism/projects/deformsite/bin/python', 'from
>> setuptools.command.easy_install import main; main()', '/tmp/tmpEduLSX')
>> (Pdb) p ws.find(pkg_resources.Requirement.parse('setuptools')).location
>> '/home/chrism/projects/deformsite/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg'
>> (Pdb)
>>
>> But sure enough, when I run it with a "real" Python instead of the
>> virtualenv it works.
>>
>> Likely, the recent developers buildout of don't use virtualenv, so they
>> won't have seen this behavior. ?I don't have the fortitude to debug
>> buildout right now, and the isolation of a virtualenv keeps us sane.
>> Ideally, I could change the buildout.cfg somehow to ignore the new
>> buildout release entirely to get back to a state where buildout could be
>> driven with a virtualenv-Python. ?Any hints?
>>
>>
>> - C
>>
>>
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
> Rok Garbas, Python.Zope.Plone consulting
> web: http://garbas.si
> phone(si): +386 70 707 300
> phone(es): +34 68 941 79 62
> email: rok at garbas.si
>



-- 
Rok Garbas, Python.Zope.Plone consulting
web: http://garbas.si
phone(si): +386 70 707 300
phone(es): +34 68 941 79 62
email: rok at garbas.si

From chrism at plope.com  Fri Apr 30 09:11:19 2010
From: chrism at plope.com (Chris McDonough)
Date: Fri, 30 Apr 2010 03:11:19 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>
References: <1272608184.6129.58.camel@thinko>
	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>
	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>
Message-ID: <1272611479.6129.63.camel@thinko>

On Fri, 2010-04-30 at 08:57 +0200, Rok Garbas wrote:
> i ment on bootstrap ..
> 
> python bootstrap.py -v1.4.3

With the bootstrap.py I had been using (pre --version option
apparently):

 [chrism at thinko deformsite]$ bin/python bootstrap.py -v1.4.3
 Error: Invalid option -1

With the bootstrap.py from the trunk:

 [chrism at thinko deformsite]$ bin/python bootstrap.py -v1.4.3
 Traceback (most recent call last):
   File "bootstrap.py", line 23, in ?
     import os, shutil, sys, tempfile, textwrap, urllib, urllib2
 ImportError: No module named shutil

Any other suggestions?

- C



From chrism at plope.com  Fri Apr 30 09:25:11 2010
From: chrism at plope.com (Chris McDonough)
Date: Fri, 30 Apr 2010 03:25:11 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>
References: <1272608184.6129.58.camel@thinko>
	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>
	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>
Message-ID: <1272612311.6129.70.camel@thinko>

On Fri, 2010-04-30 at 08:57 +0200, Rok Garbas wrote:
> i ment on bootstrap ..
> 
> python bootstrap.py -v1.4.3

Thanks Rok.  With that hint, I was able to work around it in a
virtualenv by:

- Using the bootstrap.py from the 1.4.3 tag
(<http://svn.zope.org/zc.buildout/tags/1.4.3/bootstrap/bootstrap.py?rev=106409&view=markup>)

- Adding the [versions] stanza that pins zc.buildout 1.4.3 and
zc.recipe.egg 1.2.2 to my buildout.cfg ala:

  versions = versions
  [versions]
  zc.buildout = 1.4.3
  zc.recipe.egg = 1.2.2

- Running the above bootstrap.py with the virtualenv python and the
-v1.4.3 argument.

  bin/python bootstrap.py -v1.4.3

- Run bin/buildout

That produces a working environment.

- C




From rok.garbas at gmail.com  Fri Apr 30 09:33:06 2010
From: rok.garbas at gmail.com (Rok Garbas)
Date: Fri, 30 Apr 2010 09:33:06 +0200
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <1272612311.6129.70.camel@thinko>
References: <1272608184.6129.58.camel@thinko>
	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>
	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>
	<1272612311.6129.70.camel@thinko>
Message-ID: <q2r3173b14d1004300033ic151a313j321e621b15af095a@mail.gmail.com>

2010/4/30 Chris McDonough <chrism at plope.com>:
> On Fri, 2010-04-30 at 08:57 +0200, Rok Garbas wrote:
>> i ment on bootstrap ..
>>
>> python bootstrap.py -v1.4.3
>
> Thanks Rok. ?With that hint, I was able to work around it in a
> virtualenv by:
>
> - Using the bootstrap.py from the 1.4.3 tag
> (<http://svn.zope.org/zc.buildout/tags/1.4.3/bootstrap/bootstrap.py?rev=106409&view=markup>)
>
> - Adding the [versions] stanza that pins zc.buildout 1.4.3 and
> zc.recipe.egg 1.2.2 to my buildout.cfg ala:
>
> ?versions = versions
> ?[versions]
> ?zc.buildout = 1.4.3
> ?zc.recipe.egg = 1.2.2
>
> - Running the above bootstrap.py with the virtualenv python and the
> -v1.4.3 argument.
>
> ?bin/python bootstrap.py -v1.4.3
>
> - Run bin/buildout
>
> That produces a working environment.
>
> - C
>

oh .. right .. i had pinned zc.buildout as well.
i was already thinking to start everything to do by hand ... yeah
right ... then i woke up :)


-- 
Rok Garbas, Python.Zope.Plone consulting
web: http://garbas.si
phone(si): +386 70 707 300
phone(es): +34 68 941 79 62
email: rok at garbas.si

From kiorky at cryptelium.net  Fri Apr 30 09:50:29 2010
From: kiorky at cryptelium.net (kiorky)
Date: Fri, 30 Apr 2010 09:50:29 +0200
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <q2r3173b14d1004300033ic151a313j321e621b15af095a@mail.gmail.com>
References: <1272608184.6129.58.camel@thinko>	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>	<1272612311.6129.70.camel@thinko>
	<q2r3173b14d1004300033ic151a313j321e621b15af095a@mail.gmail.com>
Message-ID: <4BDA8BC5.5030809@cryptelium.net>

Le 30/04/2010 09:33, Rok Garbas a ?crit :
> 2010/4/30 Chris McDonough<chrism at plope.com>:
>    
>> On Fri, 2010-04-30 at 08:57 +0200, Rok Garbas wrote:
>>      
>>> i ment on bootstrap ..
>>>
>>> python bootstrap.py -v1.4.3
>>      
> oh .. right .. i had pinned zc.buildout as well.
> i was already thinking to start everything to do by hand ... yeah
> right ... then i woke up :)
>
>
>    
Test with distribute support (with a recent boostrap.py file 
(svn.zope.org)):

     python bootstrap.py -d


-- 
Cordialement,
kiorky
GPG Fingerprint:


From mouadino at gmail.com  Fri Apr 30 10:08:54 2010
From: mouadino at gmail.com (mouad ben)
Date: Fri, 30 Apr 2010 10:08:54 +0200
Subject: [Distutils] how to make pip do install using a requirements-files
	in order?
Message-ID: <w2o74a7dea51004300108n2f859e19te5a1b1f55a74ebf5@mail.gmail.com>

I use pip with distribute and virtualenv to make an automatic install of a
Django project .
my problem is that i want to make the installation in the order that they
came in the requirements-files that i specified.

i use this command : pip install  -E gold/ -r gold/gold_package.txt

and the gold_package.txt   :
 -e svn+https://test/third-party/djangologging#egg=djangologging
 svn+https://test/spidertools/trunk#egg=spidertools
 -e svn+https://test/spider/branches/package#egg=spider
 -e svn+https://test/lib/trunk#egg=lib

so it should install djangologging -> spidertools -> spider -> lib , because
i have some dependencies that i can't make in the setup.py right now .


Best regards
Benchchaoui Mouad
trainer in www.foobaria.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20100430/eed845cc/attachment-0001.html>

From carl at oddbird.net  Fri Apr 30 14:22:17 2010
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 30 Apr 2010 08:22:17 -0400
Subject: [Distutils] how to make pip do install using a
 requirements-files in order?
In-Reply-To: <w2o74a7dea51004300108n2f859e19te5a1b1f55a74ebf5@mail.gmail.com>
References: <w2o74a7dea51004300108n2f859e19te5a1b1f55a74ebf5@mail.gmail.com>
Message-ID: <4BDACB79.3000109@oddbird.net>

Hi,

mouad ben wrote:
> I use pip with distribute and virtualenv to make an automatic install of
> a Django project .
> my problem is that i want to make the installation in the order that
> they came in the requirements-files that i specified.
> 
> i use this command : pip install  -E gold/ -r gold/gold_package.txt 
> 
> and the gold_package.txt   :
>  -e svn+https://test/third-party/djangologging#egg=djangologging
>  svn+https://test/spidertools/trunk#egg=spidertools
>  -e svn+https://test/spider/branches/package#egg=spider
>  -e svn+https://test/lib/trunk#egg=lib
> 
> so it should install djangologging -> spidertools -> spider -> lib ,
> because i have some dependencies that i can't make in the setup.py right
> now .

There is no way to control the order packages in a requirements file are
installed. I'm not sure of the correct parsing of your final sentence,
but if you mean that some of these packages have dependencies listed in
setup.py that you don't want pip to try to fulfill automatically, I
recommend using the --no-deps option.

Carl

From chris at simplistix.co.uk  Fri Apr 30 14:58:18 2010
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 30 Apr 2010 13:58:18 +0100
Subject: [Distutils] building with static dependencies
In-Reply-To: <3D45B04E-A8C3-4575-98D4-BC4DC36E0C1F@activestate.com>
References: <4BD834A5.8070504@simplistix.co.uk>
	<3D45B04E-A8C3-4575-98D4-BC4DC36E0C1F@activestate.com>
Message-ID: <4BDAD3EA.70705@simplistix.co.uk>

Sridhar Ratnakumar wrote:
> Similar to "STATICDEPS=True python setup.py bdist_egg" in lxml?

If that lets me build and egg that I can then shove onto another box of 
the same OS and install with easy_install without installing any other 
dependencies, then yes, sure :-)

> and build atlas yourself on linux. 

Is there really no simpler way than following the huge and convolute 
process described here:

http://www.scipy.org/Installing_SciPy/Linux#head-eecf834fad12bf7a625752528547588a93f8263c

> In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.

...which is why I guess you can't rely on the operating system supplied 
ATLAS libraries if you want them to be statically linked?

> To statically link libgfortran: on Mac, take a look at the libgfotran.a copying hack in pavement.py (scipy trunk); on Linux, I haven't yet figured out a way to reliably link libgfortran statically to numpy.]]

We're on Windows, Linux and maybe Mac...

cheers,

Chris


From chris at simplistix.co.uk  Fri Apr 30 15:26:08 2010
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 30 Apr 2010 14:26:08 +0100
Subject: [Distutils] building with static dependencies
In-Reply-To: <3D45B04E-A8C3-4575-98D4-BC4DC36E0C1F@activestate.com>
References: <4BD834A5.8070504@simplistix.co.uk>
	<3D45B04E-A8C3-4575-98D4-BC4DC36E0C1F@activestate.com>
Message-ID: <4BDADA70.8030006@simplistix.co.uk>

Sridhar Ratnakumar wrote:
> For numpy you only need to use the pre-built ATLAS libraries (see scipy installation document) on windows 32-bit (for 64-bit, you will need the proprietary MKL to build atlas), and build atlas yourself on linux. In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.

As another tack, I tried, as I'm on Ubuntu 8.04LTS:

sudo apt-get build-dep python-numpy

bin/easy_install bin/easy_install numpy

...which left me with:

Installed .../lib/python2.5/site-packages/numpy-1.4.1-py2.5-linux-x86_64.egg
Processing dependencies for numpy
Finished processing dependencies for numpy
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
   File "/usr/lib/python2.5/atexit.py", line 24, in _run_exitfuncs
     func(*targs, **kargs)
   File 
"/tmp/easy_install-TFDAD2/numpy-1.4.1/numpy/distutils/misc_util.py", 
line 248, in clean_up_temporary_directory
SystemError: Parent module 'numpy.distutils' not loaded
Error in sys.exitfunc:
Traceback (most recent call last):
   File "/usr/lib/python2.5/atexit.py", line 24, in _run_exitfuncs
     func(*targs, **kargs)
   File 
"/tmp/easy_install-TFDAD2/numpy-1.4.1/numpy/distutils/misc_util.py", 
line 248, in clean_up_temporary_directory
SystemError: Parent module 'numpy.distutils' not loaded

...and yet:

$ bin/python
Python 2.5.2 (r252:60911, Jan 20 2010, 23:14:04)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy
 >>>

Any idea what those weird atexit handlers are supposed to do?!

Chris

PS: I would ask on the numpy or scipy mailing list if I could ever 
succeed in joining them...


From gary.poster at canonical.com  Fri Apr 30 16:32:36 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Fri, 30 Apr 2010 10:32:36 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <1272612311.6129.70.camel@thinko>
References: <1272608184.6129.58.camel@thinko>
	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>
	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>
	<1272612311.6129.70.camel@thinko>
Message-ID: <96BE2898-A46A-4A1C-A3CD-BD2058E7A1D4@canonical.com>


On Apr 30, 2010, at 3:25 AM, Chris McDonough wrote:

> On Fri, 2010-04-30 at 08:57 +0200, Rok Garbas wrote:
>> i ment on bootstrap ..
>> 
>> python bootstrap.py -v1.4.3
> 
> Thanks Rok.  With that hint, I was able to work around it in a
> virtualenv by:
> 
> - Using the bootstrap.py from the 1.4.3 tag
> (<http://svn.zope.org/zc.buildout/tags/1.4.3/bootstrap/bootstrap.py?rev=106409&view=markup>)
> 
> - Adding the [versions] stanza that pins zc.buildout 1.4.3 and
> zc.recipe.egg 1.2.2 to my buildout.cfg ala:
> 
>  versions = versions
>  [versions]
>  zc.buildout = 1.4.3
>  zc.recipe.egg = 1.2.2
> 
> - Running the above bootstrap.py with the virtualenv python and the
> -v1.4.3 argument.
> 
>  bin/python bootstrap.py -v1.4.3
> 
> - Run bin/buildout
> 
> That produces a working environment.

Thanks for the clear instructions for the work-around, Chris and Rok.

While one of the goals of the new buildout is to make virtualenv unnecessary with it if desired, I think it is a priority that virtualenv and zc.buildout continue to be able to work together.  I'll investigate this report today.

Gary

From jon at multani.info  Fri Apr 30 17:16:05 2010
From: jon at multani.info (Jonathan Ballet)
Date: Fri, 30 Apr 2010 10:16:05 -0500
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
Message-ID: <14cd86b0bed6f8ca01af40a27bd87da7@mail.webfaction.com>

Hi,

On Thu, 29 Apr 2010 22:28:33 -0400, Gary Poster
<gary.poster at canonical.com>
wrote:
> I have made a beta release of zc.buildout 1.5.0. 
> 
> Among other changes, this release includes the ability to use
zc.buildout
> with a system Python, if you use the new z3c.recipe.scripts instead of
> zc.recipe.egg for generating scripts.
> 
> Changelog:
> 
> http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29

Additional comments, with respects to the mail of Chris McDonough (I hit
the same kind of issues).


1. It looks like something has changed in the way bootstrap run, in
relation to the "buildout:index" value in buildout.cfg. Here, we use an
internal Pypi index, which points to a list of *our* packages, and so, the
value of "buildout:index" refers to this server.
However, it seems now that bootstrap take this value to look for
setuptools/distribute packages while bootstrapping. In our configuration,
this fails with the following error:

$ python bootstrap.py 
Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/bin'.
Creating directory
'/home/jon/projects/sa/repos/unstable/sact.dbapi/parts'.
Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/eggs'.
Creating directory
'/home/jon/projects/sa/repos/unstable/sact.dbapi/develop-eggs'.
Couldn't find index page for 'distribute' (maybe misspelled?)
Getting distribution for 'distribute'.
While:
  Bootstrapping.
  Getting distribution for 'distribute'.
Error: Couldn't find a distribution for 'distribute'.

.. since our server doesn't contain this package.
If I create a ~/.pydistutils.cfg, with "easy_install:index-url" value to
the official Pypi, everything goes fine (bootstrap + buildout).


2. In the previous point, I omitted to say that we are using an old
bootstrap.py, a little bit customized (with a dirty hack which gets
ez_setup.py on our internal server (sounds familiar? ;))
I tried to use the latest bootstrap.py, and I ran into the same problem as
Chris, since my Python executable was from a Virtualenv.

I dig a little bit into the problem, and it looks like the -S option
bootstrap.py passes to Python executable if it detects that it runs from a
non-isolated environment causes this problem. In fact, Virtualenv currently
only copies (with symlinks) some of the stdlib modules into its environment
(I'm not sure why specifically those), and also add the global stdlib
directory into sys.path.
However, running Virtualenv's Python with the -S remove the global stdlib
from the sys.path! (in particular, it removes /usr/lib/python2.5 on my
machine):

I'm not sure this could help, but this was just a sharing of what I found.

Despite those problems, I'm eager to test further this new release of
Buildout, thanks for your work!

 Jonathan

From rodrigo.moraes at gmail.com  Fri Apr 30 17:26:45 2010
From: rodrigo.moraes at gmail.com (Rodrigo Moraes)
Date: Fri, 30 Apr 2010 12:26:45 -0300
Subject: [Distutils] buildout: bootstrap.py in 1.5.0b1 + python 2.5
Message-ID: <p2oa1f380381004300826uc622578elffee7b23d301ea9d@mail.gmail.com>

Hey,
I'm facing a problem running the bootstrap.py from buildout 1.5.0b1.
The errors are pasted here:

   http://paste.pocoo.org/show/208025/

This error occurs only when I use Python 2.5; with 2.6 it works fine.
Also it happens even if i set the flag --distribute. So for the time
being I'm using the bootstrap from 1.4.3 on python 2.5, as it works
well.

I tried to debug it but I'm not familiar with the libraries and had no
success. Could anybody please confirm that it is not working with
Python 2.5 so that I know that the problem is not in my environment?
:)

thank you,
rodrigo

From rodrigo.moraes at gmail.com  Fri Apr 30 18:08:38 2010
From: rodrigo.moraes at gmail.com (Rodrigo Moraes)
Date: Fri, 30 Apr 2010 13:08:38 -0300
Subject: [Distutils] buildout: bootstrap.py in 1.5.0b1 + python 2.5
In-Reply-To: <p2oa1f380381004300826uc622578elffee7b23d301ea9d@mail.gmail.com>
References: <p2oa1f380381004300826uc622578elffee7b23d301ea9d@mail.gmail.com>
Message-ID: <z2wa1f380381004300908x48fe7f7dwe44afc9f164f2e9d@mail.gmail.com>

After a little more investigation, it seems that the problem is caused
by adding the -S flag to python. After I commented out this part of
the code:

# In order to be more robust in the face of system Pythons, we want to
# run without site-packages loaded.  This is somewhat tricky, in
# particular because Python 2.6's distutils imports site, so starting
# with the -S flag is not sufficient.  However, we'll start with that:
if 'site' in sys.modules:
    # We will restart with python -S.
    args = sys.argv[:]
    args[0:0] = [sys.executable, '-S']
    args = map(quote, args)
    os.execv(sys.executable, args)

...it now works with python 2.5. I'm not sure about the implications
of this change, though.

-- rodrigo

From rodrigo.moraes at gmail.com  Fri Apr 30 18:30:48 2010
From: rodrigo.moraes at gmail.com (Rodrigo Moraes)
Date: Fri, 30 Apr 2010 13:30:48 -0300
Subject: [Distutils] buildout: bootstrap.py in 1.5.0b1 + python 2.5
In-Reply-To: <z2wa1f380381004300908x48fe7f7dwe44afc9f164f2e9d@mail.gmail.com>
References: <p2oa1f380381004300826uc622578elffee7b23d301ea9d@mail.gmail.com>
	<z2wa1f380381004300908x48fe7f7dwe44afc9f164f2e9d@mail.gmail.com>
Message-ID: <k2xa1f380381004300930w27bcef9aw88f6a547baa707f7@mail.gmail.com>

(and now sorry for the spam)

Just for clarification, I could reproduce the error using python 2.6.
I realized later that I had buildout installed in site-packages for
python2.5. Running the bootstrap script with buildout installed causes
a problem. Then I installed buildout on python 2.6 and the same error
happens.

Removing it from site-packages or removing the -S flag from bootstrap
fixes the problem. Should I open a ticket?

-- rodrigo

From gary.poster at canonical.com  Fri Apr 30 18:37:03 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Fri, 30 Apr 2010 12:37:03 -0400
Subject: [Distutils] buildout: bootstrap.py in 1.5.0b1 + python 2.5
In-Reply-To: <k2xa1f380381004300930w27bcef9aw88f6a547baa707f7@mail.gmail.com>
References: <p2oa1f380381004300826uc622578elffee7b23d301ea9d@mail.gmail.com>
	<z2wa1f380381004300908x48fe7f7dwe44afc9f164f2e9d@mail.gmail.com>
	<k2xa1f380381004300930w27bcef9aw88f6a547baa707f7@mail.gmail.com>
Message-ID: <B9617A9B-EF9E-4146-BA19-27B937D27261@canonical.com>


On Apr 30, 2010, at 12:30 PM, Rodrigo Moraes wrote:

> (and now sorry for the spam)
> 
> Just for clarification, I could reproduce the error using python 2.6.
> I realized later that I had buildout installed in site-packages for
> python2.5. Running the bootstrap script with buildout installed causes
> a problem. Then I installed buildout on python 2.6 and the same error
> happens.

Ah-ha!  That makes more sense.  I was pretty sure everything worked on 2.5, since we have significant use of that right now.

> Removing it from site-packages or removing the -S flag from bootstrap
> fixes the problem. Should I open a ticket?

You can (on Launchpad) but it isn't necessary.  I'm working on the bugs I've heard so far.

Thanks

Gary




From pje at telecommunity.com  Fri Apr 30 18:50:48 2010
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 30 Apr 2010 12:50:48 -0400
Subject: [Distutils] [issue107] bdist_egg --bdist-dir param not working
 quite as expected
In-Reply-To: <1272590338.83.0.399619968852.issue107@psf.upfronthosting.c o.za>
References: <1272590338.83.0.399619968852.issue107@psf.upfronthosting.co.za>
	<1272590338.83.0.399619968852.issue107@psf.upfronthosting.co.za>
Message-ID: <20100430165050.B85103A407B@sparrow.telecommunity.com>

At 01:18 AM 4/30/2010 +0000, Patrick Ting wrote:
>So when specifying a build directory through the --bdist-dir param 
>to anything than the default directory:
>1. (as expected) a build directory is created, the pyc files are 
>generated in this directory, and then the directory is deleted.
>2. (bug) build/ directory in the cwd is always created with the 
>original .py files copied into it.

Just for the benefit of anybody tracking this on the mailing list, 
the issue is that setting the build directory in this way only 
changes bdist_egg's build directory, not any other command's build 
directory (such as build_py).  This is an artifact of distutils 
design, where you set options to commands like "build" and "install", 
which are then inherited by subcommands.

The "distutils correct" way to do what the OP is trying to do is therefore:

     setup.py build --build-base=/whatever bdist_whatever

As this will pass the build base onto all the subcommands.  This is 
the same pattern for bdist_egg, bdist_wininst, bdist_dumb, etc.

Maybe distutils2 should have a cleaner way of doing this sort of 
thing, if it's not being built on the same infrastructure.


From m.van.rees at zestsoftware.nl  Fri Apr 30 19:09:14 2010
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Fri, 30 Apr 2010 19:09:14 +0200
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <96BE2898-A46A-4A1C-A3CD-BD2058E7A1D4@canonical.com>
References: <1272608184.6129.58.camel@thinko>	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>	<1272612311.6129.70.camel@thinko>
	<96BE2898-A46A-4A1C-A3CD-BD2058E7A1D4@canonical.com>
Message-ID: <hrf2rq$itu$1@dough.gmane.org>

Op 30-04-10 16:32, Gary Poster schreef:
>
> On Apr 30, 2010, at 3:25 AM, Chris McDonough wrote:
>> - Running the above bootstrap.py with the virtualenv python and the
>> -v1.4.3 argument.
>>
>>   bin/python bootstrap.py -v1.4.3
>>
>> - Run bin/buildout
>>
>> That produces a working environment.
>
> Thanks for the clear instructions for the work-around, Chris and Rok.
>
> While one of the goals of the new buildout is to make virtualenv unnecessary with it if desired, I think it is a priority that virtualenv and zc.buildout continue to be able to work together.  I'll investigate this report today.

For the record, I see the same thing.  I'll post some observations here 
in case they help in debugging.

I have a python in a virtualenv and that one is on the PATH so it gets 
used by default.  It gives the same problems the others were seeing.

Using the system python (Mac 10.6 btw) works.

"python bootstrap.py -v 1.4.3" solves it for me as well, as long as 
zc.buildout is properly pinned in the buildout.cfg, otherwise the next 
bin/buildout run will upgrade itself (when in newest mode), restart and 
give the same problems.  So a buildout.cfg like this normally works:

=====================
[buildout]
versions = versions
parts = somepart

[versions]
zc.buildout = 1.4.3

[somepart]
recipe = zc.recipe.egg
eggs = eolfixer
=====================

Sometimes it still goes wrong though, depending on what I tweak in the 
versions, and whether I run buildout in newest or non-newest mode, and 
how bootstrap has been run.  Sometimes I end up with this in 
bin/buildout, and then only a new bootstrap with -v1.4.3 helps:


===================================================
#!/Users/mauritsvanrees/envs/py24/bin/python2.4 -S

import sys
sys.path[0:0] = [
     '/Users/mauritsvanrees/tmp/foo/parts/buildout',
     ]

...
===================================================

Cheers,

Maurits


From gary.poster at canonical.com  Fri Apr 30 19:40:03 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Fri, 30 Apr 2010 13:40:03 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <hrf2rq$itu$1@dough.gmane.org>
References: <1272608184.6129.58.camel@thinko>	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>	<1272612311.6129.70.camel@thinko>
	<96BE2898-A46A-4A1C-A3CD-BD2058E7A1D4@canonical.com>
	<hrf2rq$itu$1@dough.gmane.org>
Message-ID: <1FFBBF81-087C-44B6-9887-71FF90F54A85@canonical.com>


On Apr 30, 2010, at 1:09 PM, Maurits van Rees wrote:

> Op 30-04-10 16:32, Gary Poster schreef:
>> 
>> On Apr 30, 2010, at 3:25 AM, Chris McDonough wrote:
>>> - Running the above bootstrap.py with the virtualenv python and the
>>> -v1.4.3 argument.
>>> 
>>>  bin/python bootstrap.py -v1.4.3
>>> 
>>> - Run bin/buildout
>>> 
>>> That produces a working environment.
>> 
>> Thanks for the clear instructions for the work-around, Chris and Rok.
>> 
>> While one of the goals of the new buildout is to make virtualenv unnecessary with it if desired, I think it is a priority that virtualenv and zc.buildout continue to be able to work together.  I'll investigate this report today.
> 
> For the record, I see the same thing.  I'll post some observations here in case they help in debugging.
> 
> I have a python in a virtualenv and that one is on the PATH so it gets used by default.  It gives the same problems the others were seeing.
> 
> Using the system python (Mac 10.6 btw) works.

The core problem of the virtualenv integration, from my perspective, is that I expect python -S to give a Python with the stdlib installed.  When you use virtualenv, this is not the case.

In a standard interpreter:

$ python -S
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
>>> import ConfigParser
>>> 

In virtualenv:

$ ../bin/python -S
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
>>> import ConfigParser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named ConfigParser
>>> 

The only solutions to this that I see are the following (choose one):

1. virtualenv fixes the -S flag.  I doubt the virtualenv maintainers care, but if they did, that would be great.

2. zc.buildout and bootstrap special case virtualenv: if virtualenv is sniffed (handwave...), we don't use -S.  This is messy, but it means that virtualenv users get what they want without fuss, and non-virtualenv users can use a system Python without fuss.

3. zc.buildout and bootstrap grow an explicit flag to allow site-packages in (extra points if I can do it with only one flag for both cases).  The downside, in addition to complexity, is that then somebody loses: either virtualenv users have to do something special, or people who want to use a system Python with zc.buildout transparently have to do something special.

Unless virtualenv folks offer an implementation for #1...I'll decide whether to pursue #2 or #3.

Gary

From gary.poster at canonical.com  Fri Apr 30 19:49:21 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Fri, 30 Apr 2010 13:49:21 -0400
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <14cd86b0bed6f8ca01af40a27bd87da7@mail.webfaction.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
	<14cd86b0bed6f8ca01af40a27bd87da7@mail.webfaction.com>
Message-ID: <F4FAE039-33E5-4A9F-9841-6DF6A03463FB@canonical.com>


On Apr 30, 2010, at 11:16 AM, Jonathan Ballet wrote:

> Hi,
> 
> On Thu, 29 Apr 2010 22:28:33 -0400, Gary Poster
> <gary.poster at canonical.com>
> wrote:
>> I have made a beta release of zc.buildout 1.5.0. 
>> 
>> Among other changes, this release includes the ability to use
> zc.buildout
>> with a system Python, if you use the new z3c.recipe.scripts instead of
>> zc.recipe.egg for generating scripts.
>> 
>> Changelog:
>> 
>> http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29
> 
> Additional comments, with respects to the mail of Chris McDonough (I hit
> the same kind of issues).
> 
> 
> 1. It looks like something has changed in the way bootstrap run, in
> relation to the "buildout:index" value in buildout.cfg. Here, we use an
> internal Pypi index, which points to a list of *our* packages, and so, the
> value of "buildout:index" refers to this server.
> However, it seems now that bootstrap take this value to look for
> setuptools/distribute packages while bootstrapping. In our configuration,
> this fails with the following error:
> 
> $ python bootstrap.py 
> Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/bin'.
> Creating directory
> '/home/jon/projects/sa/repos/unstable/sact.dbapi/parts'.
> Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/eggs'.
> Creating directory
> '/home/jon/projects/sa/repos/unstable/sact.dbapi/develop-eggs'.
> Couldn't find index page for 'distribute' (maybe misspelled?)
> Getting distribution for 'distribute'.
> While:
>  Bootstrapping.
>  Getting distribution for 'distribute'.
> Error: Couldn't find a distribution for 'distribute'.
> 
> .. since our server doesn't contain this package.
> If I create a ~/.pydistutils.cfg, with "easy_install:index-url" value to
> the official Pypi, everything goes fine (bootstrap + buildout).

Yes, this is the result of the new feature listed in the changes file for bootstrap.py: "The buildout script generated by bootstrap honors more of the settings in the designated configuration file (e.g., buildout.cfg)."  This matched our desire: for instance, if we want to use our own index to manage the packages needed to build out software , that should include zc.buildout.

When an earlier version of bootstrap did not honor the config file, I was surprised, so I felt this was an improvement, and perhaps even arguably a bugfix.

I'd like to hear input on whether this change should be reverted.  I prefer it, and for the particular instance you describe I'd suggest that you keep zc.buildout and your other dependencies on your own server if you already have that infrastructure, but...I won't push back too hard if folks disagree.

> 2. In the previous point, I omitted to say that we are using an old
> bootstrap.py, a little bit customized (with a dirty hack which gets
> ez_setup.py on our internal server (sounds familiar? ;))

heh :-)

> I tried to use the latest bootstrap.py, and I ran into the same problem as
> Chris, since my Python executable was from a Virtualenv.
> 
> I dig a little bit into the problem, and it looks like the -S option
> bootstrap.py passes to Python executable if it detects that it runs from a
> non-isolated environment causes this problem. In fact, Virtualenv currently
> only copies (with symlinks) some of the stdlib modules into its environment
> (I'm not sure why specifically those), and also add the global stdlib
> directory into sys.path.
> However, running Virtualenv's Python with the -S remove the global stdlib
> from the sys.path! (in particular, it removes /usr/lib/python2.5 on my
> machine):
> 
> I'm not sure this could help, but this was just a sharing of what I found.

Yes, thank you.  As I confirmed in another thread, this is exactly the problem.  I didn't understand the importance of what you wrote until I saw it myself.

> Despite those problems, I'm eager to test further this new release of
> Buildout, thanks for your work!

Thank you for your feedback.

Gary


From sridharr at activestate.com  Fri Apr 30 21:36:52 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Fri, 30 Apr 2010 12:36:52 -0700
Subject: [Distutils] building with static dependencies
In-Reply-To: <4BDAD3EA.70705@simplistix.co.uk>
References: <4BD834A5.8070504@simplistix.co.uk>
	<3D45B04E-A8C3-4575-98D4-BC4DC36E0C1F@activestate.com>
	<4BDAD3EA.70705@simplistix.co.uk>
Message-ID: <6B1D63DF-7B9D-4EAB-BD08-B321EA5B2E19@activestate.com>


On 2010-04-30, at 5:58 AM, Chris Withers wrote:

> Sridhar Ratnakumar wrote:
>> Similar to "STATICDEPS=True python setup.py bdist_egg" in lxml?
> 
> If that lets me build and egg that I can then shove onto another box of the same OS and install with easy_install without installing any other dependencies, then yes, sure :-)

Only lxml is aware of the STATICDEPS environment variable (see buildliblxml.py) and builds dependencies with static linking accordingly. AFAIK, no other package does this .. though I imagine someone can work on one of the following:

  1. [PEP] Infrastructure to automatically build dependencies with shared or static linking
  2. A cross-platform Macports/Portage like system to build 'external' dependencies for Python packages (Enthought may have something similar to this, as they have .egg for non-Python packages)

>> and build atlas yourself on linux. 
> 
> Is there really no simpler way than following the huge and convolute process described here:
> 
> http://www.scipy.org/Installing_SciPy/Linux#head-eecf834fad12bf7a625752528547588a93f8263c

Yes, no other simpler documented process that I know of. The ActiveState PyPM repository does come with scipy/numpy linked with the ATLAS libraries (except Windows 64-bit which apparently does not work with ATLAS yet; and perhaps requires MKL). You will need the non-free Business Edition subscription to access these modules.  Like virtualenv, you can install these modules on any virtualenv ("pypm -E /tmp/myvenv install scipy numpy") and any Linux/Windows/Mac machine. http://www.activestate.com/business_solutions/compare/ (Try it with the free 'pil' package, if you are interested)

>> In both cases, setup.cfg has to be modified to point to these ATLAS libraries. ATLAS will be statically linked.
> 
> ...which is why I guess you can't rely on the operating system supplied ATLAS libraries if you want them to be statically linked?
> 
>> To statically link libgfortran: on Mac, take a look at the libgfotran.a copying hack in pavement.py (scipy trunk); on Linux, I haven't yet figured out a way to reliably link libgfortran statically to numpy.]]
> 
> We're on Windows, Linux and maybe Mac...

Recently I figured out that to link with libgfortran (GPL) on Linux, you have to use GCC 4.3 or later, rebuild ATLAS with it and use the -static-libgfortran flag to the gfortran compiler. 

-srid


From regebro at gmail.com  Fri Apr 30 21:46:58 2010
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 30 Apr 2010 21:46:58 +0200
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <F4FAE039-33E5-4A9F-9841-6DF6A03463FB@canonical.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com> 
	<14cd86b0bed6f8ca01af40a27bd87da7@mail.webfaction.com>
	<F4FAE039-33E5-4A9F-9841-6DF6A03463FB@canonical.com>
Message-ID: <s2y319e029f1004301246id74ea496i418db8e17d179220@mail.gmail.com>

Another bug:

In code like this:

def setUp(test):
    zc.buildout.testing.buildoutSetUp(test)
    zc.buildout.testing.install_develop('zc.recipe.testrunner', test)
    zc.buildout.testing.install_develop('zc.recipe.egg', test)
    zc.buildout.testing.install('zope.testing', test)
    zc.buildout.testing.install('zope.testrunner', test)
    zc.buildout.testing.install('zope.interface', test)
    zc.buildout.testing.install('zope.exceptions', test)

You get the following error:

Traceback (most recent call last):
  File "/opt/python26/lib/python2.6/doctest.py", line 2120, in setUp
    self._dt_setUp(test)
  File "/home/projects/zc.recipe.testrunner/trunk/src/zc/recipe/testrunner/tests.py",
line 25, in setUp
    zc.buildout.testing.buildoutSetUp(test)
  File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/testing.py",
line 332, in buildoutSetUp
    make_buildout()
  File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/testing.py",
line 272, in make_buildout
    ).bootstrap([])
  File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/buildout.py",
line 373, in bootstrap
    include_site_packages=False,
  File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/easy_install.py",
line 1030, in install
    return installer.install(specs, working_set)
  File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/easy_install.py",
line 843, in install
    for dist in self._get_dist(requirement, ws, self._always_unzip):
  File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/easy_install.py",
line 678, in _get_dist
    raise MissingDistribution(requirement, ws)
MissingDistribution: Couldn't find a distribution for 'distribute'.


Going back to 1.4.3 (and zc.recipe.egg 1.2.2 solves the problem.

-- 
Lennart Regebro: http://regebro.wordpress.com/
Python 3 Porting: http://python-incompatibility.googlecode.com/
+33 661 58 14 64

From carl at oddbird.net  Fri Apr 30 22:34:42 2010
From: carl at oddbird.net (Carl Meyer)
Date: Fri, 30 Apr 2010 16:34:42 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <1FFBBF81-087C-44B6-9887-71FF90F54A85@canonical.com>
References: <1272608184.6129.58.camel@thinko>	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>	<1272612311.6129.70.camel@thinko>	<96BE2898-A46A-4A1C-A3CD-BD2058E7A1D4@canonical.com>	<hrf2rq$itu$1@dough.gmane.org>
	<1FFBBF81-087C-44B6-9887-71FF90F54A85@canonical.com>
Message-ID: <4BDB3EE2.905@oddbird.net>

Hi Gary,

Gary Poster wrote:
> 1. virtualenv fixes the -S flag.  I doubt the virtualenv maintainers
> care, but if they did, that would be great.

Ian can correct if I'm wrong, but AFAIK that would be quite difficult.
Virtualenv is built around a custom site.py; that's where all the magic
happens. Without site.py all you've got is a relocated Python binary
with a bare-bones mini-stdlib. You could place a complete copy of the
stdlib inside each virtualenv; I think any other solution would involve
re-creating virtualenv from the ground up.

There are some interesting alternative approaches for doing virtualenvs,
such as http://github.com/kvbik/rvirtualenv/; but that one also depends
on site.py. I'm not aware of any that don't.

> 2. zc.buildout and bootstrap special case virtualenv: if virtualenv
> is sniffed (handwave...), we don't use -S.  This is messy, but it
> means that virtualenv users get what they want without fuss, and
> non-virtualenv users can use a system Python without fuss.

FWIW, sniffing for virtualenv is quite easy (no hand-waving required!)
Just check for hasattr(sys, 'real_prefix'). Pip uses this, so it's
already depended on outside virtualenv.

Carl

From ziade.tarek at gmail.com  Fri Apr 30 22:38:43 2010
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 30 Apr 2010 22:38:43 +0200
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
Message-ID: <i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>

On Fri, Apr 30, 2010 at 4:28 AM, Gary Poster <gary.poster at canonical.com> wrote:
> I have made a beta release of zc.buildout 1.5.0.
>
> Among other changes, this release includes the ability to use zc.buildout with a system Python, if you use the new z3c.recipe.scripts instead of zc.recipe.egg for generating scripts.
>
> Changelog:
>
> http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29
>
> In my experience, working with a system ("non-clean") Python can be fragile in many ways. ?However, this release addresses the problems I have encountered so far.
>
> Your feedback is welcome. ?I hope to make a final release sometime next week.

There's a big problem in our ecosystem:  PyPI + our various installers
don't have the concept of "beta" release. Any new release is
considered to be the latest release.

Until it's fixed, I think we should avoid pushing any betas to PyPI,
especially zc.buildout, because many buildout instances
out there are getting upgraded automatically. Leading to a wave of
problems when there's an issue in the release.

I propose that we "hide" the beta version on PyPI until all problems
are cleared u, so people don't auto-upgrade their buildouts.

Moreover, zc.buildout is very fragile to all the different setups out
there. We have been doing manual tests in the past, but we should
definitely set up some continuous integration tools. We should try, if
anyone has some time, to implement a few functional tests where
buildout would be used in various environments.

- with/without virtualenv),
- a typical plone setup
- etc.

Last but not least, I think this auto-upgrade feature zc.buildout
should be removed.  I'd be in favor of an explicit update of this
software, rather than having zc.buildout auto-upgrading itself like
this.

Having a system broken because a new version was pushed at PyPI is not
something we should end up with. The auto upgrade feature should be
optional (and disabled by default IMHO, or at least asking the
question at the prompt ?).

Regards
Tarek


> Gary
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
Tarek Ziad? | http://ziade.org

From gary.poster at canonical.com  Fri Apr 30 22:46:39 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Fri, 30 Apr 2010 16:46:39 -0400
Subject: [Distutils] new zc.buildout and virtualenv no longer friends?
In-Reply-To: <4BDB3EE2.905@oddbird.net>
References: <1272608184.6129.58.camel@thinko>	<t2q3173b14d1004292355pcbe9cd39qc99ff9d50c55210b@mail.gmail.com>	<u2z3173b14d1004292357q7c79092fo568546c063ddf871@mail.gmail.com>	<1272612311.6129.70.camel@thinko>	<96BE2898-A46A-4A1C-A3CD-BD2058E7A1D4@canonical.com>	<hrf2rq$itu$1@dough.gmane.org>
	<1FFBBF81-087C-44B6-9887-71FF90F54A85@canonical.com>
	<4BDB3EE2.905@oddbird.net>
Message-ID: <9026BD73-03EA-469F-9B48-EFE8F3FB7CF1@canonical.com>


On Apr 30, 2010, at 4:34 PM, Carl Meyer wrote:

> Hi Gary,

Hi Carl.

> Gary Poster wrote:
>> 1. virtualenv fixes the -S flag.  I doubt the virtualenv maintainers
>> care, but if they did, that would be great.
> 
> Ian can correct if I'm wrong, but AFAIK that would be quite difficult.
> Virtualenv is built around a custom site.py; that's where all the magic
> happens. Without site.py all you've got is a relocated Python binary
> with a bare-bones mini-stdlib. You could place a complete copy of the
> stdlib inside each virtualenv; I think any other solution would involve
> re-creating virtualenv from the ground up.
> 
> There are some interesting alternative approaches for doing virtualenvs,
> such as http://github.com/kvbik/rvirtualenv/; but that one also depends
> on site.py. I'm not aware of any that don't.

My changes to zc.buildout rely on site.py as well, so I understand that it is critical.  However, the interpreter that z3c.recipe.scripts builds uses this trick, and honors -S, so maybe it is possible for virtualenv.  I don't know enough about the details to be confident.

>> 2. zc.buildout and bootstrap special case virtualenv: if virtualenv
>> is sniffed (handwave...), we don't use -S.  This is messy, but it
>> means that virtualenv users get what they want without fuss, and
>> non-virtualenv users can use a system Python without fuss.
> 
> FWIW, sniffing for virtualenv is quite easy (no hand-waving required!)
> Just check for hasattr(sys, 'real_prefix'). Pip uses this, so it's
> already depended on outside virtualenv.

Thank you, cool.

I decided to see if ConfigParser could be imported, since that's what virtualenv is using as a somewhat related internal bellweather.  Then, if virtualenv is changed to honor -S, normal functionality of the zc.buildout code will resume.

Gary

From sridharr at activestate.com  Fri Apr 30 23:01:20 2010
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Fri, 30 Apr 2010 14:01:20 -0700
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
	<i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>
Message-ID: <C3628350-73F6-4F8D-9DFE-1425D81E1BC4@activestate.com>


On 2010-04-30, at 1:38 PM, Tarek Ziad? wrote:

> There's a big problem in our ecosystem:  PyPI + our various installers
> don't have the concept of "beta" release. Any new release is
> considered to be the latest release.
> 
> Until it's fixed, I think we should avoid pushing any betas to PyPI,
> especially zc.buildout, because many buildout instances
> out there are getting upgraded automatically. Leading to a wave of
> problems when there's an issue in the release.
> 
> I propose that we "hide" the beta version on PyPI until all problems
> are cleared u, so people don't auto-upgrade their buildouts.

Hiding a release in PyPI does not seem to make 'easy_install NAME' *not* fetch it, does it?

-srid


From gary.poster at canonical.com  Fri Apr 30 23:02:08 2010
From: gary.poster at canonical.com (Gary Poster)
Date: Fri, 30 Apr 2010 17:02:08 -0400
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com>
	<i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>
Message-ID: <FD3E32CC-130F-452C-80FD-A666C2FC4EDF@canonical.com>


On Apr 30, 2010, at 4:38 PM, Tarek Ziad? wrote:

> On Fri, Apr 30, 2010 at 4:28 AM, Gary Poster <gary.poster at canonical.com> wrote:
>> I have made a beta release of zc.buildout 1.5.0.
>> 
>> Among other changes, this release includes the ability to use zc.buildout with a system Python, if you use the new z3c.recipe.scripts instead of zc.recipe.egg for generating scripts.
>> 
>> Changelog:
>> 
>> http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29
>> 
>> In my experience, working with a system ("non-clean") Python can be fragile in many ways.  However, this release addresses the problems I have encountered so far.
>> 
>> Your feedback is welcome.  I hope to make a final release sometime next week.
> 
> There's a big problem in our ecosystem:  PyPI + our various installers
> don't have the concept of "beta" release. Any new release is
> considered to be the latest release.
> 
> Until it's fixed, I think we should avoid pushing any betas to PyPI,
> especially zc.buildout, because many buildout instances
> out there are getting upgraded automatically. Leading to a wave of
> problems when there's an issue in the release.
> 
> I propose that we "hide" the beta version on PyPI until all problems
> are cleared u, so people don't auto-upgrade their buildouts.

I have done so, for zc.buildout and zc.recipe.egg.  However, merely hiding them is apparently insufficient, in my tests.

I am investigating other options.


> Moreover, zc.buildout is very fragile to all the different setups out
> there. We have been doing manual tests in the past, but we should
> definitely set up some continuous integration tools. We should try, if
> anyone has some time, to implement a few functional tests where
> buildout would be used in various environments.
> 
> - with/without virtualenv),
> - a typical plone setup
> - etc.
> 
> Last but not least, I think this auto-upgrade feature zc.buildout
> should be removed.  I'd be in favor of an explicit update of this
> software, rather than having zc.buildout auto-upgrading itself like
> this.
> 
> Having a system broken because a new version was pushed at PyPI is not
> something we should end up with.

I'm in favor of this general statement, certainly.

> The auto upgrade feature should be
> optional (and disabled by default IMHO, or at least asking the
> question at the prompt ?).
> 
> Regards
> Tarek
> 
> 
>> Gary
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>> 
> 
> 
> 
> -- 
> Tarek Ziad? | http://ziade.org


From jim at zope.com  Fri Apr 30 23:17:50 2010
From: jim at zope.com (Jim Fulton)
Date: Fri, 30 Apr 2010 17:17:50 -0400
Subject: [Distutils] Beta release of zc.buildout 1.5.0
In-Reply-To: <i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>
References: <E5B3BE54-EF91-414F-AE67-DA5F45DF1DB1@canonical.com> 
	<i2v94bdd2611004301338j322c1319hb687de4abbb2fca3@mail.gmail.com>
Message-ID: <l2t1099b90b1004301417ra8a7e965mb61760908a6b6a2d@mail.gmail.com>

On Fri, Apr 30, 2010 at 4:38 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Fri, Apr 30, 2010 at 4:28 AM, Gary Poster <gary.poster at canonical.com> wrote:
>> I have made a beta release of zc.buildout 1.5.0.
>>
>> Among other changes, this release includes the ability to use zc.buildout with a system Python, if you use the new z3c.recipe.scripts instead of zc.recipe.egg for generating scripts.
>>
>> Changelog:
>>
>> http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29
>>
>> In my experience, working with a system ("non-clean") Python can be fragile in many ways. ?However, this release addresses the problems I have encountered so far.
>>
>> Your feedback is welcome. ?I hope to make a final release sometime next week.
>
> There's a big problem in our ecosystem: ?PyPI + our various installers
> don't have the concept of "beta" release. Any new release is
> considered to be the latest release.

buildout does. :)

buildout has a prefer-final option.

Including:

  [buildout]
  prefer-final = true

Will cause buildout to only use final release unless there are
no final releases that satisfy a requirement.

Using this option will cause buildout to automatically downgrade
itself if it previously upgraded to a non-final release. In buildout 2,
prefer-final true will be the default.

> Until it's fixed, I think we should avoid pushing any betas to PyPI,
> especially zc.buildout, because many buildout instances
> out there are getting upgraded automatically. Leading to a wave of
> problems when there's an issue in the release.
>
> I propose that we "hide" the beta version on PyPI until all problems
> are cleared u, so people don't auto-upgrade their buildouts.

Hidden releases still show up in the simple index.

> Moreover, zc.buildout is very fragile to all the different setups out
> there. We have been doing manual tests in the past, but we should
> definitely set up some continuous integration tools. We should try, if
> anyone has some time, to implement a few functional tests where
> buildout would be used in various environments.
>
> - with/without virtualenv),
> - a typical plone setup
> - etc.
>
> Last but not least, I think this auto-upgrade feature zc.buildout
> should be removed. ?I'd be in favor of an explicit update of this
> software, rather than having zc.buildout auto-upgrading itself like
> this.

I find auto update to be useful, but I wouldn't object to a manual
command:

  bin/buildout upgrade

It would be easier to implement than what we have now.

Jim

-- 
Jim Fulton