From fonnesbeck.mailing.lists at gmail.com  Mon Mar  5 14:34:34 2007
From: fonnesbeck.mailing.lists at gmail.com (Christopher Fonnesbeck)
Date: Mon, 5 Mar 2007 08:34:34 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
Message-ID: <563dd7570703050534h277e30a7lf54e0b1ddfc27a3d@mail.gmail.com>

I have been using bdist_mpkg to build Mac distributions of Numpy, Matplotlib
and other scientific programming packages. However, when I use bdist_mpkg to
build  matplotlib, the resulting package is broken. In particular, I get:

In [3]: import pylab
---------------------------------------------------------------------------
exceptions.ImportError                               Traceback (most recent
call last)

/Users/chris/<ipython console>

/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/pylab.py

----> 1 from matplotlib.pylab import *

ImportError: No module named matplotlib.pylab

Sure enough, when I look into the problem, it seems that the __init__.py
file is being left out by distutils. If I install directly using setup.py,
this does not occur.

Any ideas?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070305/c916f26c/attachment.html 

From bob at redivi.com  Mon Mar  5 16:13:57 2007
From: bob at redivi.com (Bob Ippolito)
Date: Mon, 5 Mar 2007 07:13:57 -0800
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <563dd7570703050534h277e30a7lf54e0b1ddfc27a3d@mail.gmail.com>
References: <563dd7570703050534h277e30a7lf54e0b1ddfc27a3d@mail.gmail.com>
Message-ID: <6a36e7290703050713g1549b59fu403d64e9c0ff2557@mail.gmail.com>

On 3/5/07, Christopher Fonnesbeck <fonnesbeck.mailing.lists at gmail.com> wrote:
> I have been using bdist_mpkg to build Mac distributions of Numpy, Matplotlib
> and other scientific programming packages. However, when I use bdist_mpkg to
> build  matplotlib, the resulting package is broken. In particular, I get:
>
> In [3]: import pylab
> ---------------------------------------------------------------------------
> exceptions.ImportError                               Traceback (most recent
> call last)
>
> /Users/chris/<ipython console>
>
> /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/pylab.py
> ----> 1 from matplotlib.pylab import *
>
> ImportError: No module named matplotlib.pylab
>
> Sure enough, when I look into the problem, it seems that the __init__.py
> file is being left out by distutils. If I install directly using setup.py,
> this does not occur.
>
> Any ideas?
>

I've never seen that happen before. Must be something funny matplotlib
is doing...

-bob

From pje at telecommunity.com  Mon Mar  5 17:01:25 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 05 Mar 2007 11:01:25 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <6a36e7290703050713g1549b59fu403d64e9c0ff2557@mail.gmail.co
 m>
References: <563dd7570703050534h277e30a7lf54e0b1ddfc27a3d@mail.gmail.com>
	<563dd7570703050534h277e30a7lf54e0b1ddfc27a3d@mail.gmail.com>
Message-ID: <5.1.1.6.0.20070305110007.028da950@sparrow.telecommunity.com>

At 07:13 AM 3/5/2007 -0800, Bob Ippolito wrote:
>On 3/5/07, Christopher Fonnesbeck <fonnesbeck.mailing.lists at gmail.com> wrote:
> > I have been using bdist_mpkg to build Mac distributions of Numpy, 
> Matplotlib
> > and other scientific programming packages. However, when I use 
> bdist_mpkg to
> > build  matplotlib, the resulting package is broken. In particular, I get:
> >
> > In [3]: import pylab
> > ---------------------------------------------------------------------------
> > exceptions.ImportError                               Traceback (most recent
> > call last)
> >
> > /Users/chris/<ipython console>
> >
> > 
> /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/pylab.py
> > ----> 1 from matplotlib.pylab import *
> >
> > ImportError: No module named matplotlib.pylab
> >
> > Sure enough, when I look into the problem, it seems that the __init__.py
> > file is being left out by distutils. If I install directly using setup.py,
> > this does not occur.
> >
> > Any ideas?
> >
>
>I've never seen that happen before. Must be something funny matplotlib
>is doing...

When a namespace package is installed using a backward-compatibility mode, 
the __init__.py goes away, otherwise you'll have multiple packages 
installing it.  Is bdist_mpkg not including the .pth file that setuptools 
generates to handle this?


From radix at twistedmatrix.com  Mon Mar  5 21:13:49 2007
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Mon, 5 Mar 2007 15:13:49 -0500
Subject: [Distutils] conditionally compiling extensions
Message-ID: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>

Twisted has a few extension modules that it only wants to build if
certain things are available on the platform - like epoll support if
the epoll.h header file is available, for example.

We can call a function to run some code to decide which extensions to
build, but in certain cases having access to the compiler object in
distutils would be preferable -- to test certain things about the
platform. Currently we have some fairly heinous hacks to get our code
to run *after* that compiler object is available (i.e., after it's
constructed in build_ext.run), but these hacks are becoming
problematic especially for easy_install support.

Before showing what these hacks are (if you really care, you can check
the various setup.py files in the Twisted repository), I'd like to ask
if there is any other best-practice knowledge in the community for
conditionally building extensions based on knowledge only available by
using a compiler.

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From pje at telecommunity.com  Mon Mar  5 21:35:16 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Mon, 05 Mar 2007 15:35:16 -0500
Subject: [Distutils] conditionally compiling extensions
In-Reply-To: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.co
 m>
Message-ID: <5.1.1.6.0.20070305152958.02ba7b70@sparrow.telecommunity.com>

At 03:13 PM 3/5/2007 -0500, Christopher Armstrong wrote:
>Twisted has a few extension modules that it only wants to build if
>certain things are available on the platform - like epoll support if
>the epoll.h header file is available, for example.
>
>We can call a function to run some code to decide which extensions to
>build, but in certain cases having access to the compiler object in
>distutils would be preferable -- to test certain things about the
>platform. Currently we have some fairly heinous hacks to get our code
>to run *after* that compiler object is available (i.e., after it's
>constructed in build_ext.run), but these hacks are becoming
>problematic especially for easy_install support.
>
>Before showing what these hacks are (if you really care, you can check
>the various setup.py files in the Twisted repository), I'd like to ask
>if there is any other best-practice knowledge in the community for
>conditionally building extensions based on knowledge only available by
>using a compiler.

I don't know of any better practices, unfortunately -- certainly nothing 
that's available portably at the pure distutils level.

Setuptools has a primitive "features" system that is used by a few PEAK 
packages, but it's undocumented and it suffers from a fatal flaw of not 
noticing that the configuration has changed (and thus requires clean rebuilds).

I've thought about adding a "configure" command in 0.7 with some sort of 
cache facility (and support for wiping out build/ when this information 
changes).  However, the real problem with having any sort of feature/option 
type stuff in setuptools is how to specify egg platform information.  I 
mean, how many flags can we really put in the filename, after all?  :)

I'm definitely interested in solving this and related problems, but I have 
very few ideas about how to do it.


From robert.kern at gmail.com  Mon Mar  5 22:10:09 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 05 Mar 2007 15:10:09 -0600
Subject: [Distutils] conditionally compiling extensions
In-Reply-To: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>
References: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>
Message-ID: <esi0vl$6l7$1@sea.gmane.org>

Christopher Armstrong wrote:
> Twisted has a few extension modules that it only wants to build if
> certain things are available on the platform - like epoll support if
> the epoll.h header file is available, for example.
> 
> We can call a function to run some code to decide which extensions to
> build, but in certain cases having access to the compiler object in
> distutils would be preferable -- to test certain things about the
> platform. Currently we have some fairly heinous hacks to get our code
> to run *after* that compiler object is available (i.e., after it's
> constructed in build_ext.run), but these hacks are becoming
> problematic especially for easy_install support.
> 
> Before showing what these hacks are (if you really care, you can check
> the various setup.py files in the Twisted repository), I'd like to ask
> if there is any other best-practice knowledge in the community for
> conditionally building extensions based on knowledge only available by
> using a compiler.

We've been doing something along these lines in the numpy project. We've
extended the distutils command "config" to allow building and executing small C
programs to test for platform support of certain C functions.

Looking at the code right now ... it's possibly at the same level of heinous
hackishness that yours are, but it works for us. We use it to build a config.h
file using the build_src command which we added. build_src allows us to pass a
callable as an item in an Extension's sources list, and the callable will
generate a file. Inside the function that generates config.h, we grab the config
command, call .ensure_finalized() on it and then call the methods that we added
to try compiling, linking, and/or running the test files. By the time the
functions are called in build_src, the config command exists and is ready to be
used.

You might be able to compile the extensions unconditionally, but through the use
of the #defines in config.h and #ifdefs in the extension module only compile an
empty module if the feature were unsupported. That would turn the

  try:
    import _c_accelerator
    func = _c_accelerator.func
  except ImportError:
    pass

into

  import _c_accelerator
  if hasattr(_c_accelerator, 'func'):
    func = _c_accelerator.func

I leave it to you to decide which is better, but I find try: except ImportError:
to be too fragile. I spent a good amount of time at a customer's site trying to
figure out why Cheetah was unbearably slow behind Apache, but worked like blazes
when we ran the script manually on the same box. It turns out that the
permissions on the accelerator extension module were incorrect for mod_python to
import them.

The setup scripts for Marc-Andr? Lemburg's mx extensions seem to define an
mx_autoconf distutils command that might be useful to look at, but I don't know
the details about how he uses it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From radix at twistedmatrix.com  Mon Mar  5 22:10:59 2007
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Mon, 5 Mar 2007 16:10:59 -0500
Subject: [Distutils] conditionally compiling extensions
In-Reply-To: <5.1.1.6.0.20070305152958.02ba7b70@sparrow.telecommunity.com>
References: <5.1.1.6.0.20070305152958.02ba7b70@sparrow.telecommunity.com>
Message-ID: <60ed19d40703051310q97b1979q57c852a636971624@mail.gmail.com>

On 3/5/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> I'm definitely interested in solving this and related problems, but I have
> very few ideas about how to do it.

So, it seems it would be pretty trivial to do some refactoring of
distutils itself to make the point we need to hook into more hookable.
The heinous hack in Twisted is there because there are many points in
distutils that say "are there any Extensions specified right now? Ok,
then early-out"; but we want it to keep going until the point  that
it's created the Compiler and calls build_extensions, where we
*actually* calculate what Extensions we need to call.

The problem is that even if I make the necessary changes to distutils,
it will be 6 years before I'm able to use them, unless I decide to
package distutils *with* Twisted, which would be a huge PITA.

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From jpellerin at gmail.com  Tue Mar  6 20:29:18 2007
From: jpellerin at gmail.com (jason pellerin)
Date: Tue, 6 Mar 2007 14:29:18 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
Message-ID: <3bb02d620703061129s1a965121i83a54e8a666d072@mail.gmail.com>

> At 07:13 AM 3/5/2007 -0800, Bob Ippolito wrote:
> >On 3/5/07, Christopher Fonnesbeck <fonnesbeck.mailing.lists at gmail.com> wrote:
> > > I have been using bdist_mpkg to build Mac distributions of Numpy,
> > Matplotlib
> > > and other scientific programming packages. However, when I use
> > bdist_mpkg to
> > > build  matplotlib, the resulting package is broken. In particular, I get:
> > >
> > > In [3]: import pylab
> > > ---------------------------------------------------------------------------
> > > exceptions.ImportError                               Traceback (most recent
> > > call last)
> > >
> > > /Users/chris/<ipython console>
> > >
> > >
> > /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/pylab.py
> > > ----> 1 from matplotlib.pylab import *
> > >
> > > ImportError: No module named matplotlib.pylab
> > >
> > > Sure enough, when I look into the problem, it seems that the __init__.py
> > > file is being left out by distutils. If I install directly using setup.py,
> > > this does not occur.
> > >
> > > Any ideas?
> > >
> >
> >I've never seen that happen before. Must be something funny matplotlib
> >is doing...
>
> When a namespace package is installed using a backward-compatibility mode,
> the __init__.py goes away, otherwise you'll have multiple packages
> installing it.  Is bdist_mpkg not including the .pth file that setuptools
> generates to handle this?

I've been meaning to ask about this same issue, specifically with
respect to namespace packages, bdist_rpm and imp. I've had to patch
the spec files for all namespace packages that we build as RPMs, since
although the .pth file works when using import or __import__, imp
doesn't seem to recognize it, and that breaks Paste's app loading. So
I have to have the RPMs put a dummy __init__.py in the package --
which has problems of its own, but at least leaves me with a
functional install.

I'm wondering if it's a known issue that imp can't/doesn't use .pth
files, and if so, is there any other way to handle namespace packages
in --single-version-externally-managed besides .pth files?

JP

From bob at redivi.com  Tue Mar  6 20:43:59 2007
From: bob at redivi.com (Bob Ippolito)
Date: Tue, 6 Mar 2007 11:43:59 -0800
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <5.1.1.6.0.20070305110007.028da950@sparrow.telecommunity.com>
References: <563dd7570703050534h277e30a7lf54e0b1ddfc27a3d@mail.gmail.com>
	<5.1.1.6.0.20070305110007.028da950@sparrow.telecommunity.com>
Message-ID: <6a36e7290703061143wad53888x3078e1238993fbf0@mail.gmail.com>

On 3/5/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 07:13 AM 3/5/2007 -0800, Bob Ippolito wrote:
> >On 3/5/07, Christopher Fonnesbeck <fonnesbeck.mailing.lists at gmail.com> wrote:
> > > I have been using bdist_mpkg to build Mac distributions of Numpy,
> > Matplotlib
> > > and other scientific programming packages. However, when I use
> > bdist_mpkg to
> > > build  matplotlib, the resulting package is broken. In particular, I get:
> > >
> > > In [3]: import pylab
> > > ---------------------------------------------------------------------------
> > > exceptions.ImportError                               Traceback (most recent
> > > call last)
> > >
> > > /Users/chris/<ipython console>
> > >
> > >
> > /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/pylab.py
> > > ----> 1 from matplotlib.pylab import *
> > >
> > > ImportError: No module named matplotlib.pylab
> > >
> > > Sure enough, when I look into the problem, it seems that the __init__.py
> > > file is being left out by distutils. If I install directly using setup.py,
> > > this does not occur.
> > >
> > > Any ideas?
> > >
> >
> >I've never seen that happen before. Must be something funny matplotlib
> >is doing...
>
> When a namespace package is installed using a backward-compatibility mode,
> the __init__.py goes away, otherwise you'll have multiple packages
> installing it.  Is bdist_mpkg not including the .pth file that setuptools
> generates to handle this?
>

It should include whatever is generated. I don't recall the specifics,
and I don't really have time to look at it... but if someone can track
it down to some root cause or provide a patch then I'd see about
fixing it.

It's worked for pretty much everything I've thrown at it the past few
years. Setuptools probably throws a bit of a wrench into the mix, but
I see very little reason to bother with mpkg when there's an egg
available.

-bob

From pje at telecommunity.com  Wed Mar  7 16:48:44 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 07 Mar 2007 10:48:44 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <3bb02d620703061129s1a965121i83a54e8a666d072@mail.gmail.com
 >
Message-ID: <5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>

At 02:29 PM 3/6/2007 -0500, jason pellerin wrote:
>I've been meaning to ask about this same issue, specifically with
>respect to namespace packages, bdist_rpm and imp. I've had to patch
>the spec files for all namespace packages that we build as RPMs, since
>although the .pth file works when using import or __import__, imp
>doesn't seem to recognize it, and that breaks Paste's app loading.

Then Paste's app loading is broken, and wouldn't work with an egg either, 
if it's using "imp" and ignoring the package's __path__.


>So
>I have to have the RPMs put a dummy __init__.py in the package --

Make sure it includes a namespace declaration, or other things will 
break.  And of course you must only include this __init__.py for *one* 
member of the namespace package.


>which has problems of its own, but at least leaves me with a
>functional install.
>
>I'm wondering if it's a known issue that imp can't/doesn't use .pth
>files,

It doesn't - it's on a completely different level of the import 
structure.  However, any sanely-written use of "imp" should respect the 
__path__ that's set up by the .pth file.  So either Paste is broken, or the 
application using Paste is being run with python -s, disabling the import 
of the "site" module, and thereby the activation of .pth files and 
site-packages.  However, if import is working in the same environment (same 
Python options, etc.), then it's Paste that's broken.

>and if so, is there any other way to handle namespace packages
>in --single-version-externally-managed besides .pth files?

I truly wish there were.  But .pth files were the best I could come up with 
to make the silly thing work at all, without requiring every piece of a 
namespace pacakge to depend on a package whose only content is the 
__init__.py file.


From ianb at colorstudy.com  Wed Mar  7 18:38:44 2007
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 07 Mar 2007 11:38:44 -0600
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>
References: <5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>
Message-ID: <45EEF8A4.7030402@colorstudy.com>

Phillip J. Eby wrote:
> At 02:29 PM 3/6/2007 -0500, jason pellerin wrote:
>> I've been meaning to ask about this same issue, specifically with
>> respect to namespace packages, bdist_rpm and imp. I've had to patch
>> the spec files for all namespace packages that we build as RPMs, since
>> although the .pth file works when using import or __import__, imp
>> doesn't seem to recognize it, and that breaks Paste's app loading.
> 
> Then Paste's app loading is broken, and wouldn't work with an egg either, 
> if it's using "imp" and ignoring the package's __path__.

I'm not sure which part of Paste is in question, but the only importing 
that I can think of which it does is with pkg_resources or with __import__.


-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org

From jpellerin at gmail.com  Wed Mar  7 19:38:08 2007
From: jpellerin at gmail.com (jason pellerin)
Date: Wed, 7 Mar 2007 13:38:08 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <45EEF8A4.7030402@colorstudy.com>
References: <5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>
	<45EEF8A4.7030402@colorstudy.com>
Message-ID: <3bb02d620703071038s19a93f65wc77195ab4aa41f49@mail.gmail.com>

On 3/7/07, Ian Bicking <ianb at colorstudy.com> wrote:
> Phillip J. Eby wrote:
> > At 02:29 PM 3/6/2007 -0500, jason pellerin wrote:
> >> I've been meaning to ask about this same issue, specifically with
> >> respect to namespace packages, bdist_rpm and imp. I've had to patch
> >> the spec files for all namespace packages that we build as RPMs, since
> >> although the .pth file works when using import or __import__, imp
> >> doesn't seem to recognize it, and that breaks Paste's app loading.
> >
> > Then Paste's app loading is broken, and wouldn't work with an egg either,
> > if it's using "imp" and ignoring the package's __path__.
>
> I'm not sure which part of Paste is in question, but the only importing
> that I can think of which it does is with pkg_resources or with __import__.

I didn't save the output from trying to use the broken package load,
unfortunately. I'll try to recreate it if I can. I may easily be
putting the blame in the wrong place, though, since the failure was in
paste.modpython -- it could be mod_python that's using imp in some
egg-unfriendly way. If I can find some time today or in the next
couple of days I'll re-break a namespace package and post the output
from a failed load here.

JP

From nathan at yergler.net  Wed Mar  7 20:07:06 2007
From: nathan at yergler.net (Nathan R. Yergler)
Date: Wed, 7 Mar 2007 14:07:06 -0500
Subject: [Distutils] Setting environment variables before building eggs w/
	extensions using zc.buildout
Message-ID: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>

I had a problem come up today that I've run into before, and I'd like
to figure out a way to prevent it from sucking my time down in the
future.  Several apps we run at Creative Commons use lxml
(http://codespeak.net/lxml) for XML, XSLT and XPath processing.  lxml
builds on libxml2, and provides an element tree interface, plus
additional capabilities.

We use zc.buildout to assemble our server-side applications, and that
has generally been a huge sanity saver.  However, lxml can rob some of
that sanity periodically.  If a system like one of our CentOS machines
has an older version of libxml2 installed in /usr/lib, the extension
building process will link against it without complaint.  That is,
until you try to do something added in the later, recommended version
of libxml2 like XSLT.  Once you scratch your head and remember what
the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib,
and then re-run the buildout.

Does zc.buildout currently have any way to set an environment variable
during it's run?  I didn't see anything at
http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if
there was something at the zc.buildout level (as opposed to the recipe
level) that would do the trick.  The quick/dirty thing I'd do is
export the appropriate value of LD_LIBRARY_PATH.  The perhaps more
intelligent thing I'd do is use the cmmi recipe to build a version of
libxml2 that I *know* has the features I need as part of the buildout,
and then point LD_LIBRARY_PATH at that.

Or has anyone else run into a similar situation, who might have some insight?

Thanks,

Nathan

From optilude at gmx.net  Wed Mar  7 20:40:13 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Wed, 07 Mar 2007 19:40:13 +0000
Subject: [Distutils] Setting environment variables before building eggs
 w/ extensions using zc.buildout
In-Reply-To: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>
References: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>
Message-ID: <esn4ev$6sc$1@sea.gmane.org>

Nathan R. Yergler wrote:
> I had a problem come up today that I've run into before, and I'd like
> to figure out a way to prevent it from sucking my time down in the
> future.  Several apps we run at Creative Commons use lxml
> (http://codespeak.net/lxml) for XML, XSLT and XPath processing.  lxml
> builds on libxml2, and provides an element tree interface, plus
> additional capabilities.
> 
> We use zc.buildout to assemble our server-side applications, and that
> has generally been a huge sanity saver.  However, lxml can rob some of
> that sanity periodically.  If a system like one of our CentOS machines
> has an older version of libxml2 installed in /usr/lib, the extension
> building process will link against it without complaint.  That is,
> until you try to do something added in the later, recommended version
> of libxml2 like XSLT.  Once you scratch your head and remember what
> the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib,
> and then re-run the buildout.
> 
> Does zc.buildout currently have any way to set an environment variable
> during it's run?  I didn't see anything at
> http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if
> there was something at the zc.buildout level (as opposed to the recipe
> level) that would do the trick.  The quick/dirty thing I'd do is
> export the appropriate value of LD_LIBRARY_PATH.  The perhaps more
> intelligent thing I'd do is use the cmmi recipe to build a version of
> libxml2 that I *know* has the features I need as part of the buildout,
> and then point LD_LIBRARY_PATH at that.
> 
> Or has anyone else run into a similar situation, who might have some insight?

I would be interested in something like this, specifically for building 
Deliverance.

Deliverance does have a buildout, that attempts to do this, but the 
deliverance buildout is pretty rough. Or rather, it's not easily 
re-usable, so if you want to build deliverance as part of a bigger 
project, it can be tricky.

I'd be interested in seeing and/or helping out with such a solution 
because I need to build deliverance as part of this project anyway, and 
I'd like an elegant way to do it.

See http://www.openplans.org/projects/deliverance and 
http://codespeak.net/svn/z3/deliverance/buildout/trunk/.

Martin


From d.w.morriss at gmail.com  Wed Mar  7 21:14:18 2007
From: d.w.morriss at gmail.com (whit)
Date: Wed, 07 Mar 2007 14:14:18 -0600
Subject: [Distutils] Setting environment variables before building eggs
 w/ extensions using zc.buildout
In-Reply-To: <esn4ev$6sc$1@sea.gmane.org>
References: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>
	<esn4ev$6sc$1@sea.gmane.org>
Message-ID: <esn6eq$f5u$1@sea.gmane.org>

Martin Aspeli wrote:
> Nathan R. Yergler wrote:
>> I had a problem come up today that I've run into before, and I'd like
>> to figure out a way to prevent it from sucking my time down in the
>> future.  Several apps we run at Creative Commons use lxml
>> (http://codespeak.net/lxml) for XML, XSLT and XPath processing.  lxml
>> builds on libxml2, and provides an element tree interface, plus
>> additional capabilities.
>>
>> We use zc.buildout to assemble our server-side applications, and that
>> has generally been a huge sanity saver.  However, lxml can rob some of
>> that sanity periodically.  If a system like one of our CentOS machines
>> has an older version of libxml2 installed in /usr/lib, the extension
>> building process will link against it without complaint.  That is,
>> until you try to do something added in the later, recommended version
>> of libxml2 like XSLT.  Once you scratch your head and remember what
>> the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib,
>> and then re-run the buildout.
>>
>> Does zc.buildout currently have any way to set an environment variable
>> during it's run?  I didn't see anything at
>> http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if
>> there was something at the zc.buildout level (as opposed to the recipe
>> level) that would do the trick.  The quick/dirty thing I'd do is
>> export the appropriate value of LD_LIBRARY_PATH.  The perhaps more
>> intelligent thing I'd do is use the cmmi recipe to build a version of
>> libxml2 that I *know* has the features I need as part of the buildout,
>> and then point LD_LIBRARY_PATH at that.
>>
>> Or has anyone else run into a similar situation, who might have some insight?
> 
> I would be interested in something like this, specifically for building 
> Deliverance.
> 
> Deliverance does have a buildout, that attempts to do this, but the 
> deliverance buildout is pretty rough. Or rather, it's not easily 
> re-usable, so if you want to build deliverance as part of a bigger 
> project, it can be tricky.
> 
> I'd be interested in seeing and/or helping out with such a solution 
> because I need to build deliverance as part of this project anyway, and 
> I'd like an elegant way to do it.
> 
> See http://www.openplans.org/projects/deliverance and 
> http://codespeak.net/svn/z3/deliverance/buildout/trunk/.
> 
> Martin
> 
>

fyi, the deliverance.buildout will soon go away

-w




-- 

------ d. whit morriss ------
- senior engineer, opencore -
- http://www.openplans.org  -
- m: 415-710-8975           -

"If you don't know where you are,
  you don't know anything at all" 


Dr. Edgar Spencer, Ph.D., 1995


From jim at zope.com  Thu Mar  8 16:13:51 2007
From: jim at zope.com (Jim Fulton)
Date: Thu, 8 Mar 2007 10:13:51 -0500
Subject: [Distutils] Setting environment variables before building eggs
	w/ extensions using zc.buildout
In-Reply-To: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>
References: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>
Message-ID: <FE971CCB-F382-4A98-91E5-A779D4A35EEB@zope.com>


On Mar 7, 2007, at 2:07 PM, Nathan R. Yergler wrote:

I meant to reply earlier. Sorry.

> I had a problem come up today that I've run into before, and I'd like
> to figure out a way to prevent it from sucking my time down in the
> future.  Several apps we run at Creative Commons use lxml
> (http://codespeak.net/lxml) for XML, XSLT and XPath processing.  lxml
> builds on libxml2, and provides an element tree interface, plus
> additional capabilities.
>
> We use zc.buildout to assemble our server-side applications, and that
> has generally been a huge sanity saver.  However, lxml can rob some of
> that sanity periodically.  If a system like one of our CentOS machines
> has an older version of libxml2 installed in /usr/lib, the extension
> building process will link against it without complaint.  That is,
> until you try to do something added in the later, recommended version
> of libxml2 like XSLT.  Once you scratch your head and remember what
> the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib,
> and then re-run the buildout.
>
> Does zc.buildout currently have any way to set an environment variable
> during it's run?

No.

>   I didn't see anything at
> http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if
> there was something at the zc.buildout level (as opposed to the recipe
> level) that would do the trick.  The quick/dirty thing I'd do is
> export the appropriate value of LD_LIBRARY_PATH.

I'm a little surprised that this would help at build time.  I'm  
surprised that distutils looks at LD_LIBRARY_PATH when deciding where  
to link from.

Have you looked at:

   http://www.python.org/pypi/zc.recipe.egg#creating-eggs-with- 
extensions-neededing-custom-build-settings

>   The perhaps more
> intelligent thing I'd do is use the cmmi recipe to build a version of
> libxml2 that I *know* has the features I need as part of the buildout,
> and then point LD_LIBRARY_PATH at that.

Maybe. That's what I would do. :)

There are really 2 issues:

- Whenther to rely on your system libraries,

- How to tell distutils where the libraries are, both at build and at  
run time.

The former depends on the library.  I've had problems with this  
particular library on Red Hat-based systems, si we typically build  
our own as part of a buildout.

For the later, the zc.recipe.egg:custom recipe is what we use.  It is  
simplest to use the rpath option at build time.

The rpath option doesn't work for us in production deployments  
because our deployment-install locations are different than our build  
locations, so we use LD_LIBRARY_PATH at run time.  Because we use  
zdaemon and zdaemon lets us specify run-time environment variables,  
this isn't a problem. (Note that a Python script can't set it's own  
LD_LIBRARY_PATH, because it is too late -- the library loader has  
already looked for this before a Python script can set it.  This  
works with zdaemon because zdaemon creates a separate subprocess for  
the application and the environment variables are set before the  
subprocess runs.)

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From radix at twistedmatrix.com  Thu Mar  8 17:39:42 2007
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Thu, 8 Mar 2007 11:39:42 -0500
Subject: [Distutils] conditionally compiling extensions
In-Reply-To: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>
References: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>
Message-ID: <60ed19d40703080839n234ad27dw1491fd0d52534542@mail.gmail.com>

On 3/5/07, Christopher Armstrong <radix at twistedmatrix.com> wrote:
> Before showing what these hacks are (if you really care, you can check
> the various setup.py files in the Twisted repository), I'd like to ask
> if there is any other best-practice knowledge in the community for
> conditionally building extensions based on knowledge only available by
> using a compiler.


Ok, here's what I think I'm going to do.

Subclass Extension to have an extra method, shouldBuild, which is
called by build_extension by my subclass of build_ext. It will filter
out any extensions for which shouldBuild returns False. This way
extensions will still be declared ahead of time and so no early-out
code will be problematic.

Probably my subclass of Extension will take a callable in its
initializer to use as the default implementation of shouldBuild.

I'll try this out soon.

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From pje at telecommunity.com  Thu Mar  8 20:06:50 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 08 Mar 2007 14:06:50 -0500
Subject: [Distutils] conditionally compiling extensions
In-Reply-To: <60ed19d40703080839n234ad27dw1491fd0d52534542@mail.gmail.co
 m>
References: <60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>
	<60ed19d40703051213v55988cdbgae05e8d2f93d8cf1@mail.gmail.com>
Message-ID: <5.1.1.6.0.20070308140620.0282f8b8@sparrow.telecommunity.com>

At 11:39 AM 3/8/2007 -0500, Christopher Armstrong wrote:
>On 3/5/07, Christopher Armstrong <radix at twistedmatrix.com> wrote:
> > Before showing what these hacks are (if you really care, you can check
> > the various setup.py files in the Twisted repository), I'd like to ask
> > if there is any other best-practice knowledge in the community for
> > conditionally building extensions based on knowledge only available by
> > using a compiler.
>
>
>Ok, here's what I think I'm going to do.
>
>Subclass Extension to have an extra method, shouldBuild, which is
>called by build_extension by my subclass of build_ext. It will filter
>out any extensions for which shouldBuild returns False. This way
>extensions will still be declared ahead of time and so no early-out
>code will be problematic.
>
>Probably my subclass of Extension will take a callable in its
>initializer to use as the default implementation of shouldBuild.
>
>I'll try this out soon.

Sounds nice.  When you're done, can I steal it for setuptools?  :)


From jpellerin at gmail.com  Thu Mar  8 20:41:25 2007
From: jpellerin at gmail.com (jason pellerin)
Date: Thu, 8 Mar 2007 14:41:25 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <3bb02d620703071038s19a93f65wc77195ab4aa41f49@mail.gmail.com>
References: <5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>
	<45EEF8A4.7030402@colorstudy.com>
	<3bb02d620703071038s19a93f65wc77195ab4aa41f49@mail.gmail.com>
Message-ID: <3bb02d620703081141l2e20aad5veb27940cee9e26b2@mail.gmail.com>

On 3/7/07, jason pellerin <jpellerin at gmail.com> wrote:
> On 3/7/07, Ian Bicking <ianb at colorstudy.com> wrote:
> > Phillip J. Eby wrote:
> > > At 02:29 PM 3/6/2007 -0500, jason pellerin wrote:
> > >> I've been meaning to ask about this same issue, specifically with
> > >> respect to namespace packages, bdist_rpm and imp. I've had to patch
> > >> the spec files for all namespace packages that we build as RPMs, since
> > >> although the .pth file works when using import or __import__, imp
> > >> doesn't seem to recognize it, and that breaks Paste's app loading.
> > >
> > > Then Paste's app loading is broken, and wouldn't work with an egg either,
> > > if it's using "imp" and ignoring the package's __path__.
> >
> > I'm not sure which part of Paste is in question, but the only importing
> > that I can think of which it does is with pkg_resources or with __import__.
>
> I didn't save the output from trying to use the broken package load,
> unfortunately. I'll try to recreate it if I can. I may easily be
> putting the blame in the wrong place, though, since the failure was in
> paste.modpython --

My apologies -- it's mod_python that fails, not Paste. I removed my
working Paste package (with the forced __init__.py) and reinstalled
the one without __init__.py. Here's the result:

Mod_python error: "PythonHandler paste.modpython"

Traceback (most recent call last):

  File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
287, in HandlerDispatch
    log=debug)

  File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
461, in import_module
    f, p, d = imp.find_module(parts[i], path)

ImportError: No module named paste

So Paste is actually the victim, not the culprit. I don't know if
there's any way to work around this aside from building the package
with an __init__.py -- any ideas?

Thanks,

JP

From optilude at gmx.net  Thu Mar  8 21:46:32 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Thu, 08 Mar 2007 20:46:32 +0000
Subject: [Distutils] Setting environment variables before building eggs
 w/ extensions using zc.buildout
In-Reply-To: <esn6eq$f5u$1@sea.gmane.org>
References: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>	<esn4ev$6sc$1@sea.gmane.org>
	<esn6eq$f5u$1@sea.gmane.org>
Message-ID: <espsn9$ohn$1@sea.gmane.org>

whit wrote:

> fyi, the deliverance.buildout will soon go away

Well, I don't think it has much value for re-usability of Deliverance 
anyway.

Do you have an alternate distribution mechanism? Perhaps one we could 
leverage inside a buildout? :)

Martin


From pje at telecommunity.com  Thu Mar  8 23:58:42 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 08 Mar 2007 17:58:42 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <3bb02d620703081141l2e20aad5veb27940cee9e26b2@mail.gmail.co
 m>
References: <3bb02d620703071038s19a93f65wc77195ab4aa41f49@mail.gmail.com>
	<5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>
	<45EEF8A4.7030402@colorstudy.com>
	<3bb02d620703071038s19a93f65wc77195ab4aa41f49@mail.gmail.com>
Message-ID: <5.1.1.6.0.20070308175740.01bf4458@sparrow.telecommunity.com>

At 02:41 PM 3/8/2007 -0500, jason pellerin wrote:
>My apologies -- it's mod_python that fails, not Paste. I removed my
>working Paste package (with the forced __init__.py) and reinstalled
>the one without __init__.py. Here's the result:
>
>Mod_python error: "PythonHandler paste.modpython"
>
>Traceback (most recent call last):
>
>  File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
>287, in HandlerDispatch
>    log=debug)
>
>  File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
>461, in import_module
>    f, p, d = imp.find_module(parts[i], path)
>
>ImportError: No module named paste
>
>So Paste is actually the victim, not the culprit. I don't know if
>there's any way to work around this aside from building the package
>with an __init__.py -- any ideas?

Fix mod_python/apache.py to use __import__ instead of imp, perhaps?


From jpellerin at gmail.com  Fri Mar  9 00:21:07 2007
From: jpellerin at gmail.com (jason pellerin)
Date: Thu, 8 Mar 2007 18:21:07 -0500
Subject: [Distutils] __init__.py getting left out of mpkg
In-Reply-To: <5.1.1.6.0.20070308175740.01bf4458@sparrow.telecommunity.com>
References: <5.1.1.6.0.20070307104137.02d39300@sparrow.telecommunity.com>
	<45EEF8A4.7030402@colorstudy.com>
	<3bb02d620703071038s19a93f65wc77195ab4aa41f49@mail.gmail.com>
	<5.1.1.6.0.20070308175740.01bf4458@sparrow.telecommunity.com>
Message-ID: <3bb02d620703081521s3e3892b5mb622459981a7f3f6@mail.gmail.com>

On 3/8/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 02:41 PM 3/8/2007 -0500, jason pellerin wrote:
> >My apologies -- it's mod_python that fails, not Paste. I removed my
> >working Paste package (with the forced __init__.py) and reinstalled
> >the one without __init__.py. Here's the result:
> >
> >Mod_python error: "PythonHandler paste.modpython"
> >
> >Traceback (most recent call last):
> >
> >  File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
> >287, in HandlerDispatch
> >    log=debug)
> >
> >  File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line
> >461, in import_module
> >    f, p, d = imp.find_module(parts[i], path)
> >
> >ImportError: No module named paste
> >
> >So Paste is actually the victim, not the culprit. I don't know if
> >there's any way to work around this aside from building the package
> >with an __init__.py -- any ideas?
>
> Fix mod_python/apache.py to use __import__ instead of imp, perhaps?

I don't think that's very likely. ;) They are using imp (and seem to
be using it properly, based on what you wrote earlier about using the
package path as the path for dotted lookups) for the same reason I use
it in nose -- to load foo/bar.py and baz/bar.py as different modules
(where foo and baz are non-package directories). imp gives you enough
control to do that, but __import__ doesn't, as far as I know.

JP

From srobertson at codeit.com  Fri Mar  9 23:17:32 2007
From: srobertson at codeit.com (Scott Robertson)
Date: Fri, 9 Mar 2007 14:17:32 -0800
Subject: [Distutils] Buildout question
Message-ID: <905d10ce0703091417j629e0ae3ia2d5afa6dcc5cb9b@mail.gmail.com>

Is there a way to create a dummy part/recipe that depends on another
part without having to refer to one of the options of other part?
Similar to make's concept of a phony target I suppose.

I'm looking to make a buildout script that looks something like this:

[buildout]
parts = all

[all]
recipe=depends
parts = linux win32 mac

[linux]
recipe=depends
parts=common-part linux-specific-part

[win32]
recipe=depends
parts=common-part win32-specific-part-1 win32-specific-part-2

[mac]
recipe=depends
parts=common-part mac-specific-part

... real part specifications would follow ...



So as you can see, I'm trying to make a buildout script that will
build three different variations of the same project for the three
major platforms  Windows, Mac and Linux from the same build machine.

Depending on the platform I'm building for I may build  parts in my
buildout file. Ideally from the command line I'd like to do something
like this.

$ buildout install linux

or

$ buildout install mac
or

$ buildout install win32

or
$ buildout install all


Is this possible to do in recipe or is there some other mechanism I
could use to do this dependency? The refering to a variable of another
part sort of works for this, but I think it would be easier to read
(and less typing) if I could epxlicitly declare that a given part
depends on another.

From nathan at yergler.net  Sun Mar 11 16:08:43 2007
From: nathan at yergler.net (Nathan R. Yergler)
Date: Sun, 11 Mar 2007 10:08:43 -0500
Subject: [Distutils] buildout, zc.recipe.testrunner and tests_require
Message-ID: <bf7b44d50703110808g148c2e31m58668e9bdc630fb2@mail.gmail.com>

I'm using zc,buildout to manage the development of a very simple web
application.  I'd like to have a simple test runner that runs my
doctests, but have run into a slight problem.  I'm using
wsgi_intercept to aid in testing, but unfortunately wsgi_intercept
isn't a proper Python package ("yet", according to Titus).  So I have
a package-i-fied version in my svn repository and in my buildout.cfg I
specify:

develop = . ./wsgi_intercept_

I initially was using the "interpreter" option to get a Python
interpreter with my eggs on the path, and running "./bin/python
setup.py test".  That, however, required me to list wsgi_intercept as
a package requirement, when I really want to list it as a testing
requirement.  Moving it to a testing requirement caused it's develop
egg not to be on the PYTHONPATH, so setuptools tries to find it.
Which it couldn't.

I tried using zc.recipe.testbrowser, thinking that maybe it'd look at
the tests_require for the target eggs, but no such luck.  The
testrunner also seems pretty promiscuous in looking for things to test
(it tries to import my eggs directory and test them which predictably
doesn't work), but that's another story.

So... any suggestions on using zc.buildout along with testing
dependencies (generally or with zc.recipe.testrunner)?

Thanks,

Nathan

From jim at zope.com  Sun Mar 11 21:16:33 2007
From: jim at zope.com (Jim Fulton)
Date: Sun, 11 Mar 2007 16:16:33 -0400
Subject: [Distutils] Buildout question
In-Reply-To: <905d10ce0703091417j629e0ae3ia2d5afa6dcc5cb9b@mail.gmail.com>
References: <905d10ce0703091417j629e0ae3ia2d5afa6dcc5cb9b@mail.gmail.com>
Message-ID: <9D1256CC-B18D-4F7F-A3BE-FEAAE45582F1@zope.com>


On Mar 9, 2007, at 5:17 PM, Scott Robertson wrote:

> Is there a way to create a dummy part/recipe that depends on another
> part without having to refer to one of the options of other part?

No, but you could hide the reference in the recipe.

> Similar to make's concept of a phony target I suppose.
>
> I'm looking to make a buildout script that looks something like this:
>
> [buildout]
> parts = all
>
> [all]
> recipe=depends
> parts = linux win32 mac
>
> [linux]
> recipe=depends
> parts=common-part linux-specific-part
>
> [win32]
> recipe=depends
> parts=common-part win32-specific-part-1 win32-specific-part-2
>
> [mac]
> recipe=depends
> parts=common-part mac-specific-part
>
> ... real part specifications would follow ...
>
>
>
> So as you can see, I'm trying to make a buildout script that will
> build three different variations of the same project for the three
> major platforms  Windows, Mac and Linux from the same build machine.
>
> Depending on the platform I'm building for I may build  parts in my
> buildout file. Ideally from the command line I'd like to do something
> like this.
>
> $ buildout install linux
>
> or
>
> $ buildout install mac
> or
>
> $ buildout install win32
>
> or
> $ buildout install all
>
>
> Is this possible to do in recipe or is there some other mechanism I
> could use to do this dependency? The refering to a variable of another
> part sort of works for this, but I think it would be easier to read
> (and less typing) if I could epxlicitly declare that a given part
> depends on another.

You don't have to refer to a variable. Refering to a section is enough.
For example:

   class Depends:

       def __init__(self, buildout, name, options):
           for part in options['parts']:
               buildout[part]

       def install(self):
           pass

       update = install

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From venkatbo at yahoo.com  Sun Mar 11 23:23:07 2007
From: venkatbo at yahoo.com (Venkat Bommakanti)
Date: Sun, 11 Mar 2007 15:23:07 -0700 (PDT)
Subject: [Distutils] Ability to override location of build/ dir while
	creating a .egg
Message-ID: <20070311222307.8702.qmail@web30515.mail.mud.yahoo.com>

Hi folks,

'am working on a TurboGears1.0-based app that I need
deploy as a .egg. 'am using setuptools-0.6c3...

TG apps have a basic structure like:
  app-top/
    setup.py
    app/
      config/
      controllers/
      ...
    app-egg-info/
      ...

>From 
  app-top/
when I run:
  python setup.py bdist_egg -b tmp<tgt>dir/distdir/
                            -d tmp<tgt>dir/
I can see:
  app-top/
    setup.py
    tmp<tgt>dir/
      app.egg
    ...
    build/
    ...
after successful .egg creation. While there is no
issue with that process, I noticed that it places
the temporary
    build/
directory at the level of tmp<tgt>dir itself. Is 
there a way to specify the location of this 
temporary build/ directory ?

Looked in the online docs, but no way to specify
that. The reason I ask, is that our nightly build 
system builds both a <tgt>=i686 and <tgt>=ppc 
version of my app and there could be a potential 
conflict for both these versions if it just so 
happens that one target-build overwrites the 
other... at the same time.

If I could specify the build/ path, I could have
something like:
  app-top/
    setup.py
    tmp<tgt>dir/
      app.egg
      build/
        ...
      ...
    ...
and avoid issues of simulataneous diff-<tgt> builds.

Thanks much,
/venkat



 
____________________________________________________________________________________
We won't tell. Get more on shows you hate to love 
(and love to hate): Yahoo! TV's Guilty Pleasures list.
http://tv.yahoo.com/collections/265 

From pje at telecommunity.com  Mon Mar 12 02:06:28 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sun, 11 Mar 2007 20:06:28 -0500
Subject: [Distutils] Ability to override location of build/ dir while
 creating a .egg
In-Reply-To: <20070311222307.8702.qmail@web30515.mail.mud.yahoo.com>
Message-ID: <5.1.1.6.0.20070311200450.047568c8@sparrow.telecommunity.com>

At 03:23 PM 3/11/2007 -0700, Venkat Bommakanti wrote:
>Is
>there a way to specify the location of this
>temporary build/ directory ?

python setup.py build -b path/to/build bdist_egg otheroptions...


From jim at zope.com  Mon Mar 12 14:47:27 2007
From: jim at zope.com (Jim Fulton)
Date: Mon, 12 Mar 2007 09:47:27 -0400
Subject: [Distutils] buildout, zc.recipe.testrunner and tests_require
In-Reply-To: <bf7b44d50703110808g148c2e31m58668e9bdc630fb2@mail.gmail.com>
References: <bf7b44d50703110808g148c2e31m58668e9bdc630fb2@mail.gmail.com>
Message-ID: <13353241-2CB2-4DEB-8277-E280AF3F77D6@zope.com>


On Mar 11, 2007, at 11:08 AM, Nathan R. Yergler wrote:
...
>
> I tried using zc.recipe.testbrowser, thinking that maybe it'd look at
> the tests_require for the target eggs, but no such luck.

The tests_require option is only visible to the setup script.  It  
doesn't cause any data to be included in egg info.  It is an  
attractive nuisance IMO.

See: http://mail.python.org/pipermail/distutils-sig/2007-January/ 
007082.html

In situations like this, I use a test extra and extra_requires to  
specify testing dependencies.  Then, when using the testrunner  
recipe, I use something like:

eggs = foo [test]

to indicate that the test runner should load and test the foo egg  
with its test dependencies.

> The
> testrunner also seems pretty promiscuous in looking for things to test
> (it tries to import my eggs directory and test them which predictably
> doesn't work), but that's another story.

That's odd. I haven't seen this problem.  What settings were you  
providing?

The testrunner recipe uses the Zope test runner, http:// 
www.python.org/pypi/zope.testing, which I think has a lot of nice  
features to support development, but It doesn't yet support running  
tests from zipped eggs.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From srobertson at codeit.com  Mon Mar 12 17:40:36 2007
From: srobertson at codeit.com (Scott Robertson)
Date: Mon, 12 Mar 2007 09:40:36 -0700
Subject: [Distutils] Buildout question
In-Reply-To: <9D1256CC-B18D-4F7F-A3BE-FEAAE45582F1@zope.com>
References: <905d10ce0703091417j629e0ae3ia2d5afa6dcc5cb9b@mail.gmail.com>
	<9D1256CC-B18D-4F7F-A3BE-FEAAE45582F1@zope.com>
Message-ID: <905d10ce0703120940y3af0c03bs5ac9f308faae3596@mail.gmail.com>

Thanks Jim,
I stumbled upon that myself this  weekend wasn't sure if it was appropriate.

 One thing I noticed, is that if you use the "install" keyword on the
command line i.e.
$ buildout install some-section

Dependent sections don't get built (is that intentional?)

This patch seems to do the trick, though I'm not sure if it causes any
unindented side effects.

--- buildout.py.orig    2007-03-12 09:38:09.000000000 -0700
+++ buildout.py 2007-03-12 09:38:19.000000000 -0700
@@ -213,8 +213,8 @@

         # load and initialize recipes
         [self[part]['recipe'] for part in install_parts]
-        if not install_args:
-           install_parts = self._parts
+
+       install_parts = self._parts

         if self._log_level <= logging.DEBUG:
             sections = list(self)



On 3/11/07, Jim Fulton <jim at zope.com> wrote:
>
> On Mar 9, 2007, at 5:17 PM, Scott Robertson wrote:
>
> > Is there a way to create a dummy part/recipe that depends on another
> > part without having to refer to one of the options of other part?
>
> No, but you could hide the reference in the recipe.
>
> > Similar to make's concept of a phony target I suppose.
> >
> > I'm looking to make a buildout script that looks something like this:
> >
> > [buildout]
> > parts = all
> >
> > [all]
> > recipe=depends
> > parts = linux win32 mac
> >
> > [linux]
> > recipe=depends
> > parts=common-part linux-specific-part
> >
> > [win32]
> > recipe=depends
> > parts=common-part win32-specific-part-1 win32-specific-part-2
> >
> > [mac]
> > recipe=depends
> > parts=common-part mac-specific-part
> >
> > ... real part specifications would follow ...
> >
> >
> >
> > So as you can see, I'm trying to make a buildout script that will
> > build three different variations of the same project for the three
> > major platforms  Windows, Mac and Linux from the same build machine.
> >
> > Depending on the platform I'm building for I may build  parts in my
> > buildout file. Ideally from the command line I'd like to do something
> > like this.
> >
> > $ buildout install linux
> >
> > or
> >
> > $ buildout install mac
> > or
> >
> > $ buildout install win32
> >
> > or
> > $ buildout install all
> >
> >
> > Is this possible to do in recipe or is there some other mechanism I
> > could use to do this dependency? The refering to a variable of another
> > part sort of works for this, but I think it would be easier to read
> > (and less typing) if I could epxlicitly declare that a given part
> > depends on another.
>
> You don't have to refer to a variable. Refering to a section is enough.
> For example:
>
>    class Depends:
>
>        def __init__(self, buildout, name, options):
>            for part in options['parts']:
>                buildout[part]
>
>        def install(self):
>            pass
>
>        update = install
>
> Jim
>
> --
> Jim Fulton                      mailto:jim at zope.com             Python Powered!
> CTO                             (540) 361-1714                  http://www.python.org
> Zope Corporation        http://www.zope.com             http://www.zope.org
>
>
>
>

From jim at zope.com  Mon Mar 12 18:54:18 2007
From: jim at zope.com (Jim Fulton)
Date: Mon, 12 Mar 2007 13:54:18 -0400
Subject: [Distutils] Buildout question
In-Reply-To: <905d10ce0703120940y3af0c03bs5ac9f308faae3596@mail.gmail.com>
References: <905d10ce0703091417j629e0ae3ia2d5afa6dcc5cb9b@mail.gmail.com>
	<9D1256CC-B18D-4F7F-A3BE-FEAAE45582F1@zope.com>
	<905d10ce0703120940y3af0c03bs5ac9f308faae3596@mail.gmail.com>
Message-ID: <477BC5A3-F55B-4BE0-87E6-A7777291F534@zope.com>


On Mar 12, 2007, at 12:40 PM, Scott Robertson wrote:

> Thanks Jim,
> I stumbled upon that myself this  weekend wasn't sure if it was  
> appropriate.
>
> One thing I noticed, is that if you use the "install" keyword on the
> command line i.e.
> $ buildout install some-section
>
> Dependent sections don't get built (is that intentional?)

Yes, and documented (http://www.python.org/pypi/zc.buildout#command- 
line-usage).

If you want to install a part and its dependents, use:  
buildout:installed=partname,
or, for multiple parts: "buildout:installed=p1 p2 p3".

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From srobertson at codeit.com  Mon Mar 12 20:09:30 2007
From: srobertson at codeit.com (Scott Robertson)
Date: Mon, 12 Mar 2007 12:09:30 -0700
Subject: [Distutils] Buildout install command
Message-ID: <905d10ce0703121209w35cb452fi7aa7f20ebedd72e4@mail.gmail.com>

Ok slightly confused, (hope I'm not wasting your time with this  Jim)

I just checked that url you provided. There is no mention of the
"install" command, which I found by running buildout --help. Here's
the text:


Commands:

  install [parts]

    Install parts.  If no command arguments are given, then the parts
    definition from the configuration file is used.  Otherwise, the
    arguments specify the parts to be installed.


Interestingly (i.e. what confuses me)

If I run
$ buildout buildout:parts = part1

Part 1 is built and all of part1's dependencies.

However if I run

$ buildout install part1
Part 1 starts to build but in my case bombs because part 1's
dependencies aren't  built.

So as a new buildout user I'm confused about:
1. The purpose of the install command
2. Why it doesn't build dependencies.









On 3/12/07, Jim Fulton <jim at zope.com> wrote:
>
> On Mar 12, 2007, at 12:40 PM, Scott Robertson wrote:
>
> > Thanks Jim,
> > I stumbled upon that myself this  weekend wasn't sure if it was
> > appropriate.
> >
> > One thing I noticed, is that if you use the "install" keyword on the
> > command line i.e.
> > $ buildout install some-section
> >
> > Dependent sections don't get built (is that intentional?)
>
> Yes, and documented (http://www.python.org/pypi/zc.buildout#command-
> line-usage).
>
> If you want to install a part and its dependents, use:
> buildout:installed=partname,
> or, for multiple parts: "buildout:installed=p1 p2 p3".
>
> Jim
>
> --
> Jim Fulton                      mailto:jim at zope.com             Python Powered!
> CTO                             (540) 361-1714                  http://www.python.org
> Zope Corporation        http://www.zope.com             http://www.zope.org
>
>
>
>

From nathan at yergler.net  Wed Mar 14 22:51:46 2007
From: nathan at yergler.net (Nathan R. Yergler)
Date: Wed, 14 Mar 2007 17:51:46 -0400
Subject: [Distutils] Setting environment variables before building eggs
	w/ extensions using zc.buildout
In-Reply-To: <FE971CCB-F382-4A98-91E5-A779D4A35EEB@zope.com>
References: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>
	<FE971CCB-F382-4A98-91E5-A779D4A35EEB@zope.com>
Message-ID: <bf7b44d50703141451w5d17bac0h63e173b32f5428c6@mail.gmail.com>

Late reply...

On 3/8/07, Jim Fulton <jim at zope.com> wrote:
>
> On Mar 7, 2007, at 2:07 PM, Nathan R. Yergler wrote:
>
> I meant to reply earlier. Sorry.
>
> > I had a problem come up today that I've run into before, and I'd like
> > to figure out a way to prevent it from sucking my time down in the
> > future.  Several apps we run at Creative Commons use lxml
> > (http://codespeak.net/lxml) for XML, XSLT and XPath processing.  lxml
> > builds on libxml2, and provides an element tree interface, plus
> > additional capabilities.
> >
> > We use zc.buildout to assemble our server-side applications, and that
> > has generally been a huge sanity saver.  However, lxml can rob some of
> > that sanity periodically.  If a system like one of our CentOS machines
> > has an older version of libxml2 installed in /usr/lib, the extension
> > building process will link against it without complaint.  That is,
> > until you try to do something added in the later, recommended version
> > of libxml2 like XSLT.  Once you scratch your head and remember what
> > the problem is, you remember to export LD_LIBRARY_PATH=/usr/local/lib,
> > and then re-run the buildout.
> >
> > Does zc.buildout currently have any way to set an environment variable
> > during it's run?
>
> No.
>
> >   I didn't see anything at
> > http://cheeseshop.python.org/pypi/zc.recipe.egg/, but I didn't know if
> > there was something at the zc.buildout level (as opposed to the recipe
> > level) that would do the trick.  The quick/dirty thing I'd do is
> > export the appropriate value of LD_LIBRARY_PATH.
>
> I'm a little surprised that this would help at build time.  I'm
> surprised that distutils looks at LD_LIBRARY_PATH when deciding where
> to link from.
>
> Have you looked at:
>
>    http://www.python.org/pypi/zc.recipe.egg#creating-eggs-with-
> extensions-neededing-custom-build-settings
>
> >   The perhaps more
> > intelligent thing I'd do is use the cmmi recipe to build a version of
> > libxml2 that I *know* has the features I need as part of the buildout,
> > and then point LD_LIBRARY_PATH at that.
>
> Maybe. That's what I would do. :)
>
> There are really 2 issues:
>
> - Whenther to rely on your system libraries,
>
> - How to tell distutils where the libraries are, both at build and at
> run time.
>
> The former depends on the library.  I've had problems with this
> particular library on Red Hat-based systems, si we typically build
> our own as part of a buildout.
>
> For the later, the zc.recipe.egg:custom recipe is what we use.  It is
> simplest to use the rpath option at build time.
>
> The rpath option doesn't work for us in production deployments
> because our deployment-install locations are different than our build
> locations, so we use LD_LIBRARY_PATH at run time.  Because we use
> zdaemon and zdaemon lets us specify run-time environment variables,
> this isn't a problem. (Note that a Python script can't set it's own
> LD_LIBRARY_PATH, because it is too late -- the library loader has
> already looked for this before a Python script can set it.  This
> works with zdaemon because zdaemon creates a separate subprocess for
> the application and the environment variables are set before the
> subprocess runs.)

Ah; luckily we're using zdaemon as well, so I'll go down that path.  I
suppose I could combine what Deliverance does with zdaemon and have a
buildout which:

* builds libxml2 and libxslt
* builds lxml against them
* updates my zdaemon.conf to include the resulting lib path


>
> Jim
>
> --
> Jim Fulton                      mailto:jim at zope.com             Python Powered!
> CTO                             (540) 361-1714                  http://www.python.org
> Zope Corporation        http://www.zope.com             http://www.zope.org
>
>
>
>

From jim at zope.com  Thu Mar 15 14:08:37 2007
From: jim at zope.com (Jim Fulton)
Date: Thu, 15 Mar 2007 09:08:37 -0400
Subject: [Distutils] Improved error handling in zc.buildout
Message-ID: <B5A15B2F-D24C-4F30-815A-E18CF2A631DF@zope.com>


A major annoyance in zc.buildout has been its poor error handling.   
Generally, when an error occurred, a a traceback would be printed  
giving little or no clue about what went wrong or what was going on.   
Previously, I was focussed on important features.  The latest release  
focusses on better error reporting and debugging support.  I'd like  
to continue to provide some emphasis on this, so feedback would be  
especially appreciated now that I think I've improved error handling  
quite a bit.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From nathan at yergler.net  Thu Mar 15 15:48:02 2007
From: nathan at yergler.net (Nathan R. Yergler)
Date: Thu, 15 Mar 2007 10:48:02 -0400
Subject: [Distutils] Getting an egg's version: better way?
Message-ID: <bf7b44d50703150748k3697decaxe2087b51fd159865@mail.gmail.com>

I have a situation where I want to get the version of an egg that's
installed.  I figured out the following, but wanted to know if I'm
missing something obvious that's easier::

  >>> import pkg_resources
  >>> p = pkg_resources.get_provider("cctagutils")
  >>> for line in p.get_metadata_lines("PKG-INFO"):
  ...    if line.find('Version: ') == 0:
  ...      version = line.split()[-1]

Thanks,

Nathan

From cakebread at gmail.com  Thu Mar 15 16:20:33 2007
From: cakebread at gmail.com (Rob Cakebread)
Date: Thu, 15 Mar 2007 08:20:33 -0700
Subject: [Distutils] Getting an egg's version: better way?
In-Reply-To: <bf7b44d50703150748k3697decaxe2087b51fd159865@mail.gmail.com>
References: <bf7b44d50703150748k3697decaxe2087b51fd159865@mail.gmail.com>
Message-ID: <9b06ffb10703150820h70fbe9d8i1721ab975b009b2c@mail.gmail.com>

On 3/15/07, Nathan R. Yergler <nathan at yergler.net> wrote:
> I have a situation where I want to get the version of an egg that's
> installed.  I figured out the following, but wanted to know if I'm
> missing something obvious that's easier::
>

Assuming you just have one version installed:

>>> from pkg_resources import Environment
>>> e = Environment()
>>> print e["cctagutils"][0].version

From pje at telecommunity.com  Thu Mar 15 17:34:14 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Thu, 15 Mar 2007 11:34:14 -0500
Subject: [Distutils] Getting an egg's version: better way?
In-Reply-To: <bf7b44d50703150748k3697decaxe2087b51fd159865@mail.gmail.co
 m>
Message-ID: <5.1.1.6.0.20070315112854.04d1d1a8@sparrow.telecommunity.com>

At 10:48 AM 3/15/2007 -0400, Nathan R. Yergler wrote:
>I have a situation where I want to get the version of an egg that's
>installed.  I figured out the following, but wanted to know if I'm
>missing something obvious that's easier::
>
>   >>> import pkg_resources
>   >>> p = pkg_resources.get_provider("cctagutils")
>   >>> for line in p.get_metadata_lines("PKG-INFO"):
>   ...    if line.find('Version: ') == 0:
>   ...      version = line.split()[-1]

The problem with using get_provider() and a module name, is that it cannot 
conclusively associate a package with a specific project.  Thus, your code 
above has a bug: it will not find a version of a package that's installed 
by a system packaging tool such as RPM, or a bdist .exe or .msi on Windows.

You need to use a project name instead of a package/module name, e.g.:

    version = pkg_resources.require('ProjectNameHere')[0].version

This will work even with projects installed using system packaging tools, 
as it explicitly indicates what project's PKG-INFO will be read, if in fact 
it needs to be read at all.  (In most cases, pkg_resources will get the 
project's version number from a filename, without needing to open any files.)


From optilude at gmx.net  Fri Mar 16 01:10:57 2007
From: optilude at gmx.net (Martin Aspeli)
Date: Fri, 16 Mar 2007 11:10:57 +1100
Subject: [Distutils] Improved error handling in zc.buildout
In-Reply-To: <B5A15B2F-D24C-4F30-815A-E18CF2A631DF@zope.com>
References: <B5A15B2F-D24C-4F30-815A-E18CF2A631DF@zope.com>
Message-ID: <etcnan$83f$1@sea.gmane.org>

Jim Fulton wrote:
> A major annoyance in zc.buildout has been its poor error handling.   
> Generally, when an error occurred, a a traceback would be printed  
> giving little or no clue about what went wrong or what was going on.   
> Previously, I was focussed on important features.  The latest release  
> focusses on better error reporting and debugging support.  I'd like  
> to continue to provide some emphasis on this, so feedback would be  
> especially appreciated now that I think I've improved error handling  
> quite a bit.

That's great news, Jim!

I had a particular problem before, where the "Plone" egg (which includes 
all the Products.* packages that make up plone, and all the other proper 
eggs as dependencies) would try to compile .py files in filesystem skin 
folders (i.e. Zope Script (Python)'s) and get compile errors (return 
outside function, for example). With workingenv and easy_install, it'd 
report the errors but not stop; buildout seems to stop. Better error 
handling would hopefully help diagnose this. We couldn't find a way to 
stop setuptools from trying to compile those files, or a way to get 
buildout to ignore the errors.

Martin


From howard at eegsoftware.com  Thu Mar 15 18:45:22 2007
From: howard at eegsoftware.com (Howard Lightstone)
Date: Thu, 15 Mar 2007 17:45:22 +0000 (UTC)
Subject: [Distutils] Visual Studio 2005 changes
Message-ID: <Xns98F46D619C8DAhowardeegsoftwarecom@80.91.229.5>

I have some (minor ?) changes to msvccompiler.py to support VC8 (and 
embedding manifests).

Where should I submit these changes for someone (knowledgable) to review?



From regebro at gmail.com  Fri Mar 16 10:23:35 2007
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 16 Mar 2007 10:23:35 +0100
Subject: [Distutils] Buildout configurations in a subdirectory stopped
	working.
Message-ID: <319e029f0703160223r717e04b4u5d51e3f31a408819@mail.gmail.com>

On 2/19/07, Christian Theune <ct at gocept.com> wrote:
> [buildout]
> extends = profiles/dev.cfg
>
> and run bin/buildout.
>
> This works very well for us.

It did for me too, but it has now stopped working.
Line 89 in buildout.py says:

            data['buildout']['directory'] = os.path.dirname(config_file)

This must be a recent change (< 2 weeks) because this makes buildout
fail if you have the cfg files anywhere else than root. I'd like to
know the rationale for this change, as I'd like to be able to have the
profiles in a subdirectory.

--
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

From p.f.moore at gmail.com  Fri Mar 16 10:55:04 2007
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 16 Mar 2007 09:55:04 +0000
Subject: [Distutils] Visual Studio 2005 changes
In-Reply-To: <Xns98F46D619C8DAhowardeegsoftwarecom@80.91.229.5>
References: <Xns98F46D619C8DAhowardeegsoftwarecom@80.91.229.5>
Message-ID: <79990c6b0703160255q14c32edcl7362af7fccc171d8@mail.gmail.com>

On 15/03/07, Howard Lightstone <howard at eegsoftware.com> wrote:
> I have some (minor ?) changes to msvccompiler.py to support VC8 (and
> embedding manifests).
>
> Where should I submit these changes for someone (knowledgable) to review?

You should submit them as a patch to the Python patch tracker on
Sourceforge. Having done so, you'll need to be patient, as there
aren't many people with the relevant expertise (or resources - for
example, I might be able to understand the patch, but I don't have
VC8, and a Python built with VC8, to test it!) to review it before
applying it.

Thanks!
Paul.

From Jim.Vickroy at noaa.gov  Thu Mar 15 21:13:32 2007
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Thu, 15 Mar 2007 14:13:32 -0600
Subject: [Distutils] restructured text formatting not present in generated
 MS Windows binary installer
Message-ID: <45F9A8EC.3030106@noaa.gov>

Hello,

I have been unsuccessful in using restructured text formatting tags in 
the long_description parameter for the distutils.core.setup function.

When the attached "setup" script (see GOES-13-SXI-processor-installer.py 
attachment) is run from the command line as:

 > GOES-13-SXI-processor-installer.py clean bdist_wininst --title="FITS 
Images Processor" --bitmap=GOES-13-SXI-Processor.bmp

the first panel of the generated win32 installer has no boldface text 
(see installer.png attachment)

When I feed the long_description text to docutils/tools/rst2html.py, the 
generated html is as expected.

My setup is:

Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
 >>> import docutils
 >>> docutils.__version__
'0.5'
 >>>

on a MS Windows XP machine running service pack 2.

I am a new user of both distutils and docutils so any thoughts on what I 
am doing incorrectly would be appreciated.

Thanks,

-- jv


-------------- next part --------------
A non-text attachment was scrubbed...
Name: installer.PNG
Type: image/png
Size: 73768 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070315/1ff29be4/attachment-0001.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GOES-13-SXI-processor-installer.py
Type: application/x-python
Size: 4986 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070315/1ff29be4/attachment-0001.bin 

From nathan at yergler.net  Fri Mar 16 14:58:57 2007
From: nathan at yergler.net (Nathan R. Yergler)
Date: Fri, 16 Mar 2007 09:58:57 -0400
Subject: [Distutils] Getting an egg's version: better way?
In-Reply-To: <5.1.1.6.0.20070315112854.04d1d1a8@sparrow.telecommunity.com>
References: <5.1.1.6.0.20070315112854.04d1d1a8@sparrow.telecommunity.com>
Message-ID: <bf7b44d50703160658u3c5249e0p1ad1ca098253219d@mail.gmail.com>

Thanks guys, that helps a lot.

On 3/15/07, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 10:48 AM 3/15/2007 -0400, Nathan R. Yergler wrote:
> >I have a situation where I want to get the version of an egg that's
> >installed.  I figured out the following, but wanted to know if I'm
> >missing something obvious that's easier::
> >
> >   >>> import pkg_resources
> >   >>> p = pkg_resources.get_provider("cctagutils")
> >   >>> for line in p.get_metadata_lines("PKG-INFO"):
> >   ...    if line.find('Version: ') == 0:
> >   ...      version = line.split()[-1]
>
> The problem with using get_provider() and a module name, is that it cannot
> conclusively associate a package with a specific project.  Thus, your code
> above has a bug: it will not find a version of a package that's installed
> by a system packaging tool such as RPM, or a bdist .exe or .msi on Windows.
>
> You need to use a project name instead of a package/module name, e.g.:
>
>     version = pkg_resources.require('ProjectNameHere')[0].version
>
> This will work even with projects installed using system packaging tools,
> as it explicitly indicates what project's PKG-INFO will be read, if in fact
> it needs to be read at all.  (In most cases, pkg_resources will get the
> project's version number from a filename, without needing to open any files.)
>
>

From theller at ctypes.org  Fri Mar 16 15:29:33 2007
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 16 Mar 2007 15:29:33 +0100
Subject: [Distutils] restructured text formatting not present in
 generated MS Windows binary installer
In-Reply-To: <45F9A8EC.3030106@noaa.gov>
References: <45F9A8EC.3030106@noaa.gov>
Message-ID: <ete9kk$m9v$1@sea.gmane.org>

Jim Vickroy schrieb:
> Hello,
> 
> I have been unsuccessful in using restructured text formatting tags in 
> the long_description parameter for the distutils.core.setup function.
> 
> When the attached "setup" script (see GOES-13-SXI-processor-installer.py 
> attachment) is run from the command line as:
> 
>  > GOES-13-SXI-processor-installer.py clean bdist_wininst --title="FITS 
> Images Processor" --bitmap=GOES-13-SXI-Processor.bmp
> 
> the first panel of the generated win32 installer has no boldface text 
> (see installer.png attachment)
> 
> When I feed the long_description text to docutils/tools/rst2html.py, the 
> generated html is as expected.
> 
> My setup is:
> 
> Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on 
> win32
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import docutils
>  >>> docutils.__version__
> '0.5'
>  >>>
> 
> on a MS Windows XP machine running service pack 2.
> 
> I am a new user of both distutils and docutils so any thoughts on what I 
> am doing incorrectly would be appreciated.

The bdist_wininst installer can only display pure text, it cannot render HTML
or some other format.  And restructured text is supposed to be readable.

Thomas


From theller at ctypes.org  Fri Mar 16 15:46:36 2007
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 16 Mar 2007 15:46:36 +0100
Subject: [Distutils] restructured text formatting not present in
 generated MS Windows binary installer
In-Reply-To: <45FAAC1D.3070003@noaa.gov>
References: <45F9A8EC.3030106@noaa.gov> <ete9kk$m9v$1@sea.gmane.org>
	<45FAAC1D.3070003@noaa.gov>
Message-ID: <45FAADCC.6040809@ctypes.org>

[Add CC to distutils-sig again]
>> Jim Vickroy schrieb:
>>   
>>> Hello,
>>>
>>> I have been unsuccessful in using restructured text formatting tags in 
>>> the long_description parameter for the distutils.core.setup function.
>>>
>>> When the attached "setup" script (see GOES-13-SXI-processor-installer.py 
>>> attachment) is run from the command line as:
>>>
>>>  > GOES-13-SXI-processor-installer.py clean bdist_wininst --title="FITS 
>>> Images Processor" --bitmap=GOES-13-SXI-Processor.bmp
>>>
>>> the first panel of the generated win32 installer has no boldface text 
>>> (see installer.png attachment)
>>>
>>> When I feed the long_description text to docutils/tools/rst2html.py, the 
>>> generated html is as expected.
>>>

Jim Vickroy schrieb:
> Thanks much, Thomas.
> 
> I think this piece of wisdom would be useful to add to the distutils 
> document, or is that limitation obvious to everyone but me?

Well, you're the first one to ask this question ;-).  I have no idea where
you got the impression from that using rest in the long_description should
give visual markup in the bdist_wininst installer.  Maybe that would be the
place where a hint like 'distutils does not interpret rest' would be usefull.


Thomas

From Felix.Wiemann at ososo.de  Fri Mar 16 21:17:33 2007
From: Felix.Wiemann at ososo.de (Felix Wiemann)
Date: Fri, 16 Mar 2007 16:17:33 -0400
Subject: [Distutils] restructured text formatting not present in
 generated MS Windows binary installer
In-Reply-To: <45FAADCC.6040809@ctypes.org>
References: <45F9A8EC.3030106@noaa.gov>
	<ete9kk$m9v$1@sea.gmane.org>	<45FAAC1D.3070003@noaa.gov>
	<45FAADCC.6040809@ctypes.org>
Message-ID: <45FAFB5D.5060804@ososo.de>

Thomas Heller wrote:

> I have no idea where
> you got the impression from that using rest in the long_description should
> give visual markup in the bdist_wininst installer.

Well, the Cheeseshop uses reStructuredText for the long_description.

Best wishes,

    Felix

-- 
Felix Wiemann -- http://www.ososo.de/

From jim at zope.com  Sat Mar 17 16:01:55 2007
From: jim at zope.com (Jim Fulton)
Date: Sat, 17 Mar 2007 11:01:55 -0400
Subject: [Distutils] Buildout configurations in a subdirectory stopped
	working.
In-Reply-To: <319e029f0703160223r717e04b4u5d51e3f31a408819@mail.gmail.com>
References: <319e029f0703160223r717e04b4u5d51e3f31a408819@mail.gmail.com>
Message-ID: <FC02B4FD-A826-4773-B49E-28F4F4136209@zope.com>


On Mar 16, 2007, at 5:23 AM, Lennart Regebro wrote:

> On 2/19/07, Christian Theune <ct at gocept.com> wrote:
>> [buildout]
>> extends = profiles/dev.cfg
>>
>> and run bin/buildout.
>>
>> This works very well for us.
>
> It did for me too, but it has now stopped working.

I don't know what "it" is.  I have no problem extending a  
configuration file in a subdirectory.  In fact, i tried the example  
above (as much as is shown) without difficulty.

> Line 89 in buildout.py says:
>
>             data['buildout']['directory'] = os.path.dirname 
> (config_file)
>
> This must be a recent change (< 2 weeks)

No.  Buildout has always set the default buildout directory to the  
directory containing the configuration file given on the command line  
or to the current working directory.

BTW, svn blame shows that that line was last modified on Dec 7.


> because this makes buildout
> fail if you have the cfg files anywhere else than root.

I can't reproduce this.  Can you construct a minimal example that  
illustrates this?

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From regebro at gmail.com  Sat Mar 17 16:16:49 2007
From: regebro at gmail.com (Lennart Regebro)
Date: Sat, 17 Mar 2007 16:16:49 +0100
Subject: [Distutils] Buildout configurations in a subdirectory stopped
	working.
In-Reply-To: <FC02B4FD-A826-4773-B49E-28F4F4136209@zope.com>
References: <319e029f0703160223r717e04b4u5d51e3f31a408819@mail.gmail.com>
	<FC02B4FD-A826-4773-B49E-28F4F4136209@zope.com>
Message-ID: <319e029f0703170816j7015c0f8x709b361ece1d912d@mail.gmail.com>

On 3/17/07, Jim Fulton <jim at zope.com> wrote:
> I don't know what "it" is.  I have no problem extending a
> configuration file in a subdirectory.  In fact, i tried the example
> above (as much as is shown) without difficulty.

Extending is no problem, it's using thats the problem. :-)

> No.  Buildout has always set the default buildout directory to the
> directory containing the configuration file given on the command line
> or to the current working directory.
>
> BTW, svn blame shows that that line was last modified on Dec 7.

Thats very weird, since just two weeks ago I was able to have
configuration files in a subdirectory and have the buildout directory
be the well, buildout directory...

One difference was that I first had the configuration file in the
root, and then moved them. This time I checked it all out from
scratch...

> I can't reproduce this.  Can you construct a minimal example that
> illustrates this?

OK, I'll try.

-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

From jim at zope.com  Sat Mar 17 17:29:07 2007
From: jim at zope.com (Jim Fulton)
Date: Sat, 17 Mar 2007 12:29:07 -0400
Subject: [Distutils] Improved error handling in zc.buildout
In-Reply-To: <etcnan$83f$1@sea.gmane.org>
References: <B5A15B2F-D24C-4F30-815A-E18CF2A631DF@zope.com>
	<etcnan$83f$1@sea.gmane.org>
Message-ID: <4416BF70-BBEF-445E-950E-553BE734A6C6@zope.com>


On Mar 15, 2007, at 8:10 PM, Martin Aspeli wrote:

> Jim Fulton wrote:
>> A major annoyance in zc.buildout has been its poor error handling.
>> Generally, when an error occurred, a a traceback would be printed
>> giving little or no clue about what went wrong or what was going on.
>> Previously, I was focussed on important features.  The latest release
>> focusses on better error reporting and debugging support.  I'd like
>> to continue to provide some emphasis on this, so feedback would be
>> especially appreciated now that I think I've improved error handling
>> quite a bit.
>
> That's great news, Jim!
>
> I had a particular problem before, where the "Plone" egg (which  
> includes
> all the Products.* packages that make up plone, and all the other  
> proper
> eggs as dependencies) would try to compile .py files in filesystem  
> skin
> folders (i.e. Zope Script (Python)'s) and get compile errors (return
> outside function, for example). With workingenv and easy_install, it'd
> report the errors but not stop;

I wonder what you mean by "stop" in this context.  Normally,  
easy_install is used to install a package at a time.  Are you saying  
that easy_install continued to fetch other distributions as  
dependencies even after this error occurs?

> buildout seems to stop.

When buildout invokes setuptools, it does so as a sub-process.  It  
treats a non-0 exit status from the subprocess as an error.


> Better error
> handling would hopefully help diagnose this.

Perhaps the absence of a traceback will make it easier to see errors  
spewed when running the package setup scripts.

> We couldn't find a way to
> stop setuptools from trying to compile those files,

I wonder how setuptools (or more likely distutils) decides what files  
to compile.

(I've always been annoyed by the Zope Python script syntax which  
isn't valid Python. :()

I wonder if it would make sense to get Zope to use a different suffix  
for these things, since they aren't *really* Python code.

> or a way to get  buildout to ignore the errors.

Generally, it seems like a bad idea for buildout to ignore the exit  
status of running easy install.  Perhaps, as Phillip suggested, I  
should try using the --record option and decide that things are fine  
if files actually get installed.  I'll look into that.  It seems off  
that easy_install exits with a non-0 exit status for what is,  
essentially, a warning.

Could you please report this as a buildout bug:

   https://bugs.launchpad.net/zc.buildout/+filebug

including enough information for me to easily reproduce the problem?   
Pointing me at a sample buildout I can check  out and try might be a  
good way to do this.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From daniel.nouri at gmail.com  Sat Mar 17 17:50:17 2007
From: daniel.nouri at gmail.com (Daniel Nouri)
Date: Sat, 17 Mar 2007 17:50:17 +0100
Subject: [Distutils] Improved error handling in zc.buildout
In-Reply-To: <4416BF70-BBEF-445E-950E-553BE734A6C6@zope.com>
References: <B5A15B2F-D24C-4F30-815A-E18CF2A631DF@zope.com>	<etcnan$83f$1@sea.gmane.org>
	<4416BF70-BBEF-445E-950E-553BE734A6C6@zope.com>
Message-ID: <45FC1C49.3000402@gmail.com>

Jim Fulton wrote:
>> We couldn't find a way to
>> stop setuptools from trying to compile those files,
> 
> I wonder how setuptools (or more likely distutils) decides what files  
> to compile.

The last time I looked, setuptools (or was it easy_install?) was simply 
looking for *.py files.

> (I've always been annoyed by the Zope Python script syntax which  
> isn't valid Python. :()
> 
> I wonder if it would make sense to get Zope to use a different suffix  
> for these things, since they aren't *really* Python code.

I vote for .zpy :)


Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: daniel.nouri.vcf
Type: text/x-vcard
Size: 160 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070317/f5a6b9c2/attachment.vcf 

From jim at zope.com  Sun Mar 18 00:01:51 2007
From: jim at zope.com (Jim Fulton)
Date: Sat, 17 Mar 2007 19:01:51 -0400
Subject: [Distutils] pakage index __getitem__
Message-ID: <ACD4AF8A-5216-4510-8A54-DD077F274840@zope.com>


I want to get a list of all of the distributions I can find that  
satisfy a requirement.  I've found the following works:

   index.obtain(requirement)
   dists = [d for d in index[requirement.project_name] if d in  
requirement]

where index is a setuptools.package_index.PackageIndex.

Is there a better way?  The obtain call seems to be necessary to  
initialize the index for the given project.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From pje at telecommunity.com  Sun Mar 18 01:25:47 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 17 Mar 2007 19:25:47 -0500
Subject: [Distutils] pakage index __getitem__
In-Reply-To: <ACD4AF8A-5216-4510-8A54-DD077F274840@zope.com>
Message-ID: <5.1.1.6.0.20070317192034.039a8008@sparrow.telecommunity.com>

At 07:01 PM 3/17/2007 -0400, Jim Fulton wrote:
>I want to get a list of all of the distributions I can find that
>satisfy a requirement.  I've found the following works:
>
>    index.obtain(requirement)
>    dists = [d for d in index[requirement.project_name] if d in
>requirement]
>
>where index is a setuptools.package_index.PackageIndex.
>
>Is there a better way?

There are other ways, but the above is the simplest and most direct way, 
given your use case.  Normally, performing a resolve() or require() on a 
working set will end up invoking obtain() indirectly, if a requirement 
can't otherwise be met -- this allows network access to be lazy, so it's 
not done unless it's really needed.


>   The obtain call seems to be necessary to
>initialize the index for the given project.

Yes - Environment normally calls 'obtain()' in response to a 'best_match()' 
operation, which is in turn called by a WorkingSet's 'resolve()' 
method.  The idea is that you can subclass Environment (in this case, to 
create a PackageIndex) and override 'obtain()' to find packages in 
locations that are not part of the working set used to create the Environment.


From bob at redivi.com  Mon Mar 19 00:15:53 2007
From: bob at redivi.com (Bob Ippolito)
Date: Sun, 18 Mar 2007 16:15:53 -0700
Subject: [Distutils] Optional C extensions and setuptools
Message-ID: <6a36e7290703181615x6d12759cs32b7454b6be8b4a0@mail.gmail.com>

I've just added a small optional C extension to simplejson to speed up
decoding, but I have a feeling that it's going to cause problems for
Windows users.

Right now it's specified as setuptools Feature the same way that
PyProtocols' speedups module is implemented.

Is there something I can do in the setup.py to fail gracefully and
install anyways if it can't compile the extension? In some way that's
compatible with easy_install?

-bob

From radix at twistedmatrix.com  Mon Mar 19 01:08:50 2007
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Sun, 18 Mar 2007 20:08:50 -0400
Subject: [Distutils] Optional C extensions and setuptools
In-Reply-To: <6a36e7290703181615x6d12759cs32b7454b6be8b4a0@mail.gmail.com>
References: <6a36e7290703181615x6d12759cs32b7454b6be8b4a0@mail.gmail.com>
Message-ID: <60ed19d40703181708q3f2517ccu57ff9df96376ae02@mail.gmail.com>

On 3/18/07, Bob Ippolito <bob at redivi.com> wrote:
> I've just added a small optional C extension to simplejson to speed up
> decoding, but I have a feeling that it's going to cause problems for
> Windows users.
>
> Right now it's specified as setuptools Feature the same way that
> PyProtocols' speedups module is implemented.
>
> Is there something I can do in the setup.py to fail gracefully and
> install anyways if it can't compile the extension? In some way that's
> compatible with easy_install?

Twisted does this: it subclasses build_ext and overrides
build_extensions to check if there's a C compiler before upcalling.


-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From bob at redivi.com  Mon Mar 19 01:31:11 2007
From: bob at redivi.com (Bob Ippolito)
Date: Sun, 18 Mar 2007 17:31:11 -0700
Subject: [Distutils] Optional C extensions and setuptools
In-Reply-To: <60ed19d40703181708q3f2517ccu57ff9df96376ae02@mail.gmail.com>
References: <6a36e7290703181615x6d12759cs32b7454b6be8b4a0@mail.gmail.com>
	<60ed19d40703181708q3f2517ccu57ff9df96376ae02@mail.gmail.com>
Message-ID: <6a36e7290703181731p1c48a8aeve7aa19891b276ce9@mail.gmail.com>

On 3/18/07, Christopher Armstrong <radix at twistedmatrix.com> wrote:
> On 3/18/07, Bob Ippolito <bob at redivi.com> wrote:
> > I've just added a small optional C extension to simplejson to speed up
> > decoding, but I have a feeling that it's going to cause problems for
> > Windows users.
> >
> > Right now it's specified as setuptools Feature the same way that
> > PyProtocols' speedups module is implemented.
> >
> > Is there something I can do in the setup.py to fail gracefully and
> > install anyways if it can't compile the extension? In some way that's
> > compatible with easy_install?
>
> Twisted does this: it subclasses build_ext and overrides
> build_extensions to check if there's a C compiler before upcalling.
>

I see. Took me a while to find it in that mess, but it looks like that
might work if necessary.

Anyone have something that does it a little more directly?

-bob

From jim at zope.com  Mon Mar 19 15:29:48 2007
From: jim at zope.com (Jim Fulton)
Date: Mon, 19 Mar 2007 10:29:48 -0400
Subject: [Distutils] Improved error handling in zc.buildout
In-Reply-To: <etcnan$83f$1@sea.gmane.org>
References: <B5A15B2F-D24C-4F30-815A-E18CF2A631DF@zope.com>
	<etcnan$83f$1@sea.gmane.org>
Message-ID: <ACB3B13C-28AA-4BB6-BD55-1194848ADE2E@zope.com>


On Mar 15, 2007, at 8:10 PM, Martin Aspeli wrote:

> Jim Fulton wrote:
>> A major annoyance in zc.buildout has been its poor error handling.
>> Generally, when an error occurred, a a traceback would be printed
>> giving little or no clue about what went wrong or what was going on.
>> Previously, I was focussed on important features.  The latest release
>> focusses on better error reporting and debugging support.  I'd like
>> to continue to provide some emphasis on this, so feedback would be
>> especially appreciated now that I think I've improved error handling
>> quite a bit.
>
> That's great news, Jim!
>
> I had a particular problem before, where the "Plone" egg (which  
> includes
> all the Products.* packages that make up plone, and all the other  
> proper
> eggs as dependencies) would try to compile .py files in filesystem  
> skin
> folders (i.e. Zope Script (Python)'s) and get compile errors (return
> outside function, for example). With workingenv and easy_install, it'd
> report the errors but not stop; buildout seems to stop. Better error
> handling would hopefully help diagnose this. We couldn't find a way to
> stop setuptools from trying to compile those files, or a way to get
> buildout to ignore the errors.

I've tried and been unable to reproduce this scenario.

Can you provide a buildout that illustrates this?

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org




From nathan at yergler.net  Tue Mar 20 16:10:34 2007
From: nathan at yergler.net (Nathan R. Yergler)
Date: Tue, 20 Mar 2007 11:10:34 -0400
Subject: [Distutils] Detecting if a package was installed with an "extra"
Message-ID: <bf7b44d50703200810y4a9e995bh475a06f9b2f54a13@mail.gmail.com>

I'm working on an improvement to my RDFa parser and because it will
rely on an external tool, I'm using an extra to require it to be
explicitly required.  Is there a way to tell at runtime if an egg (for
the code running) was installed with an extra.  For example, detecting
the difference between:

easy_install foo[bar]

and

easy_install foo

when in foo.py?

Thanks,

Nathan

From pje at telecommunity.com  Tue Mar 20 16:59:04 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Tue, 20 Mar 2007 10:59:04 -0500
Subject: [Distutils] Detecting if a package was installed with an "extra"
In-Reply-To: <bf7b44d50703200810y4a9e995bh475a06f9b2f54a13@mail.gmail.co
 m>
Message-ID: <5.1.1.6.0.20070320105635.04cfec00@sparrow.telecommunity.com>

At 11:10 AM 3/20/2007 -0400, Nathan R. Yergler wrote:
>I'm working on an improvement to my RDFa parser and because it will
>rely on an external tool, I'm using an extra to require it to be
>explicitly required.  Is there a way to tell at runtime if an egg (for
>the code running) was installed with an extra.  For example, detecting
>the difference between:
>
>easy_install foo[bar]
>
>and
>
>easy_install foo
>
>when in foo.py?

First answer that comes to mind:

     try:
         pkg_resources.require('foo[bar]')
     except pkg_resources.ResolutionError:
         # it wasn't installed, or there's a version conflict
     else:
         # good to go

There might be other ways, but this is what I'd do.

Note that for entry points (including scripts), the entry point can depend 
on extras, and this means that trying to load (or .require()) the entry 
point will raise similar exceptions if the extra isn't found.


From strawman at astraw.com  Thu Mar 22 02:59:35 2007
From: strawman at astraw.com (Andrew Straw)
Date: Wed, 21 Mar 2007 18:59:35 -0700
Subject: [Distutils] Optional C extensions and setuptools
In-Reply-To: <6a36e7290703181731p1c48a8aeve7aa19891b276ce9@mail.gmail.com>
References: <6a36e7290703181615x6d12759cs32b7454b6be8b4a0@mail.gmail.com>	<60ed19d40703181708q3f2517ccu57ff9df96376ae02@mail.gmail.com>
	<6a36e7290703181731p1c48a8aeve7aa19891b276ce9@mail.gmail.com>
Message-ID: <4601E307.7070404@astraw.com>

Bob Ippolito wrote:
> On 3/18/07, Christopher Armstrong <radix at twistedmatrix.com> wrote:
>   
>> On 3/18/07, Bob Ippolito <bob at redivi.com> wrote:
>>     
>>> I've just added a small optional C extension to simplejson to speed up
>>> decoding, but I have a feeling that it's going to cause problems for
>>> Windows users.
>>>
>>> Right now it's specified as setuptools Feature the same way that
>>> PyProtocols' speedups module is implemented.
>>>
>>> Is there something I can do in the setup.py to fail gracefully and
>>> install anyways if it can't compile the extension? In some way that's
>>> compatible with easy_install?
>>>       
>> Twisted does this: it subclasses build_ext and overrides
>> build_extensions to check if there's a C compiler before upcalling.
>>
>>     
>
> I see. Took me a while to find it in that mess, but it looks like that
> might work if necessary.
>
> Anyone have something that does it a little more directly?
>
>   
See *class* ve_build_ext in 
http://visionegg.org/trac/browser/trunk/visionegg/setup.py

From public.marvin at gmail.com  Fri Mar 23 17:04:53 2007
From: public.marvin at gmail.com (Marvin)
Date: Fri, 23 Mar 2007 12:04:53 -0400
Subject: [Distutils] distutils bug(?): using absolute path for --home ,
 python 2.5 on windows
Message-ID: <4603FAA5.6020806@gmail.com>

OK, at least part of my confusion is the whole cygwin/not-cygwin thing.
I'm using distutils to install a package (PyGUI) like this:

PyGUI-1.7.2 $ c:/Python2.5/python setup.py install --home=c:/GUI-test-install

Initially I was giving --home=/cygdrive/c/GUI-test-install install (in the 
depths of some Makefile) and getting the 'cannot be absolute' message.  Fixing 
that path, though (as above), the problem is that it DOES install, but actually 
into a relative path, PyGUI-1.7.2/GUI-test-install

Isn't the whole point of --home to ALLOW an absolute path?

Note also that doing something analogous on linux (with an absolute path) works 
fine.

OK, and I've figured it out.  I need to give it a path c:\\GUI-test-install. 
Which is not shocking, but certainly it would be NICE if distutils could treat 
'/' and '\\' as synonyms -- the underlying windows API does back since DOS.

Never mind.  :-)

Here's some output with the still somewhat incorrect path for --home:

...
running install_lib
creating c:GUI-test-install
creating c:GUI-test-install\lib
creating c:GUI-test-install\lib\python
creating c:GUI-test-install\lib\python\GUI
creating c:GUI-test-install\lib\python\GUI\Cocoa
copying build\lib\GUI\Cocoa\Applications.py -> 
c:GUI-test-install\lib\python\GUI\Cocoa
...



From plone at hannosch.info  Sat Mar 24 00:52:14 2007
From: plone at hannosch.info (Hanno Schlichting)
Date: Sat, 24 Mar 2007 00:52:14 +0100
Subject: [Distutils] setuptools and sdist on Mac OS X, tarfile  flukes
In-Reply-To: <5.1.1.6.0.20070215150738.05224e48@sparrow.telecommunity.com>
References: <loom.20070215T182753-192@post.gmane.org>
	<5.1.1.6.0.20070215150738.05224e48@sparrow.telecommunity.com>
Message-ID: <4604682E.9040303@hannosch.info>

Hi,

Phillip J. Eby wrote:
> At 05:38 PM 2/15/2007 +0000, Sidnei da Silva wrote:
>> Not sure if anyone reported a similar issue here since I'm not subscribed, but
>> here it goes.
>>
>> Some of the new plone.* eggs are being built on OS X by the developers, they
>> usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py
>> sdist upload'
>>
>> Turns out that if you try to easy_install and there's only the sdist package,
>> but no egg, easy_install uses tarfile to unpack the package, build an egg and
>> install the egg. However, for some reason some of those nasty '._<filename>'
>> files that OS X creates end up in the tarball. And tarfile then fails to 
>> unpack
>> the tarball. One such package is 'plone.app.controlpanel 1.0a2'
>> (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone.app.controlpanel-1.0a2.tar.gz#md5=2b2afeaba0d067f6b1a6707c6d69311d),
>> with the following traceback:
>> ...
>> I'm left wondering if the issue is:
>>
>>  a) Windows
>>  b) The nasty OS X file
>>  c) The Python 'tarfile' module
>>  d) setuptools
>>  e) None of the above :)
> 
> It appears to be a combination of "a" and "c".  Windows Python 2.5 does not 
> have an issue with the above file, nor does Cygwin Python 2.3.  Windows 
> Python 2.3 and 2.4 do have the problem.
> 
>> Thoughts?
> 
> Dunno.  It sounds like the people making these distributions have an 
> over-eager MANIFEST.in or an over-eager source control system.  They should 
> probably add a MANIFEST.in rule to exclude the files from the source release.

I looked into this a bit more and it seems you cannot fix this issue by
using a MANIFEST.in. The problem is that those resource forks aren't
found as regular files and so in return you cannot exclude them via a
filter.

According to
http://forums.macosxhints.com/archive/index.php/t-43243.html those
resource forks are created by the Mac OS tar program on the fly, which
is used internally by distutils.archive_util.

There is no command line switch to disable this behavior but some brave
soul looked through the source code of the modified tar and found that
setting an environment variable called
'COPY_EXTENDED_ATTRIBUTES_DISABLE' to any value should disable this
behavior.

I just tried to do a 'python setup.py sdist' for plone.app.controlpanel
and a 'tar tzvf plone.app.controlpanel-1.0b3dev-r13784.tar.gz' shows me
a ._setup.py file in there.

If I first do 'export COPY_EXTENDED_ATTRIBUTES_DISABLE=true' and then
rerun the sdist command I don't see the ._setup.py file anymore in the
tarball.

I wonder if this should be fixed on the distutils level? Does including
those resource forks make any sense for source tarballs in some
situation? Or could the archive_util set this environment variable on
Mac OS X prior to calling tar and reset it afterwards?

Hanno


From bob at redivi.com  Sat Mar 24 01:05:12 2007
From: bob at redivi.com (Bob Ippolito)
Date: Fri, 23 Mar 2007 17:05:12 -0700
Subject: [Distutils] setuptools and sdist on Mac OS X, tarfile flukes
In-Reply-To: <4604682E.9040303@hannosch.info>
References: <loom.20070215T182753-192@post.gmane.org>
	<5.1.1.6.0.20070215150738.05224e48@sparrow.telecommunity.com>
	<4604682E.9040303@hannosch.info>
Message-ID: <6a36e7290703231705n76a5f048me2b067eb0fbff3d0@mail.gmail.com>

On 3/23/07, Hanno Schlichting <plone at hannosch.info> wrote:
> Hi,
>
> Phillip J. Eby wrote:
> > At 05:38 PM 2/15/2007 +0000, Sidnei da Silva wrote:
> >> Not sure if anyone reported a similar issue here since I'm not subscribed, but
> >> here it goes.
> >>
> >> Some of the new plone.* eggs are being built on OS X by the developers, they
> >> usually do 'setup.py sdist bdist_egg upload'. Sometimes they only do 'setup.py
> >> sdist upload'
> >>
> >> Turns out that if you try to easy_install and there's only the sdist package,
> >> but no egg, easy_install uses tarfile to unpack the package, build an egg and
> >> install the egg. However, for some reason some of those nasty '._<filename>'
> >> files that OS X creates end up in the tarball. And tarfile then fails to
> >> unpack
> >> the tarball. One such package is 'plone.app.controlpanel 1.0a2'
> >> (http://cheeseshop.python.org/packages/source/p/plone.app.controlpanel/plone.app.controlpanel-1.0a2.tar.gz#md5=2b2afeaba0d067f6b1a6707c6d69311d),
> >> with the following traceback:
> >> ...
> >> I'm left wondering if the issue is:
> >>
> >>  a) Windows
> >>  b) The nasty OS X file
> >>  c) The Python 'tarfile' module
> >>  d) setuptools
> >>  e) None of the above :)
> >
> > It appears to be a combination of "a" and "c".  Windows Python 2.5 does not
> > have an issue with the above file, nor does Cygwin Python 2.3.  Windows
> > Python 2.3 and 2.4 do have the problem.
> >
> >> Thoughts?
> >
> > Dunno.  It sounds like the people making these distributions have an
> > over-eager MANIFEST.in or an over-eager source control system.  They should
> > probably add a MANIFEST.in rule to exclude the files from the source release.
>
> I looked into this a bit more and it seems you cannot fix this issue by
> using a MANIFEST.in. The problem is that those resource forks aren't
> found as regular files and so in return you cannot exclude them via a
> filter.
>
> According to
> http://forums.macosxhints.com/archive/index.php/t-43243.html those
> resource forks are created by the Mac OS tar program on the fly, which
> is used internally by distutils.archive_util.
>
> There is no command line switch to disable this behavior but some brave
> soul looked through the source code of the modified tar and found that
> setting an environment variable called
> 'COPY_EXTENDED_ATTRIBUTES_DISABLE' to any value should disable this
> behavior.
>
> I just tried to do a 'python setup.py sdist' for plone.app.controlpanel
> and a 'tar tzvf plone.app.controlpanel-1.0b3dev-r13784.tar.gz' shows me
> a ._setup.py file in there.
>
> If I first do 'export COPY_EXTENDED_ATTRIBUTES_DISABLE=true' and then
> rerun the sdist command I don't see the ._setup.py file anymore in the
> tarball.
>
> I wonder if this should be fixed on the distutils level? Does including
> those resource forks make any sense for source tarballs in some
> situation? Or could the archive_util set this environment variable on
> Mac OS X prior to calling tar and reset it afterwards?

I haven't seen a resource fork used for anything of consequence in
many years. I don't think anyone using sdist would ever need them
included... Settings that env var seems like the right thing to do.

-bob

From ocfxs at chi.leoburnett.com  Tue Mar 27 02:57:24 2007
From: ocfxs at chi.leoburnett.com (Mobley)
Date: Mon, 26 Mar 2007 16:57:24 -0800
Subject: [Distutils] scary
Message-ID: <46086BF4.4050609@aeclp.com>

An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070326/f325d5f8/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thanks.gif
Type: image/gif
Size: 24119 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070326/f325d5f8/attachment-0001.gif 

From d.w.morriss at gmail.com  Tue Mar 27 21:03:34 2007
From: d.w.morriss at gmail.com (whit)
Date: Tue, 27 Mar 2007 14:03:34 -0500
Subject: [Distutils] Setting environment variables before building eggs
 w/ extensions using zc.buildout
In-Reply-To: <espsn9$ohn$1@sea.gmane.org>
References: <bf7b44d50703071107u2ad7df6fidb3c2d2922fca9e8@mail.gmail.com>	<esn4ev$6sc$1@sea.gmane.org>	<esn6eq$f5u$1@sea.gmane.org>
	<espsn9$ohn$1@sea.gmane.org>
Message-ID: <eubpq8$rgl$1@sea.gmane.org>

Martin Aspeli wrote:
> whit wrote:
> 
>> fyi, the deliverance.buildout will soon go away
> 
> Well, I don't think it has much value for re-usability of Deliverance 
> anyway.
> 
> Do you have an alternate distribution mechanism? Perhaps one we could 
> leverage inside a buildout? :)
> 
> Martin
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 

indeed... it will be replaced by a buildit driven task that could be 
also driven from buildout. not sure on the eta.

-w

-- 

------ d. whit morriss ------
- senior engineer, opencore -
- http://www.openplans.org  -
- m: 415-710-8975           -

"If you don't know where you are,
  you don't know anything at all" 


Dr. Edgar Spencer, Ph.D., 1995


From tepperly at llnl.gov  Tue Mar 27 23:19:55 2007
From: tepperly at llnl.gov (Tom Epperly)
Date: Tue, 27 Mar 2007 14:19:55 -0700 (PDT)
Subject: [Distutils] Concurrent distutils builds can fail due to mkdir
Message-ID: <Pine.LNX.4.58.0703271409550.7223@tux163.llnl.gov>

I have a build where two concurrent Python processes are both building C
extension modules using distutils in the same directory.  I have a "python
setup.py build_ext" and a "python synch_setup.py build_ext" both in the
same directory. Each setup.py refers to different files that need to be
built, but occasionally one of the builds fails because it fails when
executing mkdir("build"). With two concurrent distutils building in the
same directory, there is a race condition when making the "build"
directories.

Here is an excerpt from dir_util.py:

        if not dry_run:
            try:
                os.mkdir(head)
                created_dirs.append(head)
            except OSError, exc:
                raise DistutilsFileError, \
                      "could not create '%s': %s" % (head, exc[-1])

mkdir can fail for two reasons -- one because the directory already exists
and two because it couldn't create the directory. It would be nice if the
except clause would ignore the failure when the directory already exists
by testing if the directory exists and is accessible. This would prevent
one of the setup.py's from stopping when it could really safely continue.

I am not a member of the distutils-sig mailing list, so please Cc me on
replies.

Thanks,

Tom

--
------------------------------------------------------------------------
Tom Epperly
Center for Applied Scientific Computing   Phone: 925-424-3159
Lawrence Livermore National Laboratory      Fax: 925-424-2477
L-661, P.O. Box 808, Livermore, CA 94551  Email: tepperly at llnl.gov
------------------------------------------------------------------------

From Hui.Zhang at ucd.ie  Wed Mar 28 14:46:08 2007
From: Hui.Zhang at ucd.ie (Hui Zhang)
Date: Wed, 28 Mar 2007 13:46:08 +0100
Subject: [Distutils] help please
Message-ID: <f8beeb7f5c45.5c45f8beeb7f@ucd.ie>

Dear Phillip,

  Hello. I am a phd student from Unviersity College Dublin,Ireland. I 
have a problem when I tried to install python with setuptools egg files 
in cygwin.
  I did as the webpage(http://cheeseshop.python.org/pypi/setuptools)
says. However the
system said:
--------------------------------------------------------------------
  $sh setuptools-0.6c5-py2.4.egg
  [[: not found
  setuptools-0.6c5-py2.4egg is not the correct name for this egg file.
  Please rename it back to setuptools-0.6c5-py2.4 egg and try again.
 
---------------------------------------------------------------------
  I also tried the other version which is setuptools-0.6c5-py2.4.egg 
and it
didn't work either. Is this because I am using 'cygwin' not 'linux'?
  
  Thank you and any feedback will be much appreciated.
  Regards.
     
            Hui

From matt at matt-good.net  Wed Mar 28 16:22:03 2007
From: matt at matt-good.net (Matt Good)
Date: Wed, 28 Mar 2007 10:22:03 -0400
Subject: [Distutils] help please
In-Reply-To: <f8beeb7f5c45.5c45f8beeb7f@ucd.ie>
References: <f8beeb7f5c45.5c45f8beeb7f@ucd.ie>
Message-ID: <1175091723.6020.23.camel@nny>

On Wed, 2007-03-28 at 13:46 +0100, Hui Zhang wrote: 
> Dear Phillip,
> 
>   Hello. I am a phd student from Unviersity College Dublin,Ireland. I 
> have a problem when I tried to install python with setuptools egg files 
> in cygwin.
>   I did as the webpage(http://cheeseshop.python.org/pypi/setuptools)
> says. However the
> system said:
> --------------------------------------------------------------------
>   $sh setuptools-0.6c5-py2.4.egg
>   [[: not found
>   setuptools-0.6c5-py2.4egg is not the correct name for this egg file.
>   Please rename it back to setuptools-0.6c5-py2.4 egg and try again.
>  
> ---------------------------------------------------------------------

This version of the egg actually requires bash and is not quite
compatible with sh.  This is fixed in SVN, but for now run it with bash
instead.

-- Matt Good


From Hui.Zhang at ucd.ie  Wed Mar 28 17:13:27 2007
From: Hui.Zhang at ucd.ie (Hui Zhang)
Date: Wed, 28 Mar 2007 16:13:27 +0100
Subject: [Distutils] help please
In-Reply-To: <1175091723.6020.23.camel@nny>
References: <f8beeb7f5c45.5c45f8beeb7f@ucd.ie> <1175091723.6020.23.camel@nny>
Message-ID: <f9bb8a1a1dc6.1dc6f9bb8a1a@ucd.ie>

Hey Matt,

  Thanks so much for your reply. It works now!
  I am stuck on another probelm now. I tried to install python-devel in 
cygwin, but it seems that python-devel can not find where the python is 
and claims that a python is needed. 
  --------------------------------------------------------
  $ rpm -Uvh python-devel-2.3.4-1.m68kmint.rpm
  error: Failed dependencies:
          python = 2.3.4 is needed by python-devel-2.3.4-1
  ---------------------------------------------------------
  But acutally python 2.3.4 has been automatically installed at the 
same time when cygwin is installed.
  Have you met this problem before?
  I am looking forward to hearing from you. Many thanks again!!
  Regards.

          Hui

----- Original Message -----
From: Matt Good <matt at matt-good.net>
Date: Wednesday, March 28, 2007 3:22 pm
Subject: Re: [Distutils] help please
To: Hui Zhang <Hui.Zhang at ucd.ie>
Cc: distutils-sig at python.org

> On Wed, 2007-03-28 at 13:46 +0100, Hui Zhang wrote: 
> > Dear Phillip,
> > 
> >   Hello. I am a phd student from Unviersity College 
> Dublin,Ireland. I 
> > have a problem when I tried to install python with setuptools 
> egg files 
> > in cygwin.
> >   I did as the webpage(http://cheeseshop.python.org/pypi/setuptools)
> > says. However the
> > system said:
> > -----------------------------------------------------------------
> ---
> >   $sh setuptools-0.6c5-py2.4.egg
> >   [[: not found
> >   setuptools-0.6c5-py2.4egg is not the correct name for this egg 
> file.>   Please rename it back to setuptools-0.6c5-py2.4 egg and 
> try again.
> >  
> > -----------------------------------------------------------------
> ----
> 
> This version of the egg actually requires bash and is not quite
> compatible with sh.  This is fixed in SVN, but for now run it with 
> bashinstead.
> 
> -- Matt Good
> 
> 

From pje at telecommunity.com  Wed Mar 28 18:25:28 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Wed, 28 Mar 2007 11:25:28 -0500
Subject: [Distutils] help please
In-Reply-To: <f9bb8a1a1dc6.1dc6f9bb8a1a@ucd.ie>
References: <1175091723.6020.23.camel@nny> <f8beeb7f5c45.5c45f8beeb7f@ucd.ie>
	<1175091723.6020.23.camel@nny>
Message-ID: <5.1.1.6.0.20070328112316.04b37da8@sparrow.telecommunity.com>

At 04:13 PM 3/28/2007 +0100, Hui Zhang wrote:
>Hey Matt,
>
>   Thanks so much for your reply. It works now!
>   I am stuck on another probelm now. I tried to install python-devel in
>cygwin, but it seems that python-devel can not find where the python is
>and claims that a python is needed.
>   --------------------------------------------------------
>   $ rpm -Uvh python-devel-2.3.4-1.m68kmint.rpm
>   error: Failed dependencies:
>           python = 2.3.4 is needed by python-devel-2.3.4-1
>   ---------------------------------------------------------
>   But acutally python 2.3.4 has been automatically installed at the
>same time when cygwin is installed.

You don't need to use RPM - run the Cygwin setup program again and make 
sure you have Python set to include the source package, not just the 
binary.  That will ensure you have all the header files.  I don't believe 
Cygwin *has* a python-devel package.

In general, unless you really know what you're doing, you should always 
install or upgrade Cygwin packages using the setup program, or by 
installing them from source code.  Don't mix RPMs into the picture.


From JRBoverhof at lbl.gov  Wed Mar 28 22:43:53 2007
From: JRBoverhof at lbl.gov (Joshua Boverhof)
Date: Wed, 28 Mar 2007 13:43:53 -0700
Subject: [Distutils] setup_requires and install_requires
Message-ID: <48A370FC-B530-49B4-9CAC-EA259E431BA6@lbl.gov>

I mentioned this a while back.

If one specifies the same package in setup_requires and  
install_requires, it will not be installed.  Here is a simple script  
that demonstrates this behavior.

You could also dump setup_requires into the setup call and see the  
same behavior.

-josh



$ cat setup.py
import os, ez_setup
ez_setup.use_setuptools()
from setuptools import setup, find_packages
from setuptools.dist import Distribution

if not os.path.isdir('Dummy'):
     os.mkdir('Dummy')
     open('Dummy/__init__.py', 'w').write('# Dummy')

# Indirection so that setuptools grabs "setup_requires" dependencies
Distribution(
     dict(setup_requires="PasteScript >= 0.5",)
)
setup(name="Dummy",
       packages=find_packages(),
       install_requires=[
          "PasteScript >= 0.5",
         "TurboKid >= 0.9.3",
       ],
)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2526 bytes
Desc: not available
Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070328/6c2b6981/attachment.bin 

From venkatbo at yahoo.com  Sat Mar 31 04:05:24 2007
From: venkatbo at yahoo.com (Venkat Bommakanti)
Date: Fri, 30 Mar 2007 19:05:24 -0700 (PDT)
Subject: [Distutils] help please
In-Reply-To: <5.1.1.6.0.20070328112316.04b37da8@sparrow.telecommunity.com>
Message-ID: <20070331020524.21450.qmail@web30508.mail.mud.yahoo.com>

Hi folks,

'am using setup tools to build and package a TG 1.0.1
app. After deployment, I'm seeing a few entries in
SOURCES.txt despite specifying related wildcards in
the exclusion list.

Following are some related snippets:

setup.py:----------------
...
#
#from turbogears.finddata import find_package_data
#
# Disabled above import line, and added
find_package_data
# below to avoid installing TG on the build box...
#
import sys
from fnmatch import fnmatchcase
from distutils.util import convert_path

# Provided as an attribute, so you can append to these
instead
# of replicating them:
standard_exclude = ('*.py', '*.pyc', '*~', '.*',
'*.bak',
                    'checkout.status',
'original.status')
standard_exclude_directories = ('.*', 'CVS', 'CC',
'_darcs', './
build',
                                './dist', 'EGG-INFO',
'*.egg-info',
                                '.CC', 'cache',
'*.egg-info/.CC',
                                '*.egg-info/.CC',
'*.egg-info/.CC/*')

def find_package_data(
    where='.', package='',
    exclude=standard_exclude,
    exclude_directories=standard_exclude_directories,
    only_in_packages=True,
    show_ignored=False):
...

setup(
    name="myapp",
    version=version,
...

--------------------------------------

However, after successful build/deply, I'm seeing the
following entries in the deployed SOURCES.txt file
(myapp/myapp.egg-info/SOURCES.txt):
...
myapp.egg-info/.CC/checkout.status
myapp.egg-info/.CC/original.status
myapp.egg-info/.CC/cache/SOURCES.txt@@/main/1
...

---------------------------------------

It appears the exlusion list applies to all app
folders excpet myapp.egg-info/. Any idea how I can
specify that even in the case of myapp.egg-info/.CC/
it should include any diles or dirs...

Thanks much,
/venkat 

Full find_package_data:
-----------------------


def find_package_data(
    where='.', package='',
    exclude=standard_exclude,
    exclude_directories=standard_exclude_directories,
    only_in_packages=True,
    show_ignored=True):
    """
    Return a dictionary suitable for use in
``package_data``
    in a distutils ``setup.py`` file.

    The dictionary looks like::

        {'package': [files]}

    Where ``files`` is a list of all the files in that
package that
    don't match anything in ``exclude``.

    If ``only_in_packages`` is true, then top-level
directories that
    are not packages won't be included (but
directories under packages
    will).

    Directories matching any pattern in
``exclude_directories`` will
    be ignored; by default directories with leading
``.``, ``CVS``,
    and ``_darcs`` will be ignored.

    If ``show_ignored`` is true, then all the files
that aren't
    included in package data are shown on stderr (for
debugging
    purposes).

    Note patterns use wildcards, or can be exact paths
(including
    leading ``./``), and all searching is
case-insensitive.
    """

    out = {}
    stack = [(convert_path(where), '', package,
only_in_packages)]
    while stack:
        where, prefix, package, only_in_packages =
stack.pop(0)
        for name in os.listdir(where):
            fn = os.path.join(where, name)
            if os.path.isdir(fn):
                bad_name = False
                for pattern in exclude_directories:
                    if (fnmatchcase(name, pattern)
                        or fn.lower() ==
pattern.lower()):
                        bad_name = True
                        if show_ignored:
                            print >> sys.stderr, (
                                "Directory %s ignored
by pattern %s"
                                % (fn, pattern))
                            pass
                        break
                if bad_name:
                    continue
                if os.path.isfile(os.path.join(fn,
'__init__.py')):
                    if not package:
                        new_package = name
                    else:
                        new_package = package + '.' +
name
                    stack.append((fn, '', new_package,
False))
                else:
                    stack.append((fn, prefix + name +
'/', package, only_in_packages))
            elif package or not only_in_packages:
                # is a file
                bad_name = False
                for pattern in exclude:
                    if (fnmatchcase(name, pattern)
                        or fn.lower() ==
pattern.lower()):
                        bad_name = True
                        if show_ignored:
                            print >> sys.stderr, (
                                "File %s ignored by
pattern %s"
                                % (fn, pattern))
                            pass
                        break
                if bad_name:
                    continue
                out.setdefault(package,
[]).append(prefix+name)
    return out





 
____________________________________________________________________________________
Need Mail bonding?
Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users.
http://answers.yahoo.com/dir/?link=list&sid=396546091

From venkatbo at yahoo.com  Sat Mar 31 04:09:08 2007
From: venkatbo at yahoo.com (Venkat Bommakanti)
Date: Fri, 30 Mar 2007 19:09:08 -0700 (PDT)
Subject: [Distutils] Exclusionary list of files/dirs in my TG-app's setup.py
Message-ID: <20070331020908.22368.qmail@web30508.mail.mud.yahoo.com>

Hi folks,

(Please ignore my previous req on the same topic
 Got sent out by mistake... Thanks.)

'am using setup tools to build and package a TG 1.0.1
app. After deployment, I'm seeing a few entries in
SOURCES.txt despite specifying related wildcards in
the exclusion list.

Following are some related snippets:

setup.py:----------------
...
#
#from turbogears.finddata import find_package_data
#
# Disabled above import line, and added
find_package_data
# below to avoid installing TG on the build box...
#
import sys
from fnmatch import fnmatchcase
from distutils.util import convert_path

# Provided as an attribute, so you can append to these
instead
# of replicating them:
standard_exclude = ('*.py', '*.pyc', '*~', '.*',
'*.bak',
                    'checkout.status',
'original.status')
standard_exclude_directories = ('.*', 'CVS', 'CC',
'_darcs', './
build',
                                './dist', 'EGG-INFO',
'*.egg-info',
                                '.CC', 'cache',
'*.egg-info/.CC',
                                '*.egg-info/.CC',
'*.egg-info/.CC/*')

def find_package_data(
    where='.', package='',
    exclude=standard_exclude,
    exclude_directories=standard_exclude_directories,
    only_in_packages=True,
    show_ignored=False):
...

setup(
    name="myapp",
    version=version,
...

--------------------------------------

However, after successful build/deply, I'm seeing the
following entries in the deployed SOURCES.txt file
(myapp/myapp.egg-info/SOURCES.txt):
...
myapp.egg-info/.CC/checkout.status
myapp.egg-info/.CC/original.status
myapp.egg-info/.CC/cache/SOURCES.txt@@/main/1
...

---------------------------------------

It appears the exlusion list applies to all app
folders excpet myapp.egg-info/. Any idea how I can
specify that even in the case of myapp.egg-info/.CC/
it should include any diles or dirs...

Thanks much,
/venkat 

Full find_package_data:
-----------------------


def find_package_data(
    where='.', package='',
    exclude=standard_exclude,
    exclude_directories=standard_exclude_directories,
    only_in_packages=True,
    show_ignored=True):
    """
    Return a dictionary suitable for use in
``package_data``
    in a distutils ``setup.py`` file.

    The dictionary looks like::

        {'package': [files]}

    Where ``files`` is a list of all the files in that
package that
    don't match anything in ``exclude``.

    If ``only_in_packages`` is true, then top-level
directories that
    are not packages won't be included (but
directories under packages
    will).

    Directories matching any pattern in
``exclude_directories`` will
    be ignored; by default directories with leading
``.``, ``CVS``,
    and ``_darcs`` will be ignored.

    If ``show_ignored`` is true, then all the files
that aren't
    included in package data are shown on stderr (for
debugging
    purposes).

    Note patterns use wildcards, or can be exact paths
(including
    leading ``./``), and all searching is
case-insensitive.
    """

    out = {}
    stack = [(convert_path(where), '', package,
only_in_packages)]
    while stack:
        where, prefix, package, only_in_packages =
stack.pop(0)
        for name in os.listdir(where):
            fn = os.path.join(where, name)
            if os.path.isdir(fn):
                bad_name = False
                for pattern in exclude_directories:
                    if (fnmatchcase(name, pattern)
                        or fn.lower() ==
pattern.lower()):
                        bad_name = True
                        if show_ignored:
                            print >> sys.stderr, (
                                "Directory %s ignored
by pattern %s"
                                % (fn, pattern))
                            pass
                        break
                if bad_name:
                    continue
                if os.path.isfile(os.path.join(fn,
'__init__.py')):
                    if not package:
                        new_package = name
                    else:
                        new_package = package + '.' +
name
                    stack.append((fn, '', new_package,
False))
                else:
                    stack.append((fn, prefix + name +
'/', package, only_in_packages))
            elif package or not only_in_packages:
                # is a file
                bad_name = False
                for pattern in exclude:
                    if (fnmatchcase(name, pattern)
                        or fn.lower() ==
pattern.lower()):
                        bad_name = True
                        if show_ignored:
                            print >> sys.stderr, (
                                "File %s ignored by
pattern %s"
                                % (fn, pattern))
                            pass
                        break
                if bad_name:
                    continue
                out.setdefault(package,
[]).append(prefix+name)
    return out








 
____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/

From pje at telecommunity.com  Sat Mar 31 04:57:58 2007
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 30 Mar 2007 21:57:58 -0500
Subject: [Distutils] Exclusionary list of files/dirs in my TG-app's
 setup.py
In-Reply-To: <20070331020908.22368.qmail@web30508.mail.mud.yahoo.com>
Message-ID: <5.1.1.6.0.20070330215051.040fc2e0@sparrow.telecommunity.com>

At 07:09 PM 3/30/2007 -0700, Venkat Bommakanti wrote:
>Hi folks,
>
>(Please ignore my previous req on the same topic
>  Got sent out by mistake... Thanks.)
>
>'am using setup tools to build and package a TG 1.0.1
>app. After deployment, I'm seeing a few entries in
>SOURCES.txt despite specifying related wildcards in
>the exclusion list.


May I suggest that instead of using a custom find_package_data function, 
that you simply set "include_package_data=True" and add any files you want 
to include to your version control system (CVS or SVN)?  Or, if you aren't 
using a supported version control system (and don't want to write a plugin 
for it), use the MANIFEST.in file to select what files to include or 
exclude.  See e.g.:

http://python.org/doc/2.3/dist/source-dist.html#SECTION000510000000000000000

When you simply use "include_package_data=True" instead of specifying 
individual files or wildcards, setuptools begins by using the distutils 
defaults, plus anything listed in revision control that's inside a package 
directory and isn't a .py* file.  Then, it applies the rules listed in 
MANIFEST.in, which means that you can use it to add things that aren't 
under revision control, OR to exclude things that *are* in revision control.

Anyway, this will give you fine-grained control over what you want to 
include in your source and egg distributions, without needing to cram a 
bunch of extras into your setup script.  As far as I know, TG's 
find_package_data function was created before setuptools supported the 
include_package_data option.  I don't think it's been necessary for quite a 
while now.