From g.brandl at gmx.net  Sun Nov  1 09:31:46 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 01 Nov 2009 09:31:46 +0100
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <4AEB67B2.2020605@simplistix.co.uk>
References: <ce40232aa74412e3bb46c8c7b17d42ab@preisshare.net>	<4ADDB2CB.1090101@simplistix.co.uk>	<8e9d1c135f56a0b37f12afe144b30f1a@preisshare.net>	<4ADEB716.5020605@simplistix.co.uk>	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>	<4AE72554.6060108@simplistix.co.uk>	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>	<4AE853FE.5070602@simplistix.co.uk>	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>
	<4AEB67B2.2020605@simplistix.co.uk>
Message-ID: <hcjdmd$6ug$1@ger.gmane.org>

Chris Withers schrieb:
> Fred Drake wrote:
>> On Wed, Oct 28, 2009 at 10:23 AM, Chris Withers <chris at simplistix.co.uk> wrote:
>>> setup()
>>>
>>> ...would get everyone from the static metadata file, while any keyword
>>> parameters to setup would override the appropriate setting from the static
>>> metadata file...
>> 
>> I'd rather see the static data in setup.cfg and no setup.py if there's
>> no need for custom Python code.  
> 
> I think "python setup.py install" is so idiomatic that it seems silly to 
> break it for the sake of two lines of python. There's also an element of 
> "explicit is better than implicit" in the feel of actually running 
> setup.py...

Has anyone already suggested something like "python -m setup install"?
It would be rather similar and explicit, too.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From david.lyon at preisshare.net  Sun Nov  1 10:45:29 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 01 Nov 2009 04:45:29 -0500
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <hcjdmd$6ug$1@ger.gmane.org>
References: <ce40232aa74412e3bb46c8c7b17d42ab@preisshare.net>	<4ADDB2CB.1090101@simplistix.co.uk>	<8e9d1c135f56a0b37f12afe144b30f1a@preisshare.net>	<4ADEB716.5020605@simplistix.co.uk>	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>	<4AE72554.6060108@simplistix.co.uk>	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>	<4AE853FE.5070602@simplistix.co.uk>	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>
	<4AEB67B2.2020605@simplistix.co.uk> <hcjdmd$6ug$1@ger.gmane.org>
Message-ID: <9674904ac9e10b40b2045b24e78d987e@preisshare.net>

On Sun, 01 Nov 2009 09:31:46 +0100, Georg Brandl <g.brandl at gmx.net> wrote:

>> I think "python setup.py install" is so idiomatic that it seems silly to

>> break it for the sake of two lines of python. There's also an element of

>> "explicit is better than implicit" in the feel of actually running 
>> setup.py...
> 
> Has anyone already suggested something like "python -m setup install"?
> It would be rather similar and explicit, too.

Not as far as I know.

David

From setuptools at bugs.python.org  Sun Nov  1 12:00:42 2009
From: setuptools at bugs.python.org (Max Bowsher)
Date: Sun, 01 Nov 2009 11:00:42 +0000
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting	tarball source distributions
In-Reply-To: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
Message-ID: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>


New submission from Max Bowsher <maxb at f2s.com>:

Trying to determine why pytz installed by easy_install was broken, I located the
following problem:

When setuptools extracts a tarball source dist to install it, it silently drops
any tar members which are neither files or directories. The problem is in
archive_util.py which specifically tests "if member.isfile() or
member.isdir():". I am uncertain why it would try to do this, but it is fatally
incorrect to do so when the software being unpacked includes symlinks as a
functional part of its sourcecode, as pytz does.

----------
messages: 444
nosy: maxb
priority: bug
status: unread
title: easy_install silently drop symlinks when auto-extracting tarball source distributions

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue89>
_______________________________________________

From floris.bruynooghe at gmail.com  Sun Nov  1 12:26:43 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sun, 1 Nov 2009 11:26:43 +0000
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <hcjdmd$6ug$1@ger.gmane.org>
References: <4ADEB716.5020605@simplistix.co.uk>
	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>
	<4AE72554.6060108@simplistix.co.uk>
	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>
	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>
	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>
	<4AE853FE.5070602@simplistix.co.uk>
	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>
	<4AEB67B2.2020605@simplistix.co.uk> <hcjdmd$6ug$1@ger.gmane.org>
Message-ID: <20091101112643.GA25904@laurie.devork>

On Sun, Nov 01, 2009 at 09:31:46AM +0100, Georg Brandl wrote:
> Chris Withers schrieb:
> > Fred Drake wrote:
> >> On Wed, Oct 28, 2009 at 10:23 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> >>> setup()
> >>>
> >>> ...would get everyone from the static metadata file, while any keyword
> >>> parameters to setup would override the appropriate setting from the static
> >>> metadata file...
> >> 
> >> I'd rather see the static data in setup.cfg and no setup.py if there's
> >> no need for custom Python code.  
> > 
> > I think "python setup.py install" is so idiomatic that it seems silly to 
> > break it for the sake of two lines of python. There's also an element of 
> > "explicit is better than implicit" in the feel of actually running 
> > setup.py...
> 
> Has anyone already suggested something like "python -m setup install"?
> It would be rather similar and explicit, too.

I'm sure that approach was mentioned in one of Tarek's very early
proposals.  IIRC it went away because people didn't like the idea of a
built-in installer/uninstaller in distutils, but I'm not sure what the
arguments where.

Personally I have no objection against it, if it would means you can
omit setup.py if it's only going to call 'setup()' then why not.


Floris

From ziade.tarek at gmail.com  Sun Nov  1 13:42:16 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 1 Nov 2009 13:42:16 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
Message-ID: <94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>

Hello Martin,

The Distutils-SIG mailing list is hosting several projects (4) which
all have their own trackers, and for some their own checking/bugs
mailing lists that receive these kind of automatic emails. This makes
it possible for the people that want to follow those warning e-mails
to register to those lists specifically. And for the others that are
registered here to get less emails.

Could you deactivate these automatic warning mails please?

Thanks

---------- Forwarded message ----------
From: Max Bowsher <setuptools at bugs.python.org>
Date: Sun, Nov 1, 2009 at 12:00 PM
Subject: [Distutils] [issue89] easy_install silently drop symlinks
when auto-extracting	tarball source distributions
To: Distutils-SIG at python.org



New submission from Max Bowsher <maxb at f2s.com>:

Trying to determine why pytz installed by easy_install was broken, I located the
following problem:

When setuptools extracts a tarball source dist to install it, it silently drops
any tar members which are neither files or directories. The problem is in
archive_util.py which specifically tests "if member.isfile() or
member.isdir():". I am uncertain why it would try to do this, but it is fatally
incorrect to do so when the software being unpacked includes symlinks as a
functional part of its sourcecode, as pytz does.

----------
messages: 444
nosy: maxb
priority: bug
status: unread
title: easy_install silently drop symlinks when auto-extracting
tarball source distributions

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue89>
_______________________________________________
_______________________________________________
Distutils-SIG maillist ?- ?Distutils-SIG at python.org
http://mail.python.org/mailman/listinfo/distutils-sig



-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From ziade.tarek at gmail.com  Sun Nov  1 14:09:09 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 1 Nov 2009 14:09:09 +0100
Subject: [Distutils] Distribute 0.6.7 released
Message-ID: <94bdd2610911010509h2bf68455q402bedbbebbefdcc@mail.gmail.com>

On behalf of the Distribute team, I am pleased to announce the 0.6.7
release of Distribute.

As usual, availabe at PyPI: http://pypi.python.org/pypi/distribute

Most noticeable changes in 0.6.7 are:

- now the develop command supports the --user option, so it can use
the per-user site packages (PEP 370)
- the generated scripts now wrap their call to the script entry point
in the standard "if name == 'main'"
- better errors handling in PackageIndex when files and pages are
visited by easy_install
- a virtualenv-compatible version, so the next virtualenv release will
be able to provide a --distribute option.

You can visit http://pypi.python.org/pypi/distribute#id2 for a full
CHANGES list.

We are now starting the 0.6.8 work in parallel of 0.7.x development,
with more bugfixes coming up.

Regards
Tarek

From g.brandl at gmx.net  Sun Nov  1 17:33:24 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 01 Nov 2009 17:33:24 +0100
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <20091101112643.GA25904@laurie.devork>
References: <4ADEB716.5020605@simplistix.co.uk>	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>	<4AE72554.6060108@simplistix.co.uk>	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>	<4AE853FE.5070602@simplistix.co.uk>	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>	<4AEB67B2.2020605@simplistix.co.uk>
	<hcjdmd$6ug$1@ger.gmane.org> <20091101112643.GA25904@laurie.devork>
Message-ID: <hck9tg$c3p$1@ger.gmane.org>

Floris Bruynooghe schrieb:
> On Sun, Nov 01, 2009 at 09:31:46AM +0100, Georg Brandl wrote:
>> Chris Withers schrieb:
>> > Fred Drake wrote:
>> >> On Wed, Oct 28, 2009 at 10:23 AM, Chris Withers <chris at simplistix.co.uk> wrote:
>> >>> setup()
>> >>>
>> >>> ...would get everyone from the static metadata file, while any keyword
>> >>> parameters to setup would override the appropriate setting from the static
>> >>> metadata file...
>> >> 
>> >> I'd rather see the static data in setup.cfg and no setup.py if there's
>> >> no need for custom Python code.  
>> > 
>> > I think "python setup.py install" is so idiomatic that it seems silly to 
>> > break it for the sake of two lines of python. There's also an element of 
>> > "explicit is better than implicit" in the feel of actually running 
>> > setup.py...
>> 
>> Has anyone already suggested something like "python -m setup install"?
>> It would be rather similar and explicit, too.
> 
> I'm sure that approach was mentioned in one of Tarek's very early
> proposals.  IIRC it went away because people didn't like the idea of a
> built-in installer/uninstaller in distutils, but I'm not sure what the
> arguments where.

It's not like it would contain an installer or uninstaller, just the
equivalent of "import setup; setup()" or distributions that don't need
a customized setup.py.

> Personally I have no objection against it, if it would means you can
> omit setup.py if it's only going to call 'setup()' then why not.

Exactly.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Sun Nov  1 21:00:58 2009
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 01 Nov 2009 21:00:58 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
 auto-extracting tarball source distributions
In-Reply-To: <94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	
	<1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
Message-ID: <4AEDE8FA.6060602@v.loewis.de>

> The Distutils-SIG mailing list is hosting several projects (4) which
> all have their own trackers, and for some their own checking/bugs
> mailing lists that receive these kind of automatic emails. This makes
> it possible for the people that want to follow those warning e-mails
> to register to those lists specifically. And for the others that are
> registered here to get less emails.
> 
> Could you deactivate these automatic warning mails please?

Hi Tarek,

I have no real clue what you are talking about. The message you've
copied is a roundup message from the setuptools tracker, not a warning
message of some kind. What specific warning messages are you referring
to, who is sending them, and why do you think it is within my power
to make them stop?

Regards,
Martin

From ziade.tarek at gmail.com  Sun Nov  1 22:51:41 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 1 Nov 2009 22:51:41 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <4AEDE8FA.6060602@v.loewis.de>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
Message-ID: <94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>

2009/11/1 "Martin v. L?wis" <martin at v.loewis.de>:
>> The Distutils-SIG mailing list is hosting several projects (4) which
>> all have their own trackers, and for some their own checking/bugs
>> mailing lists that receive these kind of automatic emails. This makes
>> it possible for the people that want to follow those warning e-mails
>> to register to those lists specifically. And for the others that are
>> registered here to get less emails.
>>
>> Could you deactivate these automatic warning mails please?
>
> Hi Tarek,
>
> I have no real clue what you are talking about. The message you've
> copied is a roundup message from the setuptools tracker, not a warning
> message of some kind.

Yes, that's a warning from the setuptools tracker. I call that a
"warning" because
it warns you that someone added an issue in the tracker.

> What specific warning messages are you referring
> to, who is sending them, and why do you think it is within my power
> to make them stop?

Because you are the one who set it up some time ago as far as this message says:

http://markmail.org/message/kk5jnugeqnr53iuy

Regards
Tarek

From martin at v.loewis.de  Mon Nov  2 00:01:34 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 02 Nov 2009 00:01:34 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
 auto-extracting tarball source distributions
In-Reply-To: <94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
Message-ID: <4AEE134E.9040201@v.loewis.de>

>> I have no real clue what you are talking about. The message you've
>> copied is a roundup message from the setuptools tracker, not a warning
>> message of some kind.
> 
> Yes, that's a warning from the setuptools tracker. I call that a
> "warning" because
> it warns you that someone added an issue in the tracker.

Ah, ok. That meaning of "warning" was unknown to me, sorry.

So you want the roundup tracker to send its notifications elsewhere.
Where exactly?

>> What specific warning messages are you referring
>> to, who is sending them, and why do you think it is within my power
>> to make them stop?
> 
> Because you are the one who set it up some time ago as far as this message says:
> 
> http://markmail.org/message/kk5jnugeqnr53iuy

It's actually Jeff Rush who is in charge of this tracker instance.
I can't change the setup without him agreeing.

So, Jeff, where else should the roundup instance for setuptools send
its notifications?

Regards,
Martin

From ziade.tarek at gmail.com  Mon Nov  2 00:08:31 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 2 Nov 2009 00:08:31 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <4AEE134E.9040201@v.loewis.de>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
Message-ID: <94bdd2610911011508u1051f981tf58b9d9bd9f25bee@mail.gmail.com>

2009/11/2 "Martin v. L?wis" <martin at v.loewis.de>:
[...]
>
> Ah, ok. That meaning of "warning" was unknown to me, sorry.

That's my translation, so it might be wrong :)

[..]
>
> It's actually Jeff Rush who is in charge of this tracker instance.
> I can't change the setup without him agreeing.
>
> So, Jeff, where else should the roundup instance for setuptools send
> its notifications?

I can create a dedicated google group if it can help, that's what we
did in another project.

Regards
Tarek

From pje at telecommunity.com  Mon Nov  2 01:06:42 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 01 Nov 2009 19:06:42 -0500
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
 auto-extracting tarball source distributions
In-Reply-To: <4AEE134E.9040201@v.loewis.de>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
Message-ID: <20091102000640.452F83A40A8@sparrow.telecommunity.com>

At 12:01 AM 11/2/2009 +0100, Martin v. L?wis wrote:
>It's actually Jeff Rush who is in charge of this tracker instance.
>I can't change the setup without him agreeing.
>
>So, Jeff, where else should the roundup instance for setuptools send
>its notifications?

In over 1 year, there are now a grand total of 89 such messages (at 
most) that have been sent to the list, vs. 5 messages just sent now 
(6 when you count this one) discussing whether or not they should 
continue to be sent here.

So far, there has also only been one complaint or request to change 
this, as far as I'm aware.  (Personally, I like them just fine where 
they are.  Among other things it makes it easy for someone to answer 
the ticket via email, which is one of the reasons I asked for it to 
be set up this way in the first place.)


From ben+python at benfinney.id.au  Mon Nov  2 01:28:14 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 02 Nov 2009 11:28:14 +1100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
	<94bdd2610911011508u1051f981tf58b9d9bd9f25bee@mail.gmail.com>
Message-ID: <87y6mplqht.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> 2009/11/2 "Martin v. L?wis" <martin at v.loewis.de>:
> [...]
> >
> > Ah, ok. That meaning of "warning" was unknown to me, sorry.
>
> That's my translation, so it might be wrong :)

?Warning? implies the system is complaining something is wrong. The
arrival of a new issue in the tracker doesn't qualify, I think. Better
would be ?notification?.

-- 
 \          ?Generally speaking, the errors in religion are dangerous; |
  `\    those in philosophy only ridiculous.? ?David Hume, _A Treatise |
_o__)                                           of Human Nature_, 1739 |
Ben Finney


From david.lyon at preisshare.net  Mon Nov  2 01:23:36 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 01 Nov 2009 19:23:36 -0500
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <20091101112643.GA25904@laurie.devork>
References: <4ADEB716.5020605@simplistix.co.uk>
	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>
	<4AE72554.6060108@simplistix.co.uk>
	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>
	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>
	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>
	<4AE853FE.5070602@simplistix.co.uk>
	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>
	<4AEB67B2.2020605@simplistix.co.uk> <hcjdmd$6ug$1@ger.gmane.org>
	<20091101112643.GA25904@laurie.devork>
Message-ID: <3f5ef4b807c37035b9ace60abbb27912@preisshare.net>

On Sun, 1 Nov 2009 11:26:43 +0000, Floris Bruynooghe
>> Has anyone already suggested something like "python -m setup install"?
>> It would be rather similar and explicit, too.

Actually, I never came across that command line option and had no idea
that it even existed. But this is normal, because I don't try to find
out every command line possibility or even try to remember many past
half a dozen.

When I run it, I even get something (which is interesting):

c:\python26\lib\distutils\dist.py:266: UserWarning: Unknown distribution
option:
 'console'
  warnings.warn(msg)
running install
running build
running install_egg_info
Writing c:\python26\Lib\site-packages\UNKNOWN-0.0.0-py2.6.egg-info
running install
running build
running install_egg_info
Removing c:\python26\Lib\site-packages\UNKNOWN-0.0.0-py2.6.egg-info
Writing c:\python26\Lib\site-packages\UNKNOWN-0.0.0-py2.6.egg-info

> I'm sure that approach was mentioned in one of Tarek's very early
> proposals.  IIRC it went away because people didn't like the idea of a
> built-in installer/uninstaller in distutils...

That would be normal. It's not always easy to get any new idea up and 
flying. Even for somebody with all the good skills that Tarek has.

> .. but I'm not sure what the arguments where.

Perphaps along the lines that it is too much work and too hard. And
all the people who could actually do it were off working on their
own external projects (ie setuptools, distribute, pip, etc). Let's 
just move on and not worry why not.

> Personally I have no objection against it, if it would means you can
> omit setup.py if it's only going to call 'setup()' then why not.

I will try to figure out how to include this mechanism in my
alternate metadata PEP submission...

After trying running that, it seems that it could be quite cool...

(Thanks for the tip - Georg)


David


From jeff at taupro.com  Mon Nov  2 02:12:43 2009
From: jeff at taupro.com (Jeff Rush)
Date: Sun, 01 Nov 2009 19:12:43 -0600
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
 auto-extracting tarball source distributions
In-Reply-To: <20091102000640.452F83A40A8@sparrow.telecommunity.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	<4AEDE8FA.6060602@v.loewis.de>	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>	<4AEE134E.9040201@v.loewis.de>
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
Message-ID: <4AEE320B.6030107@taupro.com>

P.J. Eby wrote:
> At 12:01 AM 11/2/2009 +0100, Martin v. L?wis wrote:
>> It's actually Jeff Rush who is in charge of this tracker instance.
>> I can't change the setup without him agreeing.
>>
>> So, Jeff, where else should the roundup instance for setuptools send
>> its notifications?
> 
> In over 1 year, there are now a grand total of 89 such messages (at
> most) that have been sent to the list, vs. 5 messages just sent now (6
> when you count this one) discussing whether or not they should continue
> to be sent here.
> 
> So far, there has also only been one complaint or request to change
> this, as far as I'm aware.  (Personally, I like them just fine where
> they are.  Among other things it makes it easy for someone to answer the
> ticket via email, which is one of the reasons I asked for it to be set
> up this way in the first place.)

I'm happy with the existing arrangement - I think sending low-traffic
bug reports to a separate list is a bad idea because few folk will
actually see them.  If the traffic level increases, then we can consider
moving notifications to a separate list hosted at python.org.  With our
own listserver I see no reason to use Google Groups though as there are
people who refuse to get an account there.

-Jeff


From chris at simplistix.co.uk  Mon Nov  2 08:12:30 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 02 Nov 2009 07:12:30 +0000
Subject: [Distutils] setuptools tracker notifications
In-Reply-To: <4AEE320B.6030107@taupro.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	<4AEDE8FA.6060602@v.loewis.de>	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>	<4AEE134E.9040201@v.loewis.de>	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com>
Message-ID: <4AEE865E.7090900@simplistix.co.uk>

Jeff Rush wrote:
> I'm happy with the existing arrangement - I think sending low-traffic
> bug reports to a separate list is a bad idea because few folk will
> actually see them. 

Agreed.

Any chance the mail subject could be prefixed with [setuptools] to make 
it clear which tracker it came from?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From martin at v.loewis.de  Mon Nov  2 08:51:15 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 02 Nov 2009 08:51:15 +0100
Subject: [Distutils] setuptools tracker notifications
In-Reply-To: <4AEE865E.7090900@simplistix.co.uk>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	<4AEDE8FA.6060602@v.loewis.de>	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>	<4AEE134E.9040201@v.loewis.de>	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com> <4AEE865E.7090900@simplistix.co.uk>
Message-ID: <4AEE8F73.9080200@v.loewis.de>

> Any chance the mail subject could be prefixed with [setuptools] to make
> it clear which tracker it came from?

If you want to filter them out (or in a separate folder), I recommend to
set your filter for the X-Roundup-Name header.

Adding something to the subject is tricky, IIRC.

Regards,
Martin

From ziade.tarek at gmail.com  Mon Nov  2 11:24:52 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 2 Nov 2009 11:24:52 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <4AEE320B.6030107@taupro.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com>
Message-ID: <94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>

>> In over 1 year, there are now a grand total of 89 such messages (at
>> most) that have been sent to the list, vs. 5 messages just sent now (6
>> when you count this one) discussing whether or not they should continue
>> to be sent here.

Yes, exactly. Because those are notifications, and have nothing to do on
a human mailing list.

[...]
>>
>> So far, there has also only been one complaint or request to change
>> this, as far as I'm aware.
>?(Personally, I like them just fine where
>> they are. ?Among other things it makes it easy for someone to answer the
>> ticket via email, which is one of the reasons I asked for it to be set
>> up this way in the first place.)

I don't remember you asking in Distutils-SIG to the people if they
were OK with that.

Anyways, that's not the way an issue tracker works. If you want to
answer to issues, you do it in the
tracker itself. That's what you have been doing most of the time IIRC.

You can't follow a discussion on an issue otherwise.

For example, in the Python issue tracker, there's a dedicated ist for
that , but it's just notifications,
and you answer afterwards in the tracker to keep track of it.

>
> I'm happy with the existing arrangement - I think sending low-traffic
> bug reports to a separate list is a bad idea because few folk will
> actually see them. ?If the traffic level increases, then we can consider
> moving notifications to a separate list hosted at python.org. ?With our
> own listserver I see no reason to use Google Groups though as there are
> people who refuse to get an account there.

Google group was just a suggestion, a python.org list is fine of course.

But I am wondering : who are those "folk" ? Receiving those
notifications is good when you are the *maintainer*
of the project. Setuptools has one maintainer (Philip), So why not sending these
notifications to his personal email ? I don't think there a lot of
folks that are interested in folllowing
those notifications in here in the first place. And other folks can
registered to this dedicated list.

I don't see any good reason why Setuptools would push notifications
like that in the Distutils-SIG.
(zc.buildout or Distribute don't do that) . Plus, with headers like
"[issue #123] bug"  it looks that this is the Distutils project (we
are on the *Distutils*-SIG).

If the subject can be changed with the word "setuptools" in it, I'll
accomodate with it.

Tarek

From martin at v.loewis.de  Mon Nov  2 11:40:58 2009
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 02 Nov 2009 11:40:58 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
 auto-extracting tarball source distributions
In-Reply-To: <94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	
	<4AEDE8FA.6060602@v.loewis.de>	
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>	
	<4AEE134E.9040201@v.loewis.de>	
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>	
	<4AEE320B.6030107@taupro.com>
	<94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
Message-ID: <4AEEB73A.3050102@v.loewis.de>

> Anyways, that's not the way an issue tracker works. If you want to
> answer to issues, you do it in the
> tracker itself. That's what you have been doing most of the time IIRC.

You may be missing one of the primary strengths of roundup: the ability
to carry out bug discussions (nearly) entirely by email.

> You can't follow a discussion on an issue otherwise.

Sure you can. As soon as you respond by email, you get added to the
nosy list of the issue, which means you'll also get all follow-ups.

> For example, in the Python issue tracker, there's a dedicated ist for
> that , but it's just notifications,
> and you answer afterwards in the tracker to keep track of it.

No no no. Some people use the web interface, some use the email
interface. I'd have to perform exact counting to find out which
one is the majority, however, both are used significantly (my guess
is that committers prefer email; bug submitters use the web interface
because they don't expect email responses to work).

> If the subject can be changed with the word "setuptools" in it, I'll
> accomodate with it.

Did you ask for that specifically because I said it would be difficult
to provide?

Regards,
Martin

From p.f.moore at gmail.com  Mon Nov  2 11:50:55 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 2 Nov 2009 10:50:55 +0000
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <79990c6b0911020250p776079fby39391e023878b8f3@mail.gmail.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com>
	<94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
	<79990c6b0911020250p776079fby39391e023878b8f3@mail.gmail.com>
Message-ID: <79990c6b0911020250i4fe97acka8bffc447b93571a@mail.gmail.com>

Sorry, that was meant to go to the list, not as a personal comment to Tarek.

2009/11/2 Paul Moore <p.f.moore at gmail.com>:
> 2009/11/2 Tarek Ziad? <ziade.tarek at gmail.com>:
> (original quote here from PJE)
>>>>?(Personally, I like them just fine where
>>>> they are. ?Among other things it makes it easy for someone to answer the
>>>> ticket via email, which is one of the reasons I asked for it to be set
>>>> up this way in the first place.)
>>
>> I don't remember you asking in Distutils-SIG to the people if they
>> were OK with that.
>
> Can we please avoid this flaring up into another Tarek vs PJE fight?
>
> FWIW, I'm mildly annoyed by setuptools tracker emails appearing here,
> but I live with it. I suggest others do the same.
>
> Paul.
>

From ziade.tarek at gmail.com  Mon Nov  2 11:57:03 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 2 Nov 2009 11:57:03 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <4AEEB73A.3050102@v.loewis.de>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com>
	<94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
	<4AEEB73A.3050102@v.loewis.de>
Message-ID: <94bdd2610911020257g19f2bd77pc963c5784a205a19@mail.gmail.com>

2009/11/2 "Martin v. L?wis" <martin at v.loewis.de>:
>> Anyways, that's not the way an issue tracker works. If you want to
>> answer to issues, you do it in the
>> tracker itself. That's what you have been doing most of the time IIRC.
>
> You may be missing one of the primary strengths of roundup: the ability
> to carry out bug discussions (nearly) entirely by email.
>
>> You can't follow a discussion on an issue otherwise.
>
> Sure you can. As soon as you respond by email, you get added to the
> nosy list of the issue, which means you'll also get all follow-ups.
>
>> For example, in the Python issue tracker, there's a dedicated ist for
>> that , but it's just notifications,
>> and you answer afterwards in the tracker to keep track of it.
>
> No no no. Some people use the web interface, some use the email
> interface. I'd have to perform exact counting to find out which
> one is the majority, however, both are used significantly (my guess
> is that committers prefer email; bug submitters use the web interface
> because they don't expect email responses to work).
>

This feature is great, I didn't know about it, but you don't put these
emails in a -SIG list as
far as I know, do you ? You don't push email at python-dev either.

But what is the rationale to have a python.org hosted tracker with
emails sent in this list here ?

Does it means any third party project can get such a tracker at
Python.org and push its mails in a -SIG list ?


>> If the subject can be changed with the word "setuptools" in it, I'll
>> accomodate with it.
>
> Did you ask for that specifically because I said it would be difficult
> to provide?

Uh ? not at all.

Look in your emails, and look in the first private mail I have sent you
a few days ago, when the last issue came up (you didn't answer then).
I said in that mail that is was hard to follow and make a distinction
between the different projects because it only has a [distutils]
headers.

And I have I said it again here.

From ziade.tarek at gmail.com  Mon Nov  2 12:13:20 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 2 Nov 2009 12:13:20 +0100
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <79990c6b0911020250i4fe97acka8bffc447b93571a@mail.gmail.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com>
	<94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
	<79990c6b0911020250p776079fby39391e023878b8f3@mail.gmail.com>
	<79990c6b0911020250i4fe97acka8bffc447b93571a@mail.gmail.com>
Message-ID: <94bdd2610911020313yc7c3a8j6214ee22af5bdc4e@mail.gmail.com>

2009/11/2 Paul Moore <p.f.moore at gmail.com>:
> Sorry, that was meant to go to the list, not as a personal comment to Tarek.

No problem.

>> Can we please avoid this flaring up into another Tarek vs PJE fight?
>>
>> FWIW, I'm mildly annoyed by setuptools tracker emails appearing here,
>> but I live with it. I suggest others do the same.

I don't want to get into a fight (but I am getting used at receiving
personal attacks and try not flame things) ,
but these notifications, among other things, makes it unclear that the
Setuptools project is a distinct project from Distutils.

For example, I am sometimes receiving mails from people complaining
that easy_install doesn't work right
and that Distutils should be fixed.

It's great that Distutils-SIG host several projects, especially since
many people want to have a unified
project for the packaging topic.

But these notifications have to be straightened up, and I don't see
any rationale yet why they should stay in this list.

Tarek

From ndbecker2 at gmail.com  Mon Nov  2 13:14:58 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 02 Nov 2009 07:14:58 -0500
Subject: [Distutils] option to force build from source
Message-ID: <hcmig2$653$1@ger.gmane.org>

I just did
easy_install blist
on fedora F11 x86_64.  I was surprised that it did not build from source, 
instead d/l somebody's blist.so.

No surprise, testing 'import blist' fails with an undefined symbol.

This is distribute 0.6.7.  Please add option to force build from source.


From floris.bruynooghe at gmail.com  Mon Nov  2 13:15:52 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Mon, 2 Nov 2009 12:15:52 +0000
Subject: [Distutils] Setuptools roundup emails (was: Re: [issue89]
 easy_install silently drop symlinks when auto-extracting tarball source
 distributions)
In-Reply-To: <79990c6b0911020250i4fe97acka8bffc447b93571a@mail.gmail.com>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>
	<4AEDE8FA.6060602@v.loewis.de>
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>
	<4AEE134E.9040201@v.loewis.de>
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com>
	<94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
	<79990c6b0911020250p776079fby39391e023878b8f3@mail.gmail.com>
	<79990c6b0911020250i4fe97acka8bffc447b93571a@mail.gmail.com>
Message-ID: <20091102121552.GA16528@laurie.devork>

On Mon, Nov 02, 2009 at 10:50:55AM +0000, Paul Moore wrote:
> Sorry, that was meant to go to the list, not as a personal comment to Tarek.
> 
> 2009/11/2 Paul Moore <p.f.moore at gmail.com>:
> > 2009/11/2 Tarek Ziad? <ziade.tarek at gmail.com>:
> > (original quote here from PJE)
> >>>>?(Personally, I like them just fine where
> >>>> they are. ?Among other things it makes it easy for someone to answer the
> >>>> ticket via email, which is one of the reasons I asked for it to be set
> >>>> up this way in the first place.)
> >>
> >> I don't remember you asking in Distutils-SIG to the people if they
> >> were OK with that.
> >
> > Can we please avoid this flaring up into another Tarek vs PJE fight?
> >
> > FWIW, I'm mildly annoyed by setuptools tracker emails appearing here,
> > but I live with it. I suggest others do the same.

FWIW I'd be +1 on moving the setuptool tracker emails elsewhere.  I
just always assumed there was some secret cabal agreement on them
being here, but apparently not.  ;-)

OTOH I'm pretty used to ignoring threads that don't interest me.  :-)


Regards
Floris

From ziade.tarek at gmail.com  Mon Nov  2 13:32:19 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 2 Nov 2009 13:32:19 +0100
Subject: [Distutils] option to force build from source
In-Reply-To: <hcmig2$653$1@ger.gmane.org>
References: <hcmig2$653$1@ger.gmane.org>
Message-ID: <94bdd2610911020432g4f717b03n74e02bcd436eb1e1@mail.gmail.com>

On Mon, Nov 2, 2009 at 1:14 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> I just did
> easy_install blist
> on fedora F11 x86_64. ?I was surprised that it did not build from source,
> instead d/l somebody's blist.so.
>
> No surprise, testing 'import blist' fails with an undefined symbol.
>
> This is distribute 0.6.7. ?Please add option to force build from source.

I have added an issue here for follow up:
http://bitbucket.org/tarek/distribute/issue/82

Tarek

From ssteinerx at gmail.com  Mon Nov  2 15:32:41 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 2 Nov 2009 09:32:41 -0500
Subject: [Distutils] [issue89] easy_install silently drop symlinks when
	auto-extracting tarball source distributions
In-Reply-To: <4AEEB73A.3050102@v.loewis.de>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	
	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	
	<4AEDE8FA.6060602@v.loewis.de>	
	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>	
	<4AEE134E.9040201@v.loewis.de>	
	<20091102000640.452F83A40A8@sparrow.telecommunity.com>	
	<4AEE320B.6030107@taupro.com>
	<94bdd2610911020224o59c64074ic990152afeeee0aa@mail.gmail.com>
	<4AEEB73A.3050102@v.loewis.de>
Message-ID: <003E90AF-5944-4199-8B6B-A56AF364FD42@gmail.com>


On Nov 2, 2009, at 5:40 AM, Martin v. L?wis wrote:
>> If the subject can be changed with the word "setuptools" in it, I'll
>> accomodate with it.
>
> Did you ask for that specifically because I said it would be difficult
> to provide?

Uh, personal animosities aside, Tarek didn't ask for that, Chris did.

S


From g.brandl at gmx.net  Mon Nov  2 19:14:56 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 02 Nov 2009 19:14:56 +0100
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <3f5ef4b807c37035b9ace60abbb27912@preisshare.net>
References: <4ADEB716.5020605@simplistix.co.uk>	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>	<4AE72554.6060108@simplistix.co.uk>	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>	<4AE853FE.5070602@simplistix.co.uk>	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>	<4AEB67B2.2020605@simplistix.co.uk>
	<hcjdmd$6ug$1@ger.gmane.org>	<20091101112643.GA25904@laurie.devork>
	<3f5ef4b807c37035b9ace60abbb27912@preisshare.net>
Message-ID: <hcn7fk$jv0$1@ger.gmane.org>

David Lyon schrieb:
> On Sun, 1 Nov 2009 11:26:43 +0000, Floris Bruynooghe
>>> Has anyone already suggested something like "python -m setup install"?
>>> It would be rather similar and explicit, too.
> 
> Actually, I never came across that command line option and had no idea
> that it even existed. But this is normal, because I don't try to find
> out every command line possibility or even try to remember many past
> half a dozen.

Of course you wouldn't have to guess that, it would supersede "python setup.py"
in the respective manuals.

> When I run it, I even get something (which is interesting):

Not very.  You had a setup.py in the current directory.

The beauty of the "python -m setup" approach is that it will execute the
setup.py in the current directory if present, or a stdlib-supplied one
if not.

Georg


From tseaver at palladion.com  Mon Nov  2 20:32:24 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 02 Nov 2009 14:32:24 -0500
Subject: [Distutils] zc.buildout bootstrap errors
Message-ID: <hcnc48$84q$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Why would a "bootstrap" fail due to missing / less-than-required version
of setuptools in the host python?  The bootstrap should be
self-contained enough to make things work no matter what, since the
point is too build an environment which doesn't rely on anything beyond
a "bare" Python install.

 $ svn co $ZSVN/Acquisition/trunk Aq-trunk
 ...
 $ cd Aq-trun
 $ /path/to/python2.6 bootstrap/bootstrap.py
 The required version of setuptools (>=0.6c11) is not available, and
 can't be installed while this script is running. Please install
  a more recent version first, using 'easy_install -U setuptools'.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkrvM8gACgkQ+gerLs4ltQ5hagCfYX7m2w+kuVtebdN3Nc26mL7k
cpQAn1P5UmomD0j1q9/ulYuAZJI2Lpdw
=ly5E
-----END PGP SIGNATURE-----


From chris at simplistix.co.uk  Mon Nov  2 23:23:43 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 02 Nov 2009 22:23:43 +0000
Subject: [Distutils] zc.buildout bootstrap errors
In-Reply-To: <hcnc48$84q$1@ger.gmane.org>
References: <hcnc48$84q$1@ger.gmane.org>
Message-ID: <4AEF5BEF.4000406@simplistix.co.uk>

Tres Seaver wrote:
>  $ cd Aq-trun
>  $ /path/to/python2.6 bootstrap/bootstrap.py

Maybe try and earlier version of bootstrap?
I idley wonder if this is some side effect of the make-distribute-work work?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From fdrake at acm.org  Mon Nov  2 23:34:26 2009
From: fdrake at acm.org (Fred Drake)
Date: Mon, 2 Nov 2009 17:34:26 -0500
Subject: [Distutils] zc.buildout bootstrap errors
In-Reply-To: <4AEF5BEF.4000406@simplistix.co.uk>
References: <hcnc48$84q$1@ger.gmane.org> <4AEF5BEF.4000406@simplistix.co.uk>
Message-ID: <9cee7ab80911021434y50dddd86oebbefabbff6b5d8b@mail.gmail.com>

On Mon, Nov 2, 2009 at 5:23 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> Maybe try and earlier version of bootstrap?
> I idley wonder if this is some side effect of the make-distribute-work work?

This could happen before that; I saw it as soon as Philip started
releasing new versions of setuptools.

I suspect that it's due to buildout trying to use the latest
ez_setup.py or something like that with a fresh build, and that wants
to use the latest setuptools.

I've been using distribute ever since, and have been happy with it.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From jon at multani.info  Mon Nov  2 23:37:32 2009
From: jon at multani.info (Jonathan Ballet)
Date: Mon, 2 Nov 2009 23:37:32 +0100
Subject: [Distutils] zc.buildout bootstrap errors
In-Reply-To: <4AEF5BEF.4000406@simplistix.co.uk>
References: <hcnc48$84q$1@ger.gmane.org>
	<4AEF5BEF.4000406@simplistix.co.uk>
Message-ID: <20091102233732.238be09f@gourami>

Le Mon, 02 Nov 2009 22:23:43 +0000, Chris Withers <chris at simplistix.co.uk> a
?crit :

> Tres Seaver wrote:
> >  $ cd Aq-trun
> >  $ /path/to/python2.6 bootstrap/bootstrap.py
> 
> Maybe try and earlier version of bootstrap?
> I idley wonder if this is some side effect of the make-distribute-work work?
> 
> Chris

I'll not help, but I got the very same issue with a component ran
from a virtualenv-ed Python, which was created using setuptools 0.6c9 (and I did
this before Buildout was updated to run with Distribute (and, of course, after
the update of setuptools).

Finally, I gave up and updated my sandbox-ed version of setuptools installation,
but... oh well.


 Jonathan

From jim at zope.com  Mon Nov  2 23:45:24 2009
From: jim at zope.com (Jim Fulton)
Date: Mon, 2 Nov 2009 17:45:24 -0500
Subject: [Distutils] zc.buildout bootstrap errors
In-Reply-To: <hcnc48$84q$1@ger.gmane.org>
References: <hcnc48$84q$1@ger.gmane.org>
Message-ID: <1099b90b0911021445l175b97f7ke1fa2c1026efae1e@mail.gmail.com>

On Mon, Nov 2, 2009 at 2:32 PM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Why would a "bootstrap" fail due to missing / less-than-required version
> of setuptools in the host python? ?The bootstrap should be
> self-contained enough to make things work no matter what, since the
> point is too build an environment which doesn't rely on anything beyond
> a "bare" Python install.
>
> ?$ svn co $ZSVN/Acquisition/trunk Aq-trunk
> ?...
> ?$ cd Aq-trun
> ?$ /path/to/python2.6 bootstrap/bootstrap.py
> ?The required version of setuptools (>=0.6c11) is not available, and
> ?can't be installed while this script is running. Please install
> ?a more recent version first, using 'easy_install -U setuptools'.

This is probably due to an attempt to use an existing setuptools if
it's available.  I could probably add an option to get it to not do
that. I'll look into that.

Jim

-- 
Jim Fulton

From tseaver at palladion.com  Mon Nov  2 23:57:00 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 02 Nov 2009 17:57:00 -0500
Subject: [Distutils] zc.buildout bootstrap errors
In-Reply-To: <1099b90b0911021445l175b97f7ke1fa2c1026efae1e@mail.gmail.com>
References: <hcnc48$84q$1@ger.gmane.org>
	<1099b90b0911021445l175b97f7ke1fa2c1026efae1e@mail.gmail.com>
Message-ID: <4AEF63BC.1070901@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Fulton wrote:
> On Mon, Nov 2, 2009 at 2:32 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Why would a "bootstrap" fail due to missing / less-than-required version
>> of setuptools in the host python?  The bootstrap should be
>> self-contained enough to make things work no matter what, since the
>> point is too build an environment which doesn't rely on anything beyond
>> a "bare" Python install.
>>
>>  $ svn co $ZSVN/Acquisition/trunk Aq-trunk
>>  ...
>>  $ cd Aq-trun
>>  $ /path/to/python2.6 bootstrap/bootstrap.py
>>  The required version of setuptools (>=0.6c11) is not available, and
>>  can't be installed while this script is running. Please install
>>  a more recent version first, using 'easy_install -U setuptools'.
> 
> This is probably due to an attempt to use an existing setuptools if
> it's available.  I could probably add an option to get it to not do
> that. I'll look into that.

Great, thanks.  I can see re-using that version if it is present, and
new enough, but we ought to be able to use a downloaded version if that
fails.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkrvY7wACgkQ+gerLs4ltQ7HtwCfaVVBBg5Ti87BTgc7WjmdX19+
u7cAoICgsoFUVWS4rAdjhGQCp+zyfQ4e
=hKL9
-----END PGP SIGNATURE-----


From ziade.tarek at gmail.com  Tue Nov  3 00:11:34 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 3 Nov 2009 00:11:34 +0100
Subject: [Distutils] zc.buildout bootstrap errors
In-Reply-To: <4AEF5BEF.4000406@simplistix.co.uk>
References: <hcnc48$84q$1@ger.gmane.org> <4AEF5BEF.4000406@simplistix.co.uk>
Message-ID: <94bdd2610911021511j4f8cdb18wf42561461ad71088@mail.gmail.com>

On Mon, Nov 2, 2009 at 11:23 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> Tres Seaver wrote:
>>
>> ?$ cd Aq-trun
>> ?$ /path/to/python2.6 bootstrap/bootstrap.py
>
> Maybe try and earlier version of bootstrap?
> I idley wonder if this is some side effect of the make-distribute-work work?

No, I've checked for that, it's not related.

Tarek

From david.lyon at preisshare.net  Tue Nov  3 05:30:56 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 02 Nov 2009 23:30:56 -0500
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <hcn7fk$jv0$1@ger.gmane.org>
References: <4ADEB716.5020605@simplistix.co.uk>	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>	<4AE72554.6060108@simplistix.co.uk>	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>	<4AE853FE.5070602@simplistix.co.uk>	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>	<4AEB67B2.2020605@simplistix.co.uk>
	<hcjdmd$6ug$1@ger.gmane.org>	<20091101112643.GA25904@laurie.devork>
	<3f5ef4b807c37035b9ace60abbb27912@preisshare.net>
	<hcn7fk$jv0$1@ger.gmane.org>
Message-ID: <e5b2451d952b0ecf72a2e138a76463b4@preisshare.net>

On Mon, 02 Nov 2009 19:14:56 +0100, Georg Brandl <g.brandl at gmx.net> wrote:

> The beauty of the "python -m setup" approach is that it will execute the
> setup.py in the current directory if present, or a stdlib-supplied one
> if not.

Ok - I copied the file to \pythonX.Y\lib and ran 'python -m setup' and
got an instant result. The program showed some screen stuff and so I
could see that it was called.

That felt very easy.

Thank you very much for pointing this out.

Actually I find there are less steps with this method than with the 
traditional way of having to change directory into the package 
directory and then running setup.py from there. After that delete
the directory and so forth.

I will keep working on it....

once again - many thanks

David


 

From david.lyon at preisshare.net  Tue Nov  3 07:20:12 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 03 Nov 2009 01:20:12 -0500
Subject: [Distutils] Alternate static metadata PEP submission...
In-Reply-To: <9cee7ab80910301847s749aa574r892d4281f6130330@mail.gmail.com>
References: <ce40232aa74412e3bb46c8c7b17d42ab@preisshare.net>
	<4ADEB716.5020605@simplistix.co.uk>
	<4ADECB30.10605@ar.media.kyoto-u.ac.jp>
	<4AE72554.6060108@simplistix.co.uk>
	<9dcde5f5c75b1dcd6fbd361d5b259474@preisshare.net>
	<4AE7B9DB.8080100@ar.media.kyoto-u.ac.jp>
	<9cee7ab80910280538jc9d999dk97111787c48b1999@mail.gmail.com>
	<4AE853FE.5070602@simplistix.co.uk>
	<9cee7ab80910280816y1b20ca14q8c03ea7ba886ee25@mail.gmail.com>
	<4AEB67B2.2020605@simplistix.co.uk>
	<9cee7ab80910301847s749aa574r892d4281f6130330@mail.gmail.com>
Message-ID: <4f2ef6fb481354fc3ca4c9ed7f721a1d@preisshare.net>

> On Fri, Oct 30, 2009 at 6:24 PM, Chris Withers <chris at simplistix.co.uk>
> wrote:
>> I think "python setup.py install" is so idiomatic that it seems silly to
>> break it for the sake of two lines of python.

Ok. Well, I've been experimenting with "python -m setup install" and
I'm sure that I can get it to do the right thing.

"python -m setup install", thanks to George, is my new answer to life,
universe and packaging.

> This may well have been a reasonable choice at the time it was made
> over a decade ago (and yes, I was there, discussing distutils with
> Greg Ward during the birthing, so you can probably even lay some of
> the blame at my feet).  

Don't be so harsh on yourself. setup.py has served a lot of people
extremely well. It's only a minor change to go to "python -m setup 
install". It's pretty much a way of preserving the setup.py but
executing it in a more 'secure' manner.

> I worry more about having to review someone else's ad-hoc setup.py, 
> usually under deadline pressure, especially given how many variations 
> there are.

Doing the conversions is going to be the interesting part.

I'm already tossing around ways I can run a legacy 'python setup.py
install' and then compare it with a 'python -m setup install'

In most cases, you'd hope that they'd be the same.

Fortunately, there's not a shortage of packages on pypi that can
be used as test data.

Moving setup.py from the package into stdlib, isn't such a big
change. I'm sure many security type people would be comfortable
with this approach.

David







From ziade.tarek at gmail.com  Tue Nov  3 10:01:28 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 3 Nov 2009 10:01:28 +0100
Subject: [Distutils] Distribute sprint #2, next sunday
In-Reply-To: <94bdd2610910300541w7f772962rcf620d88140df8d3@mail.gmail.com>
References: <94bdd2610910300541w7f772962rcf620d88140df8d3@mail.gmail.com>
Message-ID: <94bdd2610911030101q5926e206u60721671ebaa8325@mail.gmail.com>

On Fri, Oct 30, 2009 at 1:41 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Hey,
>
> We will have a sprint sunday afternoon, starting at 3 pm in
> Berlin/Germany. If you are interested you can join online in the irc
> (#distutils on Freenode)
> or in Berlin of course.

A quick wrapup on this sprint. We were sprinting in Berlin with Jannis.

== virtualenv ==

We worked on the finalization of the --distribute option for
Distribute 0.6.x and this is now available in a test version Ian
released.

http://pypi.python.org/pypi/virtualenv/1.4rc1

(It's hidden on PyPI, so you should only get this version if you
specify it explicitly.)

== distribute ==

I've worked on "new" distutils commands for 0.7:

- install_egg_info : that's the "egg_info" command, renamed and
reworked. It now uses only distutils, and created a directory with the
PKG-INFO file. The next step is to add the pluggable part like 0.6
has, but with very a restricted execution environment unlike what we
have today. IOW: the plugin will not receive the command or the
distribution object so it can't hack on it.

- develop: I have continued the develop command work. It creates a
.pth file only, and does not create an egg-link file anymore. It just
contains a path to the place the distribution is located. The next
step is to list in the .pth file all the packages and modules paths
instead of one single path. I am not sure of the best way to do it
yet.

Cheers
Tarek

From jon at multani.info  Tue Nov  3 14:53:15 2009
From: jon at multani.info (Jonathan Ballet)
Date: Tue, 03 Nov 2009 07:53:15 -0600
Subject: [Distutils] Content of the .installed.cfg Buildout file
Message-ID: <dfb09feb1ebee58d53668e64c13ac801@mail.webfaction.com>

Hi,

I know we are not supposed to have a look at the content of the
.installed.cfg
file and that buildout managed this by itself, but I took a look at it
anyway.

I found something, which looks strange at the first glance, and I wondered
if it
can any problem (or if I should _really_ let buildout handle this...). At
the
end of the file, there are a bunch of lines like this::

[buildout]
installed_develop_eggs =
[buildout]
parts = database i18n test doc ...

(actually, more than thirty). Those which have "parts" have different those
parts in different order each one; there are some with
"installed_develop_eggs"
with a value and some with no value, despite there isn't any more developed
component in the develop option of our configuration file.

So, I wonder: is it normal or not?

Actually, I ask this question, since we had a strange behavior this
morning:
there were some components which where still used as developed components,
despite the fact that they have been removed from the buildout
configuration
file some hours ago (and some buildout execution latter) (there were still
links
lying in the develop-eggs/ directory), and I was a bit astonished to find
that
there were still values in .installed.cfg which referred to those developed
paths...


Cheers,

Jonathan

From setuptools at bugs.python.org  Wed Nov  4 19:11:31 2009
From: setuptools at bugs.python.org (Sridhar Ratnakumar)
Date: Wed, 04 Nov 2009 18:11:31 +0000
Subject: [Distutils] [issue90] AttributeError: 'NoneType' object has no
	attribute 'clone'
In-Reply-To: <1257358291.3.0.747371135918.issue90@psf.upfronthosting.co.za>
Message-ID: <1257358291.3.0.747371135918.issue90@psf.upfronthosting.co.za>


New submission from Sridhar Ratnakumar <sridharr at activestate.com>:

> easy_install TracAddHeadersPlugin
Searching for TracAddHeadersPlugin
Reading http://pypi.python.org/simple/TracAddHeadersPlugin/
Reading http://www.trac-hacks.org/wiki/AddHeadersPlugin
No local packages or download links found for TracAddHeadersPlugin
Best match: None
Traceback (most recent call last):
  File "/Users/sridharr/.local/bin/easy_install", line 8, in <module>
    load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')()
  File "/Users/sridharr/.local/lib/python2.6/site-
packages/setuptools/command/easy_install.py", line 1712, in main
    with_ei_usage(lambda:
  File "/Users/sridharr/.local/lib/python2.6/site-
packages/setuptools/command/easy_install.py", line 1700, in with_ei_usage  
    return f()
  File "/Users/sridharr/.local/lib/python2.6/site-
packages/setuptools/command/easy_install.py", line 1716, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File 
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.
py", line 152, in setup
    dist.run_commands()
  File 
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.
py", line 975, in run_commands
    self.run_command(cmd)
  File 
"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.
py", line 995, in run_command
    cmd_obj.run()
  File "/Users/sridharr/.local/lib/python2.6/site-
packages/setuptools/command/easy_install.py", line 211, in run
    self.easy_install(spec, not self.no_deps)
  File "/Users/sridharr/.local/lib/python2.6/site-
packages/setuptools/command/easy_install.py", line 434, in easy_install
    self.local_index
  File "/Users/sridharr/.local/lib/python2.6/site-
packages/setuptools/package_index.py", line 475, in fetch_distribution
    return dist.clone(location=self.download(dist.location, tmpdir))
AttributeError: 'NoneType' object has no attribute 'clone'

----------
messages: 445
nosy: srid
priority: bug
status: unread
title: AttributeError: 'NoneType' object has no attribute 'clone'

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue90>
_______________________________________________

From reinout at vanrees.org  Thu Nov  5 09:32:16 2009
From: reinout at vanrees.org (Reinout van Rees)
Date: Thu, 5 Nov 2009 08:32:16 +0000 (UTC)
Subject: [Distutils] buildout/distribute recursion error fixed
Message-ID: <slrnhf53cf.tdr.reinout@isengrim.local>

Hi all,

Several people have seen a hard-to-reproduce recursion error when running
buildout in combination with distribute.  You'd get 200 lines of
"distribute".

Good news: the problem has been identified.  And I've fixed it on a
zc.buildout branch (branches/reinout_distribute_upgrade_check).  Tarek will
test it and probably/hopefully release it real soon, wink, wink.


Buildout's easy_install.py module detects the location of setuptools and
buildout at import time and uses those paths when creating scripts.
Setuptools's path is also used when calling easy_install to install .tgz
files.

Problem: the install() method that calls easy_install detects the setuptools
location all over again at runtime.  And there's a recursion in there.  The
method that searches for the available setuptools detects that there's a new
version available and tries to install() it.  And searches for the setuptools
location.  And tries to install() it.  Repeat.

I fixed it by simply using the pre-detected setuptools location.

Note that setuptools did not have this recursion problem solely because it was
packaged as an ``.egg``, which does not have to go through the easy_install
step!  So distribute's (necessary) packaging as a tgz exposed this bug.


Reinout

-- 
Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"


From ziade.tarek at gmail.com  Thu Nov  5 12:58:02 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 12:58:02 +0100
Subject: [Distutils] buildout/distribute recursion error fixed
In-Reply-To: <slrnhf53cf.tdr.reinout@isengrim.local>
References: <slrnhf53cf.tdr.reinout@isengrim.local>
Message-ID: <94bdd2610911050358n4cb88665wfcf4825301182bc2@mail.gmail.com>

On Thu, Nov 5, 2009 at 9:32 AM, Reinout van Rees <reinout at vanrees.org> wrote:
> Hi all,
>
> Several people have seen a hard-to-reproduce recursion error when running
> buildout in combination with distribute. ?You'd get 200 lines of
> "distribute".
>
> Good news: the problem has been identified. ?And I've fixed it on a
> zc.buildout branch (branches/reinout_distribute_upgrade_check). ?Tarek will
> test it and probably/hopefully release it real soon, wink, wink.

Yes, thanks for the fix/work ! It looks nice and also simplifies the code.

Before it's merged in trunk and eventually released, we need to :

- add a test demonstrating this bug in easy_install.txt

- manually test the various scenarii (these are not fully covered with
zc.buildout unittests because the assmption is made that python don't
have setuptools installed):

    - nude python, buildout without --distribute, plone buildout cfg file
    - python with setuptools, buildout witout --distribute, plone
buildout cfg file
    - python with distribute < 0.6.8, buildout witout --distribute,
plone buildout cfg file
    - these 3 tests *with* --distribute

Regards
Tarek

From cool-rr at cool-rr.com  Thu Nov  5 20:58:09 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Thu, 5 Nov 2009 19:58:09 +0000 (UTC)
Subject: [Distutils] Bundling Distribute
Message-ID: <loom.20091105T205342-413@post.gmane.org>

Hello,

I would like to bundle Distribute with my project
so that I could use it without making my users
install it. (I like to keep things as simple as
possible for my users.)

Is it possible? I thought of just copying the 
"setuptools" folder from the source distribution
into a package in my project, so instead of doing
`from setuptools import setup`, I would do 
`from third_party.setuptools import setup`.

Does it make sense? Will it work? Anything else I
should keep in mind?

Thanks,
Ram.


From sridharr at activestate.com  Thu Nov  5 21:31:59 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Thu, 05 Nov 2009 12:31:59 -0800
Subject: [Distutils] PEP 376 -- support for release versions?
Message-ID: <op.u2xvbld41q4jlt@whymac.activestate.com>

Hi,

While reading the PEP I noticed that there seems to be no straightforward  
to specify the package release versions.

   >>> egginfo_dirname('python-ldap', '2.3.10')
   'python_ldap-2.3.10.egg-info'

Consider the following PyPM package file name:

   python-ldap-2.3.10_linux-x86_2.6_2.pypm

Here 2.3.10 is python-ldap's version, 2.6 is the Python version .. and 2  
(the last component) is the binary release number (for ActiveState  
repository). This is to upgrade previous releases of the same python-ldap  
(say, python-ldap-2.3.10_linux-x86_2.6_1.pypm).

If PyPM is to rely on PEP 376 style .egg-info directories (and the RECORD  
files within them), how is it to store/retrieve this release number?  
Currently we store everything (including RECORD) in a sqlite database  
under site-packages. I can continue using the db for release numbers  
alone, but I really prefer that this is covered by a standard that  
evidently provides support for installer tools.

-srid

From ziade.tarek at gmail.com  Thu Nov  5 21:49:42 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 21:49:42 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T205342-413@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>
Message-ID: <94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>

On Thu, Nov 5, 2009 at 8:58 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> Hello,
>
> I would like to bundle Distribute with my project
> so that I could use it without making my users
> install it. (I like to keep things as simple as
> possible for my users.)
>
> Is it possible? I thought of just copying the
> "setuptools" folder from the source distribution
> into a package in my project, so instead of doing
> `from setuptools import setup`, I would do
> `from third_party.setuptools import setup`.
>
> Does it make sense? Will it work? Anything else I
> should keep in mind?

Hello,

It is possible to bundle, but the best strategy depends on what you are
doing in your project with Distribute. e.g. using it to provide
specific options in your setup.py, or using its
APIs, like what pkg_resources provides to query entry points for example.

Tarek

From cool-rr at cool-rr.com  Thu Nov  5 21:53:48 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Thu, 5 Nov 2009 20:53:48 +0000 (UTC)
Subject: [Distutils] Bundling Distribute
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
Message-ID: <loom.20091105T215050-831@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:

> 
> On Thu, Nov 5, 2009 at 8:58 PM, Ram Rachum <cool-rr <at> cool-rr.com> wrote:
> > <Pruned>
> > Does it make sense? Will it work? Anything else I
> > should keep in mind?
> 
> Hello,
> 
> It is possible to bundle, but the best strategy depends on what you are
> doing in your project with Distribute. e.g. using it to provide
> specific options in your setup.py, or using its
> APIs, like what pkg_resources provides to query entry points for example.
> 
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 
> 


I'll be using it my setup.py, and I've also heard about pkg_resources and I'll 
want to use that. And maybe there will be more interesting things in Distribute 
that I'll want to use. What do you suggest I should do?

Ram.









From ziade.tarek at gmail.com  Thu Nov  5 22:03:45 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 22:03:45 +0100
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <op.u2xvbld41q4jlt@whymac.activestate.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
Message-ID: <94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>

On Thu, Nov 5, 2009 at 9:31 PM, Sridhar Ratnakumar
<sridharr at activestate.com> wrote:
> Hi,
>
> While reading the PEP I noticed that there seems to be no straightforward to
> specify the package release versions.
>
> ?>>> egginfo_dirname('python-ldap', '2.3.10')
> ?'python_ldap-2.3.10.egg-info'
>
> Consider the following PyPM package file name:
>
> ?python-ldap-2.3.10_linux-x86_2.6_2.pypm
>
> Here 2.3.10 is python-ldap's version, 2.6 is the Python version .. and 2
> (the last component) is the binary release number (for ActiveState
> repository). This is to upgrade previous releases of the same python-ldap
> (say, python-ldap-2.3.10_linux-x86_2.6_1.pypm).

If I understand correctly, ActiveState has its own releases of
python-ldap repackaged in pypm files,
and has several releases per python-ldap versions.

So, why it doesn't have its own versions ? like: 2.3.10.2 in that
case. (following a PEP 386-compatible scheme)

So this would be:

>>> egginfo_dirname('python-ldap', '2.3.10.2')
'python_ldap-2.3.10.2.egg-info'


Tarek

From regebro at gmail.com  Thu Nov  5 22:22:24 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 5 Nov 2009 22:22:24 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T215050-831@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
Message-ID: <319e029f0911051322vdf08631ta749a88be53d4b0b@mail.gmail.com>

2009/11/5 Ram Rachum <cool-rr at cool-rr.com>:
> I'll be using it my setup.py, and I've also heard about pkg_resources and I'll
> want to use that. And maybe there will be more interesting things in Distribute
> that I'll want to use. What do you suggest I should do?

In that case you need it installed, and then your users should install it.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From ziade.tarek at gmail.com  Thu Nov  5 22:27:24 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 22:27:24 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T215050-831@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
Message-ID: <94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>

On Thu, Nov 5, 2009 at 9:53 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
[cut]
>
> I'll be using it my setup.py, and I've also heard about pkg_resources and I'll
> want to use that. And maybe there will be more interesting things in Distribute
> that I'll want to use. What do you suggest I should do?

There's a trick to use it without installing it :

- add the distribute_setup.py script besides your setup.py script
- call the use_setuptools API in setup.py, so distribute gets
downloaded then added in the path
  (see the documentation for that)

Although I think the best practice is to ask your users to install
Distribute explicitely, because
using Distribute under the hood implies that everytime someone runs
your setup.py script,
it will behave specifically. (and not like Distutils is supposed to behave)
For example, when "install" is called, it will install the
distribution following its own standard.

We are pushing in favor of an unified installation standard for all
tools (read PEP 376), and until this goal is
reached, I am now also suggesting people not to use the use_setuptools
trick, so the end-user knows what
is going on.

Tarek

From sridharr at activestate.com  Thu Nov  5 22:31:58 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Thu, 05 Nov 2009 13:31:58 -0800
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
Message-ID: <op.u2xx3kxx1q4jlt@whymac.activestate.com>

On Thu, 05 Nov 2009 13:03:45 -0800, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

> If I understand correctly, ActiveState has its own releases of
> python-ldap repackaged in pypm files,
> and has several releases per python-ldap versions.

Correct. Much like what Linux distributions (like debian) do.

> So, why it doesn't have its own versions ? like: 2.3.10.2 in that
> case. (following a PEP 386-compatible scheme)
> So this would be:
>
>>>> egginfo_dirname('python-ldap', '2.3.10.2')
> 'python_ldap-2.3.10.2.egg-info'

Because python-ldap may release a similar version in future. We don't have  
to cause collisions. This is why the release version is not part of  
package version in the *.pypm filename.

I actually now think that storing the release number inside the .egg-info/  
directory may be a better idea. Like:

$ cat python_ldap-2.3.10.egg-info/pypm.txt
release: 2
$

What do you think?

-srid



From robert.kern at gmail.com  Thu Nov  5 22:31:45 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 05 Nov 2009 15:31:45 -0600
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
Message-ID: <hcvg82$t65$1@ger.gmane.org>

On 2009-11-05 15:03 PM, Tarek Ziad? wrote:
> On Thu, Nov 5, 2009 at 9:31 PM, Sridhar Ratnakumar
> <sridharr at activestate.com>  wrote:
>> Hi,
>>
>> While reading the PEP I noticed that there seems to be no straightforward to
>> specify the package release versions.
>>
>>   >>>  egginfo_dirname('python-ldap', '2.3.10')
>>   'python_ldap-2.3.10.egg-info'
>>
>> Consider the following PyPM package file name:
>>
>>   python-ldap-2.3.10_linux-x86_2.6_2.pypm
>>
>> Here 2.3.10 is python-ldap's version, 2.6 is the Python version .. and 2
>> (the last component) is the binary release number (for ActiveState
>> repository). This is to upgrade previous releases of the same python-ldap
>> (say, python-ldap-2.3.10_linux-x86_2.6_1.pypm).
>
> If I understand correctly, ActiveState has its own releases of
> python-ldap repackaged in pypm files,
> and has several releases per python-ldap versions.
>
> So, why it doesn't have its own versions ? like: 2.3.10.2 in that
> case. (following a PEP 386-compatible scheme)
>
> So this would be:
>
>>>> egginfo_dirname('python-ldap', '2.3.10.2')
> 'python_ldap-2.3.10.2.egg-info'

Generally speaking, it is desirable to distinguish between the upstream 
package's version and the build number rather than munging them together. You 
never know when the upstream package will decide that they need to make a really 
quick bugfix release and just append ".1". The main package version is and 
should remain entirely under the control of the upstream developers, not the 
binary packagers. However, the binary packagers need a place to mark revisions 
of the binary packages distinct from the upstream sources.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From cool-rr at cool-rr.com  Thu Nov  5 22:45:39 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Thu, 5 Nov 2009 21:45:39 +0000 (UTC)
Subject: [Distutils] Bundling Distribute
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
Message-ID: <loom.20091105T224336-528@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:

> There's a trick to use it without installing it :
> 
> - add the distribute_setup.py script besides your setup.py script
> - call the use_setuptools API in setup.py, so distribute gets
> downloaded then added in the path
>   (see the documentation for that)
> 
> Although I think the best practice is to ask your users to install
> Distribute explicitely,

I'm against the online-install thing. I guess I'll tell my users to install 
Distribute, which I guess is not too bad since I have to tell that only to the 
users who install from source and not from binaries. Which leads me to my next 
point: When creating a binary with Distribute, will it install Distribute 
automatically on the user's computer? I would like it to.

Ram.


From ziade.tarek at gmail.com  Thu Nov  5 23:14:19 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 23:14:19 +0100
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <op.u2xx3kxx1q4jlt@whymac.activestate.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
Message-ID: <94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>

On Thu, Nov 5, 2009 at 10:31 PM, Sridhar Ratnakumar
<sridharr at activestate.com> wrote:
> Because python-ldap may release a similar version in future. We don't have
> to cause collisions. This is why the release version is not part of package
> version in the *.pypm filename.

Ok

>
> I actually now think that storing the release number inside the .egg-info/
> directory may be a better idea. Like:
>
> $ cat python_ldap-2.3.10.egg-info/pypm.txt
> release: 2
> $
>
> What do you think?

In that case, an arbitrary file that gets installed in the .egg-info/
directory, that you can read back when needed through the PEP 376
pkgutil new API, seem to be a good idea.

This implies that Python+distutils itself will not be able to
distinguish two different releases that have the same version, but a
different build version.

e.g. that just Active State's package manager is able to manage this
extra number.

The other option, if this extra binary version number is a generic
need, would be to add a new metadata field in PEP 345. (like in RPMs
IIRC) and build an egg info folder name that contains this extra info
when present.

Regards
Tarek

From carl at meyerloewen.net  Thu Nov  5 23:19:46 2009
From: carl at meyerloewen.net (Carl Meyer)
Date: Thu, 05 Nov 2009 17:19:46 -0500
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
Message-ID: <4AF34F82.30507@meyerloewen.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Tarek Ziad? wrote:
> In that case, an arbitrary file that gets installed in the .egg-info/
> directory, that you can read back when needed through the PEP 376
> pkgutil new API, seem to be a good idea.

PEP 376 doesn't include a pkgutil API for reading arbitrary metadata
files from .egg-info. But IMO it probably should.

Carl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFK80+CFRcxmeyPUXIRAkojAJ41bHVhe0Omasz5fOR1sxaRWURvMQCfYxSq
JteyNimZuPYNdo9b9Pz+dHU=
=1G0s
-----END PGP SIGNATURE-----

From ziade.tarek at gmail.com  Thu Nov  5 23:20:05 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 23:20:05 +0100
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <hcvg82$t65$1@ger.gmane.org>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<hcvg82$t65$1@ger.gmane.org>
Message-ID: <94bdd2610911051420i464aaa78y2855cc38e900cc7@mail.gmail.com>

On Thu, Nov 5, 2009 at 10:31 PM, Robert Kern <robert.kern at gmail.com> wrote:
[..]
> Generally speaking, it is desirable to distinguish between the upstream
> package's version and the build number rather than munging them together.
> You never know when the upstream package will decide that they need to make
> a really quick bugfix release and just append ".1". The main package version
> is and should remain entirely under the control of the upstream developers,
> not the binary packagers. However, the binary packagers need a place to mark
> revisions of the binary packages distinct from the upstream sources.

I see. The question is: do we want to handle this at Distutils level
(e.g. by adding a new metadata
field as I said in my earlier mail), or is this to be done by the
package manager that is on the top
of Python on the target system (pypm, debian dpkg, etc..) and can
drive installations with PEP 376 's
APIs we have described earlier.

I would say the latter is the best pick.

Tarek

From ziade.tarek at gmail.com  Thu Nov  5 23:21:42 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 23:21:42 +0100
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <4AF34F82.30507@meyerloewen.net>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
	<4AF34F82.30507@meyerloewen.net>
Message-ID: <94bdd2610911051421g7f326d98v83f0e5b6ef4f27a5@mail.gmail.com>

On Thu, Nov 5, 2009 at 11:19 PM, Carl Meyer <carl at meyerloewen.net> wrote:
>
> PEP 376 doesn't include a pkgutil API for reading arbitrary metadata
> files from .egg-info. But IMO it probably should.


It does, that's "Distribution.get_egginfo_file()"


Regards
Tarek

From regebro at gmail.com  Thu Nov  5 23:32:53 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 5 Nov 2009 23:32:53 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T224336-528@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
Message-ID: <319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>

2009/11/5 Ram Rachum <cool-rr at cool-rr.com>:
> Which leads me to my next
> point: When creating a binary with Distribute, will it install Distribute
> automatically on the user's computer?

1. No.
2. You should only create binary releases if your module contains
C-code and then only for Windows.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From sridharr at activestate.com  Thu Nov  5 23:39:24 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Thu, 05 Nov 2009 14:39:24 -0800
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
Message-ID: <op.u2x07yfb1q4jlt@whymac.activestate.com>

On Thu, 05 Nov 2009 14:14:19 -0800, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

>> I actually now think that storing the release number inside the  
>> .egg-info/
>> directory may be a better idea. Like:
>>
>> $ cat python_ldap-2.3.10.egg-info/pypm.txt
>> release: 2
>> $
>>
>> What do you think?
> In that case, an arbitrary file that gets installed in the .egg-info/
> directory, that you can read back when needed through the PEP 376
> pkgutil new API, seem to be a good idea.
> This implies that Python+distutils itself will not be able to
> distinguish two different releases that have the same version, but a
> different build version.

I assume that the user is not supposed to have two copies of  
python-ldap-2.3.10 (but with different release versions) anyway. The  
purpose of multiple release version is to provide upgrades to a package  
within a single package version.

The *value* of the release version itself is specific to PyPM, of course.  
Other binary installers will not have a need for it (they will use their  
own format -- see below).

> e.g. that just Active State's package manager is able to manage this
> extra number.

Yes. And package manager FooBar may use different release number formats,  
say, timestamps.

$ foobar install python-ldap
Getting python-ldap_2.3.10_linux_20091023.fbar

FooBar should not read PyPM's release number.

> The other option, if this extra binary version number is a generic
> need, would be to add a new metadata field in PEP 345. (like in RPMs
> IIRC) and build an egg info folder name that contains this extra info
> when present.

Since release numbers are specific to the installer (and repository), how  
about having it in the .egg-info/INSTALLER file?

$ cat .egg-info/INSTALLER
name: PyPM

[package]
release: 2
url: http://pypm.activestate.com/..../python-ldap....pypm
pinned: True
...

This will also enable adding extra metadata (specific to installer) like  
version pinning.

-srid




From ziade.tarek at gmail.com  Thu Nov  5 23:39:29 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 23:39:29 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
Message-ID: <94bdd2610911051439n1e27483do2b755688269f6d1e@mail.gmail.com>

On Thu, Nov 5, 2009 at 11:32 PM, Lennart Regebro <regebro at gmail.com> wrote:
> 2009/11/5 Ram Rachum <cool-rr at cool-rr.com>:
>> Which leads me to my next
>> point: When creating a binary with Distribute, will it install Distribute
>> automatically on the user's computer?
>
> 1. No.
> 2. You should only create binary releases if your module contains
> C-code and then only for Windows.

3. If your project is for a specific target system that is not win32,
another option is
    to re-package your distribution into a platform-specific binary
distribution, like a .deb file
    or a .rpm file.

Regards
Tarek

From ziade.tarek at gmail.com  Thu Nov  5 23:45:47 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 5 Nov 2009 23:45:47 +0100
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <op.u2x07yfb1q4jlt@whymac.activestate.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
	<op.u2x07yfb1q4jlt@whymac.activestate.com>
Message-ID: <94bdd2610911051445r73efe9beq3e2a8901584f7172@mail.gmail.com>

On Thu, Nov 5, 2009 at 11:39 PM, Sridhar Ratnakumar
<sridharr at activestate.com> wrote:
[..]
>
> Since release numbers are specific to the installer (and repository), how
> about having it in the .egg-info/INSTALLER file?
>
> $ cat .egg-info/INSTALLER
> name: PyPM
>
> [package]
> release: 2
> url: http://pypm.activestate.com/..../python-ldap....pypm
> pinned: True
> ...

We specified an INSTALLER file already, see:
http://www.python.org/dev/peps/pep-0376/#id33

But just with the name of the installer. The use case was to be able
to avoid removing a distribution
that was installed by another installer.

I don't think the extra bit ([package]) is necessary in the INSTALLER
file, because it's specific to the installer tool.
And, as long as you can read/write arbitrary files in /.egg-info, you
can put it in a specific file.

Regards
Tarek

From cool-rr at cool-rr.com  Thu Nov  5 23:48:18 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Thu, 5 Nov 2009 22:48:18 +0000 (UTC)
Subject: [Distutils] Bundling Distribute
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
Message-ID: <loom.20091105T234736-654@post.gmane.org>

Lennart Regebro <regebro <at> gmail.com> writes:

> 2. You should only create binary releases if your module contains
> C-code and then only for Windows.
> 

Why only if there's C code? I personally find it more convenient to install 
binaries than from source.


From regebro at gmail.com  Thu Nov  5 23:50:58 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 5 Nov 2009 23:50:58 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T234736-654@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
Message-ID: <319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>

2009/11/5 Ram Rachum <cool-rr at cool-rr.com>:
> Why only if there's C code?

Because in general you want to compile the code on the system when
it's Linux or osx etc. It works better, and you don't have to provide
both 32bit and 64 bit versions and yadayada.

> I personally find it more convenient to install binaries than from source.

Why?

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From sridharr at activestate.com  Thu Nov  5 23:56:13 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Thu, 05 Nov 2009 14:56:13 -0800
Subject: [Distutils] Bundling Distribute
In-Reply-To: <319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
Message-ID: <op.u2x1zz161q4jlt@whymac.activestate.com>

On Thu, 05 Nov 2009 14:50:58 -0800, Lennart Regebro <regebro at gmail.com>  
wrote:

>> I personally find it more convenient to install binaries than from  
>> source.
> Why?

1. Not having to deal build errors (due to missing libs or corrupt lib  
installations)
2. Not having to have a compiler suite installed (VS, xcode)
3. Installing binaries take less time than for source (which you have to  
compile)
4. Support for binary repositories (eg: pypm.activestate.com) where you  
can put custom builds (eg: launchpad repos)

Point (1) and (2) are very important for Windows.

-srid

From cool-rr at cool-rr.com  Fri Nov  6 00:00:21 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Thu, 5 Nov 2009 23:00:21 +0000 (UTC)
Subject: [Distutils] Bundling Distribute
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
Message-ID: <loom.20091105T235457-769@post.gmane.org>

Lennart Regebro <regebro <at> gmail.com> writes:

> 
> 2009/11/5 Ram Rachum <cool-rr <at> cool-rr.com>:
> > Why only if there's C code?
> 
> Because in general you want to compile the code on the system when
> it's Linux or osx etc. It works better, and you don't have to provide
> both 32bit and 64 bit versions and yadayada.
> 
> > I personally find it more convenient to install binaries than from source.
> 
> Why?
> 

Because it requires less work on the user's side.

Anyway, I guess I'll supply both binaries and code, so people will install what 
they want.



From cool-rr at cool-rr.com  Fri Nov  6 00:02:35 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Fri, 6 Nov 2009 01:02:35 +0200
Subject: [Distutils] Bundling Distribute
In-Reply-To: <94bdd2610911051439n1e27483do2b755688269f6d1e@mail.gmail.com>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com> 
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com> 
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com> 
	<94bdd2610911051439n1e27483do2b755688269f6d1e@mail.gmail.com>
Message-ID: <a422b72d0911051502k63dc4779q4fc11546feee1d92@mail.gmail.com>

On Fri, Nov 6, 2009 at 12:39 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> On Thu, Nov 5, 2009 at 11:32 PM, Lennart Regebro <regebro at gmail.com>
> wrote:
> > 2009/11/5 Ram Rachum <cool-rr at cool-rr.com>:
> >> Which leads me to my next
> >> point: When creating a binary with Distribute, will it install
> Distribute
> >> automatically on the user's computer?
> >
> > 1. No.
> > 2. You should only create binary releases if your module contains
> > C-code and then only for Windows.
>
> 3. If your project is for a specific target system that is not win32,
> another option is
>    to re-package your distribution into a platform-specific binary
> distribution, like a .deb file
>    or a .rpm file.
>
> Regards
> Tarek
>

I'm a bit lost. Is it possible in any way to have Distribute automatically
installed when installing my program, without requiring an internet
connection?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091106/68f78840/attachment.htm>

From regebro at gmail.com  Fri Nov  6 00:09:22 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 6 Nov 2009 00:09:22 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <op.u2x1zz161q4jlt@whymac.activestate.com>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
	<op.u2x1zz161q4jlt@whymac.activestate.com>
Message-ID: <319e029f0911051509t7178d2c3qe31729b97774c72@mail.gmail.com>

2009/11/5 Sridhar Ratnakumar <sridharr at activestate.com>:
> 1. Not having to deal build errors (due to missing libs or corrupt lib
> installations)

Then you need to compile the libs statically, and then you open up
another can of worms with incompatibilities there.

> 2. Not having to have a compiler suite installed (VS, xcode)

If you install a python library, you have those installed. (Except on Windows).

> 3. Installing binaries take less time than for source (which you have to
> compile)

Which is only relevant of you have massive amounts of C-code. Even big
projects take 15 seconds instead if 3 seconds. Hardly a big problem.

> 4. Support for binary repositories (eg: pypm.activestate.com) where you can
> put custom builds (eg: launchpad repos)

This I don't understand.

> Point (1) and (2) are very important for Windows.

Yes. Which is why I explicitly said that you shouldn't distribute
binaries, except for Windows.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From floris.bruynooghe at gmail.com  Fri Nov  6 01:05:26 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 6 Nov 2009 00:05:26 +0000
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <94bdd2610911051445r73efe9beq3e2a8901584f7172@mail.gmail.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<94bdd2610911051414i51ffc7afra8a055d6ebe247c5@mail.gmail.com>
	<op.u2x07yfb1q4jlt@whymac.activestate.com>
	<94bdd2610911051445r73efe9beq3e2a8901584f7172@mail.gmail.com>
Message-ID: <20091106000526.GB32130@laurie.devork>

On Thu, Nov 05, 2009 at 11:45:47PM +0100, Tarek Ziad? wrote:
> On Thu, Nov 5, 2009 at 11:39 PM, Sridhar Ratnakumar
> <sridharr at activestate.com> wrote:
> [..]
> >
> > Since release numbers are specific to the installer (and repository), how
> > about having it in the .egg-info/INSTALLER file?
> >
> > $ cat .egg-info/INSTALLER
> > name: PyPM
> >
> > [package]
> > release: 2
> > url: http://pypm.activestate.com/..../python-ldap....pypm
> > pinned: True
> > ...
> 
> We specified an INSTALLER file already, see:
> http://www.python.org/dev/peps/pep-0376/#id33
> 
> But just with the name of the installer. The use case was to be able
> to avoid removing a distribution
> that was installed by another installer.
> 
> I don't think the extra bit ([package]) is necessary in the INSTALLER
> file, because it's specific to the installer tool.
> And, as long as you can read/write arbitrary files in /.egg-info, you
> can put it in a specific file.

It seems a bad idea to put this extra info in a distutils standard,
the binary release version have been managed outside of distutils for
a long time and it works perfect.

Allowing an arbitrary file to to be read via the egg-metadata API
seems a much better solution, then any packager using that facility
can completely decide what format to use and freely encode more
information without hiting new distutils limitations (like your
pinning).  Of course there's nothing wrong with pypm keeping it's
sqlite db for it's own data either, other binary package managers keep
that info outside too.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ziade.tarek at gmail.com  Fri Nov  6 02:22:42 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 6 Nov 2009 02:22:42 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <a422b72d0911051502k63dc4779q4fc11546feee1d92@mail.gmail.com>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<94bdd2610911051439n1e27483do2b755688269f6d1e@mail.gmail.com>
	<a422b72d0911051502k63dc4779q4fc11546feee1d92@mail.gmail.com>
Message-ID: <94bdd2610911051722u1ad89667n606f00a9c98de5da@mail.gmail.com>

On Fri, Nov 6, 2009 at 12:02 AM, cool-RR <cool-rr at cool-rr.com> wrote:
>
> I'm a bit lost. Is it possible in any way to have Distribute automatically
> installed when installing my program, without requiring an internet
> connection?

You need an internet connection unless you provide an archive of
distribute alongside
the distribute_setup.py script in your archive. In that case, it will
use it rather than getting
it at PyPI,

So yes, there is a way to get it installed with your distribution.
That's what the next virtualenv release
does in fact.

zc.buildout has a similar mechanism with its eggs folder. As a matter
of fact, some people build
their buildout instance and just zip it with all included
(pre-downloaded) dependencies. Then you can just unzip it
and use it with no internet connection on a similar target system.
That's a "big" binary distribution :)

Regards,
Tarek

From pje at telecommunity.com  Fri Nov  6 05:44:22 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 05 Nov 2009 23:44:22 -0500
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T235457-769@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
	<loom.20091105T235457-769@post.gmane.org>
Message-ID: <20091106044426.EF0E33A40A8@sparrow.telecommunity.com>

At 11:00 PM 11/5/2009 +0000, Ram Rachum wrote:
>Lennart Regebro <regebro <at> gmail.com> writes:
>
> >
> > 2009/11/5 Ram Rachum <cool-rr <at> cool-rr.com>:
> > > Why only if there's C code?
> >
> > Because in general you want to compile the code on the system when
> > it's Linux or osx etc. It works better, and you don't have to provide
> > both 32bit and 64 bit versions and yadayada.
> >
> > > I personally find it more convenient to install binaries than 
> from source.
> >
> > Why?
> >
>
>Because it requires less work on the user's side.

Not for people using easy_install or its derivatives (like pip and 
buildout).  easy_install-based tools work as easily with source as binaries.


From regebro at gmail.com  Fri Nov  6 07:21:47 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 6 Nov 2009 07:21:47 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <20091106044426.EF0E33A40A8@sparrow.telecommunity.com>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
	<loom.20091105T235457-769@post.gmane.org>
	<20091106044426.EF0E33A40A8@sparrow.telecommunity.com>
Message-ID: <319e029f0911052221h1abdf285j315db6a43e1dc04f@mail.gmail.com>

2009/11/6 P.J. Eby <pje at telecommunity.com>:
> Not for people using easy_install or its derivatives (like pip and
> buildout). ?easy_install-based tools work as easily with source as binaries.

Right. And untaring and running setup.py install is not exactly a
massive amount of work either. :)

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From ben+python at benfinney.id.au  Fri Nov  6 07:36:34 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 06 Nov 2009 17:36:34 +1100
Subject: [Distutils] Bundling Distribute
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
	<loom.20091105T235457-769@post.gmane.org>
	<20091106044426.EF0E33A40A8@sparrow.telecommunity.com>
	<319e029f0911052221h1abdf285j315db6a43e1dc04f@mail.gmail.com>
Message-ID: <87r5sc6txp.fsf@benfinney.id.au>

Lennart Regebro <regebro at gmail.com> writes:

> 2009/11/6 P.J. Eby <pje at telecommunity.com>:
> > Not for people using easy_install or its derivatives (like pip and
> > buildout). ?easy_install-based tools work as easily with source as
> > binaries.

?binary package? != ?setuptools egg?. At least, I didn't take the parent
of P.J.'s post to mean that.

> Right. And untaring and running setup.py install is not exactly a
> massive amount of work either. :)

Time spent installing is only part of the cost. Neither of thosr options
cooperate with the operating system's *own* package management. Which
causes much frustration and wasted time for the user when trying to
later interact with the installed package in some way (use another
package that depends on it, detect whether its installed, remove it
reliably, etc.).

That's when a ?binary package?, built by an operating system vendor or
some other trusted third party, is much less effort over time; and many
users know this, and will opt for such packages by preference.

-- 
 \        ?Members of the general public commonly find copyright rules |
  `\        implausible, and simply disbelieve them.? ?Jessica Litman, |
_o__)                                              _Digital Copyright_ |
Ben Finney


From regebro at gmail.com  Fri Nov  6 08:03:11 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Fri, 6 Nov 2009 08:03:11 +0100
Subject: [Distutils] Bundling Distribute
In-Reply-To: <87r5sc6txp.fsf@benfinney.id.au>
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
	<319e029f0911051450m519b7aabt4f942471bdf8ecc@mail.gmail.com>
	<loom.20091105T235457-769@post.gmane.org>
	<20091106044426.EF0E33A40A8@sparrow.telecommunity.com>
	<319e029f0911052221h1abdf285j315db6a43e1dc04f@mail.gmail.com>
	<87r5sc6txp.fsf@benfinney.id.au>
Message-ID: <319e029f0911052303v6aafca15r6b754cf22ecc1b8a@mail.gmail.com>

2009/11/6 Ben Finney <ben+python at benfinney.id.au>:
> Time spent installing is only part of the cost. Neither of thosr options
> cooperate with the operating system's *own* package management.

Well, do to THAT you have to make packages for that systems own
package management, and that's a completely different issue, with it's
own set of problems, religions and wars.

> That's when a ?binary package?

> Using the word "binary package" to mean a, built by an operating system vendor or
> some other trusted third party, is much less effort over time; and many
> users know this, and will opt for such packages by preference.

For end user applications, sure. It has not been specified if we are
talking about and end-user application that should have an installer
etc.

Note that this discussion started with bundling distribute so it was
imported from third_party.setuptools instead of just setuptools. A
fairly specific question. Then the discussion suddenly was about
project that had long compile times. And now about making
distro-specific distributions for easy installation.

This is a whole different ballgame from where the discussion started.
Arguments are being made against statements by switching the context
in which the question is being made. That's not a constructive type of
discussion. There is nothing from the initial question that indicates
that it's a big project for end users with loads of C-code in it.
Let's not make the issue more complicated than it already is.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From cool-rr at cool-rr.com  Fri Nov  6 09:38:11 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Fri, 6 Nov 2009 08:38:11 +0000 (UTC)
Subject: [Distutils] Bundling Distribute
References: <loom.20091105T205342-413@post.gmane.org>
	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>
	<loom.20091105T215050-831@post.gmane.org>
	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>
	<loom.20091105T224336-528@post.gmane.org>
	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<94bdd2610911051439n1e27483do2b755688269f6d1e@mail.gmail.com>
	<a422b72d0911051502k63dc4779q4fc11546feee1d92@mail.gmail.com>
	<94bdd2610911051722u1ad89667n606f00a9c98de5da@mail.gmail.com>
Message-ID: <loom.20091106T093622-269@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:
> You need an internet connection unless you provide an archive of
> distribute alongside
> the distribute_setup.py script in your archive. In that case, it will
> use it rather than getting
> it at PyPI,
> 
> So yes, there is a way to get it installed with your distribution.
> That's what the next virtualenv release
> does in fact.

Hey, that sounds pretty good! How can I do that exactly?

========================================================


From Jim.Vickroy at noaa.gov  Fri Nov  6 12:38:58 2009
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Fri, 06 Nov 2009 04:38:58 -0700
Subject: [Distutils] distutils.core.setup --install-script option in Python
	2.6 ?
Message-ID: <4AF40AD2.6060108@noaa.gov>

Hello,

I have just upgraded from Python 2.5 to 2.6.4 (MS Windows XP service 
pack 3).

My setup.py scripts (based on distutils.core.setup) no longer work with 
the --install-script option.  Here is the error:

    * error: option --install-script not recognized

The above error occurs when passing either bdist_wininst or bdist_msi 
parameters to the setup.py scripts.

I have been unable to locate any on-line mention of this option not 
being supported.

I apologize for missing something obvious here, but I am stuck.

Thanks in advance for your feedback.

-- jv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091106/6f29f38a/attachment.htm>

From setuptools at bugs.python.org  Fri Nov  6 15:24:43 2009
From: setuptools at bugs.python.org (Sridhar Ratnakumar)
Date: Fri, 06 Nov 2009 14:24:43 +0000
Subject: [Distutils] [issue91] setuptools fails with non-ascii urls in pypi
In-Reply-To: <1257517483.24.0.377044203808.issue91@psf.upfronthosting.co.za>
Message-ID: <1257517483.24.0.377044203808.issue91@psf.upfronthosting.co.za>


New submission from Sridhar Ratnakumar <sridharr at activestate.com>:

Calling `fetch_distribution` with req = "fl-gelform" and source = True results 
in the following traceback.

    sdist = pi.fetch_distribution(req, directory, source=True)
  File "/home/sridharr/.local/lib/python2.6/site-packages/setuptools/
package_index.py", line 465, in fetch_distribution
    self.find_packages(requirement)
  File "/home/sridharr/.local/lib/python2.6/site-packages/setuptools/
package_index.py", line 303, in find_packages
    self.scan_url(self.index_url + requirement.unsafe_name+'/')
  File "/home/sridharr/.local/lib/python2.6/site-packages/setuptools/
package_index.py", line 617, in scan_url
    self.process_url(url, True)
  File "/home/sridharr/.local/lib/python2.6/site-packages/setuptools/
package_index.py", line 190, in process_url
    f = self.open_url(url, "Download error: %s -- Some packages may not be 
found!")
  File "/home/sridharr/.local/lib/python2.6/site-packages/setuptools/
package_index.py", line 578, in open_url
    return open_with_auth(url)
  File "/home/sridharr/.local/lib/python2.6/site-packages/setuptools/
package_index.py", line 717, in open_with_auth
    fp = urllib2.urlopen(request)
  File "/opt/ActivePython-2.6/lib/python2.6/urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "/opt/ActivePython-2.6/lib/python2.6/urllib2.py", line 395, in open
    response = meth(req, response)
  File "/opt/ActivePython-2.6/lib/python2.6/urllib2.py", line 508, in 
http_response
    'http', request, response, code, msg, hdrs)
  File "/opt/ActivePython-2.6/lib/python2.6/urllib2.py", line 427, in error
    result = self._call_chain(*args)
  File "/opt/ActivePython-2.6/lib/python2.6/urllib2.py", line 367, in 
_call_chain
    result = func(*args)
  File "/opt/ActivePython-2.6/lib/python2.6/urllib2.py", line 577, in 
http_error_302
    newurl = urlparse.urljoin(req.get_full_url(), newurl)
  File "/opt/ActivePython-2.6/lib/python2.6/urlparse.py", line 219, in urljoin
    params, query, fragment))
  File "/opt/ActivePython-2.6/lib/python2.6/urlparse.py", line 184, in 
urlunparse
    return urlunsplit((scheme, netloc, url, query, fragment))
  File "/opt/ActivePython-2.6/lib/python2.6/urlparse.py", line 190, in 
urlunsplit
    url = '//' + (netloc or '') + url
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: 
ordinal not in range(128)

----------
messages: 450
nosy: srid
priority: bug
status: unread
title: setuptools fails with non-ascii urls in pypi

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue91>
_______________________________________________

From tseaver at palladion.com  Fri Nov  6 20:53:45 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 06 Nov 2009 14:53:45 -0500
Subject: [Distutils] Bundling Distribute
In-Reply-To: <loom.20091105T234736-654@post.gmane.org>
References: <loom.20091105T205342-413@post.gmane.org>	<94bdd2610911051249o4a983929sf9e248b9d224eeaf@mail.gmail.com>	<loom.20091105T215050-831@post.gmane.org>	<94bdd2610911051327h51a5ba7dg517aa3087e868118@mail.gmail.com>	<loom.20091105T224336-528@post.gmane.org>	<319e029f0911051432gb4a2ebekf0bcf9438438df22@mail.gmail.com>
	<loom.20091105T234736-654@post.gmane.org>
Message-ID: <4AF47EC9.1010107@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ram Rachum wrote:
> Lennart Regebro <regebro <at> gmail.com> writes:
> 
>> 2. You should only create binary releases if your module contains
>> C-code and then only for Windows.
>>
> 
> Why only if there's C code? I personally find it more convenient to install 
> binaries than from source.

I find it hard to believe that you can even tell the difference:  the
time to create a binary egg from an sdist is vanishingly small, dwarfed
by the time to download either.  Binary eggs are seriously problematic
in some cases:  e.g., UCS2 vs. UCS4 builds on Linux, or finc / macports
/ system python on OS/X.

Windows is the only platform where distributing binary eggs publically
makes sense, but only because Windows users can't build C extensions.
Some Windows folks prefer the bdist_wininst format to get integratoin
with the system's "Add / Remove Programs" feature, too.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkr0fskACgkQ+gerLs4ltQ6XXgCfcQiPjFXZ7de+JRAX5Br98TyW
6VkAnixqA9aLe+l+ADUmFPqrVsScpiii
=L35w
-----END PGP SIGNATURE-----


From tepperly at llnl.gov  Fri Nov  6 17:40:21 2009
From: tepperly at llnl.gov (Tom Epperly)
Date: Fri, 06 Nov 2009 08:40:21 -0800
Subject: [Distutils] Looking for portable what to determine directory where
 extensions are installed?
Message-ID: <4AF45175.3020001@llnl.gov>

I work on a language interoperability tool, Babel
https://computation.llnl.gov/casc/components/components.html; and I need
a portable way to determine the directory where Python extension modules
are installed by distutils (i.e., lib or lib64) to resolve this issue:
https://www.cca-forum.org/bugs/babel/issue670

For example, I will invoke "python setup.py install
--prefix=/home/foo/_inst --exec-prefix=/home/foo/_inst", and it usually
installs the extensions in either
/home/foo/_inst/lib/python2.5/site-packages or
/home/foo/_inst/lib64/python2.5/site-packages. Initially, I did the
following to determine where the extensions actually got installed:

# assume exec_prefix=/home/foo/_inst
# assume python_version=`python -c 'import sys; print sys.version' | sed 
'1s/^\(...\).*/\1/g;1q'`
RUNTIME_PYTHON="$exec_prefix/lib/python$python_verbose/site-packages"

This worked fine until I started running on 64-bit Python machines where
the correct result was

RUNTIME_PYTHON="$exec_prefix/lib64/python$python_verbose/site-packages"

The first 64-bit machine I was using seemed to have this patch:
http://bugs.python.org/file14726/Python-2.6.2-multilib.patch, so I
amended my script to the following:

pylib=`$PYTHON -c "import sys; print sys.__dict__.get('lib','lib')"`
RUNTIME_PYTHON="$exec_prefix/$pylib/python$python_version/site-packages"

However, this approach doesn't work with Fedora Core 12 prerelease or
some other 64-bit Linux distributions. They don't define "sys.lib".

I thought distutils.sysconfig.get_python_lib() might be helpful, but it
returns "/usr/lib/python2.6/site-packages" even on the system where
"/usr/lib64/python2.6/site-packages" is the right answer.

Is there a canonical, Pythonic way to determine whether an installation
of Python uses "lib" or "lib64"?  Or is there a way to determine the
full path of where distutils will install extensions relative to the
specified exec prefix?

Regards,

Tom Epperly


From guido at python.org  Fri Nov  6 18:53:44 2009
From: guido at python.org (Guido van Rossum)
Date: Fri, 6 Nov 2009 09:53:44 -0800
Subject: [Distutils] People want CPAN :-)
Message-ID: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>

I just found this comment on my blog. People have told me this in
person too, so I believe it is real pain (even if the solution may be
elusive and the suggested solutions may not work). But I don't know
how to improve the world. Is the work on distutils-sig going to be
enough? Or do we need some other kind of work in addition? Do we need
more than PyPI?

--Guido

---------- Forwarded message ----------
From: dalloliogm <noreply-comment at blogger.com>
Date: Fri, Nov 6, 2009 at 8:01 AM
Subject: [Neopythonic] New comment on Python in the Scientific World.
To: gvanrossum at gmail.com


dalloliogm has left a new comment on your post "Python in the Scientific World":

Python is suffering a lot in the scientific word, because it has not a
CPAN-like repository.

PyPI is fine, but it is still far from the level of CPAN, CRAN,
Bioconductor, etc..

Scientists who use programming usually have a lot of different
interests and approaches, therefore it is really difficult to write a
package that can be useful to everyone.
Other programming language like Perl and R have repository-like
structure which enable people to download packages easily, and to
upload new ones and organize them withouth having to worry about
having to integrate them into existing packages.

This is what is happening to biopython now: it is a monolitic package
that it is supposed to work for any bioinformatic problem; but this is
so general that to accomplish that you would need to add a lot of
dependencies, to numpy, networkx, suds, any kind of library.
However, since easy_install is not as ready yet as the counterparts in
other languages, if the biopython developers add too many
dependencies, nobody will be able to install it properly, and nobody
will use it.



Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM


-- 
--Guido van Rossum (python.org/~guido)

From chris at simplistix.co.uk  Fri Nov  6 22:39:16 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 06 Nov 2009 21:39:16 +0000
Subject: [Distutils] setuptools tracker notifications
In-Reply-To: <4AEE8F73.9080200@v.loewis.de>
References: <1257073241.98.0.632218460882.issue89@psf.upfronthosting.co.za>	<94bdd2610911010442v5be7fed3rc7b066c82b56b8f6@mail.gmail.com>	<4AEDE8FA.6060602@v.loewis.de>	<94bdd2610911011351na5e83b2y3c89825d976ce0ca@mail.gmail.com>	<4AEE134E.9040201@v.loewis.de>	<20091102000640.452F83A40A8@sparrow.telecommunity.com>
	<4AEE320B.6030107@taupro.com> <4AEE865E.7090900@simplistix.co.uk>
	<4AEE8F73.9080200@v.loewis.de>
Message-ID: <4AF49784.50606@simplistix.co.uk>

Martin v. L?wis wrote:
>> Any chance the mail subject could be prefixed with [setuptools] to make
>> it clear which tracker it came from?
> 
> If you want to filter them out (or in a separate folder), I recommend to
> set your filter for the X-Roundup-Name header.

I don't, I just want to know that a message is for setuptools rather 
than distribute.

> Adding something to the subject is tricky, IIRC.

I find that hard to believe...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From setuptools at bugs.python.org  Fri Nov  6 22:59:14 2009
From: setuptools at bugs.python.org (Eddy Pronk)
Date: Fri, 06 Nov 2009 21:59:14 +0000
Subject: [Distutils] [issue92] [PATCH] Added download directory option
In-Reply-To: <1257544754.82.0.452696907547.issue92@psf.upfronthosting.co.za>
Message-ID: <1257544754.82.0.452696907547.issue92@psf.upfronthosting.co.za>


New submission from Eddy Pronk <epronk at muftor.com>:

I added the option --download-directory to keep the downloaded archives so I can
install them on another machine.

diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
index af4e349..c58cf91 100755
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -60,6 +60,8 @@ class easy_install(Command):
             "no longer needed; don't use this"),
         ("build-directory=", "b",
             "download/extract/build in DIR; keep the results"),
+        ("download-directory=", "z",
+            "download in DIR; keep the results"),
         ('optimize=', 'O',
          "also compile with optimization: -O1 for \"python -O\", "
          "-O2 for \"python -OO\", and -O0 to disable [default: -O0]"),
@@ -86,6 +88,7 @@ class easy_install(Command):
         self.index_url = None
         self.find_links = None
         self.build_directory = None
+        self.download_directory = None
         self.args = None
         self.optimize = self.record = None
         self.upgrade = self.always_copy = self.multi_version = None
@@ -208,7 +211,7 @@ class easy_install(Command):
             log.set_verbosity(self.verbose)
         try:
             for spec in self.args:
-                self.easy_install(spec, not self.no_deps)
+                self.easy_install(spec, deps=not self.no_deps)
             if self.record:
                 outputs = self.outputs
                 if self.root:               # strip any package prefix
@@ -410,15 +413,20 @@ Please make the appropriate changes for your system and
try again.
 
     def easy_install(self, spec, deps=False):
         tmpdir = tempfile.mkdtemp(prefix="easy_install-")
+        if self.download_directory:
+            download_directory = self.download_directory
+        else:
+            download_directory = tmpdir
+        
         download = None
         if not self.editable: self.install_site_py()
 
         try:
             if not isinstance(spec,Requirement):
                 if URL_SCHEME(spec):
-                    # It's a url, download it to tmpdir and process
+                    # It's a url, download it to download_directory and process
                     self.not_editable(spec)
-                    download = self.package_index.download(spec, tmpdir)
+                    download = self.package_index.download(spec,
download_directory)
                     return self.install_item(None, download, tmpdir, deps, True)
 
                 elif os.path.exists(spec):
@@ -430,7 +438,7 @@ Please make the appropriate changes for your system and try
again.
 
             self.check_editable(spec)
             dist = self.package_index.fetch_distribution(
-                spec, tmpdir, self.upgrade, self.editable, not self.always_copy,
+                spec, download_directory, self.upgrade, self.editable, not
self.always_copy,
                 self.local_index
             )
             if dist is None:
-- 
1.5.6.5

----------
messages: 458
nosy: epronk
priority: feature
status: unread
title: [PATCH] Added download directory option

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue92>
_______________________________________________

From alex.gronholm at nextday.fi  Fri Nov  6 23:14:36 2009
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sat, 07 Nov 2009 00:14:36 +0200
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <4AF49FCC.7050403@nextday.fi>

Guido van Rossum kirjoitti:
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?
>
> --Guido
>
> ---------- Forwarded message ----------
> From: dalloliogm <noreply-comment at blogger.com>
> Date: Fri, Nov 6, 2009 at 8:01 AM
> Subject: [Neopythonic] New comment on Python in the Scientific World.
> To: gvanrossum at gmail.com
>
>
> dalloliogm has left a new comment on your post "Python in the Scientific World":
>
> Python is suffering a lot in the scientific word, because it has not a
> CPAN-like repository.
>
> PyPI is fine, but it is still far from the level of CPAN, CRAN,
> Bioconductor, etc..
>
> Scientists who use programming usually have a lot of different
> interests and approaches, therefore it is really difficult to write a
> package that can be useful to everyone.
> Other programming language like Perl and R have repository-like
> structure which enable people to download packages easily, and to
> upload new ones and organize them withouth having to worry about
> having to integrate them into existing packages.
>
> This is what is happening to biopython now: it is a monolitic package
> that it is supposed to work for any bioinformatic problem; but this is
> so general that to accomplish that you would need to add a lot of
> dependencies, to numpy, networkx, suds, any kind of library.
> However, since easy_install is not as ready yet as the counterparts in
> other languages, if the biopython developers add too many
> dependencies, nobody will be able to install it properly, and nobody
> will use it.
>   
I for one did not understand the problem. What does CPAN have that PyPI 
doesn't?
It is natural for packages (distributions, in distutils terms) to have 
dependencies on each other. Why is this a problem?
>
>
> Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM
>
>
>   


From exarkun at twistedmatrix.com  Fri Nov  6 23:53:32 2009
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Fri, 06 Nov 2009 22:53:32 -0000
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF49FCC.7050403@nextday.fi>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
Message-ID: <20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>

On 10:14 pm, alex.gronholm at nextday.fi wrote:
>Guido van Rossum kirjoitti:
>>I just found this comment on my blog. People have told me this in
>>person too, so I believe it is real pain (even if the solution may be
>>elusive and the suggested solutions may not work). But I don't know
>>how to improve the world. Is the work on distutils-sig going to be
>>enough? Or do we need some other kind of work in addition? Do we need
>>more than PyPI?
>>
>>--Guido
>>
>>---------- Forwarded message ----------
>>From: dalloliogm <noreply-comment at blogger.com>
>>Date: Fri, Nov 6, 2009 at 8:01 AM
>>Subject: [Neopythonic] New comment on Python in the Scientific World.
>>To: gvanrossum at gmail.com
>>
>>
>>dalloliogm has left a new comment on your post "Python in the 
>>Scientific World":
>>
>>Python is suffering a lot in the scientific word, because it has not a
>>CPAN-like repository.
>>
>>PyPI is fine, but it is still far from the level of CPAN, CRAN,
>>Bioconductor, etc..
>>
>>Scientists who use programming usually have a lot of different
>>interests and approaches, therefore it is really difficult to write a
>>package that can be useful to everyone.
>>Other programming language like Perl and R have repository-like
>>structure which enable people to download packages easily, and to
>>upload new ones and organize them withouth having to worry about
>>having to integrate them into existing packages.
>>
>>This is what is happening to biopython now: it is a monolitic package
>>that it is supposed to work for any bioinformatic problem; but this is
>>so general that to accomplish that you would need to add a lot of
>>dependencies, to numpy, networkx, suds, any kind of library.
>>However, since easy_install is not as ready yet as the counterparts in
>>other languages, if the biopython developers add too many
>>dependencies, nobody will be able to install it properly, and nobody
>>will use it.
>I for one did not understand the problem. What does CPAN have that PyPI 
>doesn't?
>It is natural for packages (distributions, in distutils terms) to have 
>dependencies on each other. Why is this a problem?

I'm also not sure I see what problem CPAN is solving that PyPI is 
failing at.  At most, it sounds like the OP is complaining that the 
software available to scientists in perl is better than the software 
available to scientists in python (for some definition of better - 
there's more of it, it solves their particular problems better, 
whatever).

PyPI *does* let you download packages easily, it does let you upload new 
ones and organize them without having to worry about integrating them 
into existing packages.  So the features of CPAN that are described as 
desirable are already available in PyPI.

Jean-Paul

From bob at redivi.com  Fri Nov  6 23:56:51 2009
From: bob at redivi.com (Bob Ippolito)
Date: Fri, 6 Nov 2009 14:56:51 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
Message-ID: <6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>

On Fri, Nov 6, 2009 at 2:53 PM,  <exarkun at twistedmatrix.com> wrote:
> On 10:14 pm, alex.gronholm at nextday.fi wrote:
>>
>> Guido van Rossum kirjoitti:
>>>
>>> I just found this comment on my blog. People have told me this in
>>> person too, so I believe it is real pain (even if the solution may be
>>> elusive and the suggested solutions may not work). But I don't know
>>> how to improve the world. Is the work on distutils-sig going to be
>>> enough? Or do we need some other kind of work in addition? Do we need
>>> more than PyPI?
>>>
>>> --Guido
>>>
>>> ---------- Forwarded message ----------
>>> From: dalloliogm <noreply-comment at blogger.com>
>>> Date: Fri, Nov 6, 2009 at 8:01 AM
>>> Subject: [Neopythonic] New comment on Python in the Scientific World.
>>> To: gvanrossum at gmail.com
>>>
>>>
>>> dalloliogm has left a new comment on your post "Python in the Scientific
>>> World":
>>>
>>> Python is suffering a lot in the scientific word, because it has not a
>>> CPAN-like repository.
>>>
>>> PyPI is fine, but it is still far from the level of CPAN, CRAN,
>>> Bioconductor, etc..
>>>
>>> Scientists who use programming usually have a lot of different
>>> interests and approaches, therefore it is really difficult to write a
>>> package that can be useful to everyone.
>>> Other programming language like Perl and R have repository-like
>>> structure which enable people to download packages easily, and to
>>> upload new ones and organize them withouth having to worry about
>>> having to integrate them into existing packages.
>>>
>>> This is what is happening to biopython now: it is a monolitic package
>>> that it is supposed to work for any bioinformatic problem; but this is
>>> so general that to accomplish that you would need to add a lot of
>>> dependencies, to numpy, networkx, suds, any kind of library.
>>> However, since easy_install is not as ready yet as the counterparts in
>>> other languages, if the biopython developers add too many
>>> dependencies, nobody will be able to install it properly, and nobody
>>> will use it.
>>
>> I for one did not understand the problem. What does CPAN have that PyPI
>> doesn't?
>> It is natural for packages (distributions, in distutils terms) to have
>> dependencies on each other. Why is this a problem?
>
> I'm also not sure I see what problem CPAN is solving that PyPI is failing
> at. ?At most, it sounds like the OP is complaining that the software
> available to scientists in perl is better than the software available to
> scientists in python (for some definition of better - there's more of it, it
> solves their particular problems better, whatever).
>
> PyPI *does* let you download packages easily, it does let you upload new
> ones and organize them without having to worry about integrating them into
> existing packages. ?So the features of CPAN that are described as desirable
> are already available in PyPI.

It might be the CPAN user interface that makes the difference. Using
CPAN you can search for and install applications from an interactive
prompt. No web browser needed, just one command line invocation (perl
-MCPAN -e shell).

-bob

From david.lyon at preisshare.net  Fri Nov  6 23:52:55 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 06 Nov 2009 17:52:55 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <a21398cfb92acc7d47f8151c249291c4@preisshare.net>

On Fri, 6 Nov 2009 09:53:44 -0800, Guido van Rossum <guido at python.org>
wrote:
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). 

Hi Guido,

I'm a mere application developer - developing things for companies as 
they require. Sometimes in perl more often in python. 

I've worked in Perl with CPAN and yes I can attest to how good CPAN is
to the application developer. Anything you need is there. It's easy to
pull down stuff and load it in.

CPan have an 'over-the-top' philosophy towards developers. It works.

> Is the work on distutils-sig going to be enough? 

It's not on the roadmap circulated thus far - but to answer the 
question probably not.

At the moment, we don't have our target set towards working towards
that.

CPAN actually does so much, things that we could do, but are not.

For example, if Fred the developer makes a package 'microscope-
calibrate', and uploads, it will get tested and analysed on
CPAN with a buildbot. We don't do that yet.

So the packages on CPAN are typically of a higher quality, simply
because they've been machine checked. I like that.

Then, one the client machine, packages are so easy to install.

We don't have anything quite like that with Python yet but PJE
did get us one third of the way with setuptools. The skills to
get totally up with CPAN in this regard are there collectively 
with us now. Not in any one individual, but as a group. No one
individual can do it - but one individual can say - 'make it so'

> Or do we need some other kind of work in addition? Do we need
> more than PyPI?

Yes we do.

The Perl/CPAN developer experience is so smooth.

May I say that the current pypi website is 100% as good as
cpan. As a developer, I can't find any significant fault with
pypi. Apart from not having a buildbot running in the background
for package smoke testing. 

It's just when a developer goes to download and install a
package that things get complicated with python.

In an ideal world, a developer would just like to get eggs
or packages off pypi, install them and go back to work. It
would be fantastic for us to forget about what platform we
have and forget about what python version we have. CPAN
does that.

The totally best thing would be to see a package on pypi,
click on it and have it installed automatically there and
then.

The technical framework to do all this exists or is in
close to working order. Some things need doing but I would
regard the work as bridging rather than doing things from
scratch.

To me, the things Tarek is doing seem pretty sound and
sane. 

We just need to have a 'developer experience' program
or something along those lines to connect the dots and
make things run more smoothly.

Then having something as good as CPAN is totally realistic.

David




From kaelin at acm.org  Sat Nov  7 00:02:30 2009
From: kaelin at acm.org (Kaelin Colclasure)
Date: Fri, 6 Nov 2009 15:02:30 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF49FCC.7050403@nextday.fi>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
Message-ID: <7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>


On Nov 6, 2009, at 2:14 PM, Alex Gr?nholm wrote:

> Guido van Rossum kirjoitti:
>> I just found this comment on my blog. People have told me this in
>> person too, so I believe it is real pain (even if the solution may be
>> elusive and the suggested solutions may not work). But I don't know
>> how to improve the world. Is the work on distutils-sig going to be
>> enough? Or do we need some other kind of work in addition? Do we need
>> more than PyPI?
>>
>> --Guido
>>
>> ---------- Forwarded message ----------
>> From: dalloliogm <noreply-comment at blogger.com>
>> Date: Fri, Nov 6, 2009 at 8:01 AM
>> Subject: [Neopythonic] New comment on Python in the Scientific World.
>> To: gvanrossum at gmail.com
>>
>>
>> dalloliogm has left a new comment on your post "Python in the  
>> Scientific World":
>>
>> Python is suffering a lot in the scientific word, because it has  
>> not a
>> CPAN-like repository.
>>
>> PyPI is fine, but it is still far from the level of CPAN, CRAN,
>> Bioconductor, etc..
>>
>> Scientists who use programming usually have a lot of different
>> interests and approaches, therefore it is really difficult to write a
>> package that can be useful to everyone.
>> Other programming language like Perl and R have repository-like
>> structure which enable people to download packages easily, and to
>> upload new ones and organize them withouth having to worry about
>> having to integrate them into existing packages.
>>
>> This is what is happening to biopython now: it is a monolitic package
>> that it is supposed to work for any bioinformatic problem; but this  
>> is
>> so general that to accomplish that you would need to add a lot of
>> dependencies, to numpy, networkx, suds, any kind of library.
>> However, since easy_install is not as ready yet as the counterparts  
>> in
>> other languages, if the biopython developers add too many
>> dependencies, nobody will be able to install it properly, and nobody
>> will use it.
>>
> I for one did not understand the problem. What does CPAN have that  
> PyPI doesn't?
> It is natural for packages (distributions, in distutils terms) to  
> have dependencies on each other. Why is this a problem?


As both a CPAN contributor and a recent PyPI contributor I think I can  
speak to this:

When I packaged my first Perl module for CPAN distribution, there was  
no question about how to go about it. There was a clear recipe and it  
worked without a hitch the first time I tried.

When I recently packaged my first Python module for PyPI I was a bit  
dismayed to learn that easy_install is just one of several ways to  
disseminate Python code. The documentation is fragmented... The stuff  
I read on easy_install basically assumed you already knew all about  
distutils, and I eventually had to reorganize my code from a simple  
module to a package just to get a few non-code files to be pulled in  
by the install. A less persistent would-be contributor might have  
given up in frustration.

I joined this mailing list instead, and got my package working with  
some kind assistance. :-) But CPAN was just easier to contribute to.

-- Kaelin

>>
>>
>> Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM
>>
>>
>>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


From david.lyon at preisshare.net  Sat Nov  7 00:01:41 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 06 Nov 2009 18:01:41 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>
Message-ID: <e0ee771d9e908f207e9dba75a4182beb@preisshare.net>

On Fri, 6 Nov 2009 14:56:51 -0800, Bob Ippolito <bob at redivi.com> wrote:

> It might be the CPAN user interface that makes the difference. Using
> CPAN you can search for and install applications from an interactive
> prompt. No web browser needed, just one command line invocation (perl
> -MCPAN -e shell).

That's definitely one thing. We have nothing like that yet.

David

From guido at python.org  Sat Nov  7 00:14:00 2009
From: guido at python.org (Guido van Rossum)
Date: Fri, 6 Nov 2009 15:14:00 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <a21398cfb92acc7d47f8151c249291c4@preisshare.net>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com> 
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
Message-ID: <ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>

On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net> wrote:
> So the packages on CPAN are typically of a higher quality, simply
> because they've been machine checked. I like that.

Speaking purely on hearsay, I don't believe that. In fact, I've heard
plenty of laments about the complete lack of quality control on CPAN.

-- 
--Guido van Rossum (python.org/~guido)

From ziade.tarek at gmail.com  Sat Nov  7 00:44:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 00:44:32 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <94bdd2610911061544k86f28e2n3ef732415a86e5b7@mail.gmail.com>

On Fri, Nov 6, 2009 at 6:53 PM, Guido van Rossum <guido at python.org> wrote:
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?

I don't think PyPI has a structural problem. I do think the biggest
issue is related to the fact that it's not really possible to
describe a distribution and its dependencies today with a plain
Distutils (==stdlib). They can use Setuptools or Distribute, third
party tools. But then, there are using different installation
standards.

Changes are being done in PEP 345 + PEP 376 to address this. The goal
is to make sure Distutils provides what is required to describe
dependencies, a standard installation format, and a way to query
installed distributions.

Now, there's one thing CPAN has and PyPI don't: a network of package
repositories and an easy way to set your own
repository to serve distributions. IOW, one can sets its own CPAN
server, or add its server to an existing network.

That's useful for example if you want to have your own private PyPI
server and combine it with the public PyPI.

In PyPI, everything is centralized, and while this has some benefits,
it also makes it a single point of failure (SPF)
when you want to download distributions. We have suffered for that
problem in some sub communities
like Plone, that gets hundreds of packages at PyPI to get built. If
PyPI is down, there's nothing we can do unless we
have a local cache of packages. CPAN is way better in that field.

One work has been started to try to fix this SPF: a mirroring
standard, see PEP 381 - http://www.python.org/dev/peps/pep-0381/

In parallel, some server-side softwares are now allowing people to
register and upload packages using Distutils, like
what they would do with PyPI. For instance, you can use the "register"
and "upload" command to push
your distribution at plone.org That was made possible after some
changes to the .pypirc configuration file
(which is now allowing people to define several servers)

The next step is to add fail-over capabilities to tools like Pip or
Distribute, and ways to merge several sources of
distributions repositories: pypi and any other server that implements
the same protocol.

Tarek

From ziade.tarek at gmail.com  Sat Nov  7 00:46:01 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 00:46:01 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
Message-ID: <94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>

On Fri, Nov 6, 2009 at 11:53 PM,  <exarkun at twistedmatrix.com> wrote:
[..]
>
> PyPI *does* let you download packages easily, it does let you upload new
> ones and organize them without having to worry about integrating them into
> existing packages. ?So the features of CPAN that are described as desirable
> are already available in PyPI.

imho,

PyPI + PEP 381 + PEP 381 compliant clients == CPAN

Tarek

From ziade.tarek at gmail.com  Sat Nov  7 00:50:22 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 00:50:22 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>
Message-ID: <94bdd2610911061550t44acefe6r6f2bc53a8f8aa983@mail.gmail.com>

On Fri, Nov 6, 2009 at 11:56 PM, Bob Ippolito <bob at redivi.com> wrote:
[...]
>
> It might be the CPAN user interface that makes the difference. Using
> CPAN you can search for and install applications from an interactive
> prompt. No web browser needed, just one command line invocation (perl
> -MCPAN -e shell).

Although, we have an XML-RPC service for  PyPI, and Yolk for example
is a client that will let you query PyPI
through a prompt. I agree it could be enhanced, but it exists. Maybe
such a client could be added in Distutils ?

Plus, once the work on PEP 390 (or an equivalent one) is over, we will
be able to list a distribution's dependencies
on a given target system just by querying PYPI.

Tarek

From ianb at colorstudy.com  Sat Nov  7 00:50:46 2009
From: ianb at colorstudy.com (Ian Bicking)
Date: Fri, 6 Nov 2009 17:50:46 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <e0ee771d9e908f207e9dba75a4182beb@preisshare.net>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com> 
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com> 
	<e0ee771d9e908f207e9dba75a4182beb@preisshare.net>
Message-ID: <b654cd2e0911061550t2621ddcw9fc8e86accb4bf50@mail.gmail.com>

On Fri, Nov 6, 2009 at 5:01 PM, David Lyon <david.lyon at preisshare.net> wrote:
> On Fri, 6 Nov 2009 14:56:51 -0800, Bob Ippolito <bob at redivi.com> wrote:
>> It might be the CPAN user interface that makes the difference. Using
>> CPAN you can search for and install applications from an interactive
>> prompt. No web browser needed, just one command line invocation (perl
>> -MCPAN -e shell).
>
> That's definitely one thing. We have nothing like that yet.

My understanding is that they have all the same problems we do --
version management, dependencies, isolation, varying quality, etc.
When I've tried to install things from CPAN it's been surprisingly
obtuse.  This is not to say there aren't good things, but "we need
CPAN" doesn't really get to the heart of it.

>From (server-side) development perspective, CPAN seems to be a less
monolithic environment.  The core of it (at least from what Perl
people have told me) is really a very simple mirrored file repository,
with loose naming conventions.  Everything else is built up as added,
optional, decentralized services from that.  (I wonder if they are
using mirroring as a way to distribute these services?  E.g., register
your I-test-everything mirror, and it'll get packages uploaded to it
automatically.)

We have some of the same things.  Cheesecake is one example, and I
believe borrowed ideas and terminology from CPAN's equivalent service.
 Cheesecake, as an example, seems to be largely forgotten and unused.
It might be useful to unforget that product.

There are also places where (ironically?) Perl has One Way To Do It,
and benefits as a result.  Documentation of course, though maybe now
with Sphinx  we can start doing vertical integration using that tool.
Testing is a bit off; "setup.py test" hasn't caught on that well, and
I believe it is more prescriptive than Perl's system (which is really
just a script that outputs results in a specific format, while
setup.py test is based on unittest).  I think we'd do well to do
*less* vertical integration for testing (though maybe it's fine,
things like the nose entry point split the difference).

In terms of how a setup.py should be structured, and other developer
packaging issues, I think we're getting closer but we *really* need
someone to document the conventions and lessons that are usually just
spread through copy-and-paste and developer feedback.  (In some
communities paster create as had some positive influence, as a kind of
documentation-through-code, but it's adoption is somewhat coincidental
and goes along social lines.)

In terms specifically of the CPAN interactive shell, I don't think
there's any reason to duplicate it, as no one else makes interactive
shell installers; I think it's an accident of history.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

From ziade.tarek at gmail.com  Sat Nov  7 00:54:01 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 00:54:01 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>
Message-ID: <94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>

2009/11/7 Kaelin Colclasure <kaelin at acm.org>:
[...]
> As both a CPAN contributor and a recent PyPI contributor I think I can speak
> to this:
>
> When I packaged my first Perl module for CPAN distribution, there was no
> question about how to go about it. There was a clear recipe and it worked
> without a hitch the first time I tried.
>
> When I recently packaged my first Python module for PyPI I was a bit
> dismayed to learn that easy_install is just one of several ways to
> disseminate Python code. The documentation is fragmented... The stuff I read
> on easy_install basically assumed you already knew all about distutils, and
> I eventually had to reorganize my code from a simple module to a package
> just to get a few non-code files to be pulled in by the install. A less
> persistent would-be contributor might have given up in frustration.
>
> I joined this mailing list instead, and got my package working with some
> kind assistance. :-) But CPAN was just easier to contribute to.

What was in particular, the pieces missing in Distutils (doc+code)
that made you use Setuptools (doc+code) ?

++
Tarek

From ianb at colorstudy.com  Sat Nov  7 00:57:28 2009
From: ianb at colorstudy.com (Ian Bicking)
Date: Fri, 6 Nov 2009 17:57:28 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061550t44acefe6r6f2bc53a8f8aa983@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com> 
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com> 
	<94bdd2610911061550t44acefe6r6f2bc53a8f8aa983@mail.gmail.com>
Message-ID: <b654cd2e0911061557q3139b22cm256b7e22c3da75ad@mail.gmail.com>

On Fri, Nov 6, 2009 at 5:50 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Fri, Nov 6, 2009 at 11:56 PM, Bob Ippolito <bob at redivi.com> wrote:
> [...]
>>
>> It might be the CPAN user interface that makes the difference. Using
>> CPAN you can search for and install applications from an interactive
>> prompt. No web browser needed, just one command line invocation (perl
>> -MCPAN -e shell).
>
> Although, we have an XML-RPC service for ?PyPI, and Yolk for example
> is a client that will let you query PyPI
> through a prompt. I agree it could be enhanced, but it exists. Maybe
> such a client could be added in Distutils ?
>
> Plus, once the work on PEP 390 (or an equivalent one) is over, we will
> be able to list a distribution's dependencies
> on a given target system just by querying PYPI.

It's quite possible that pip will grow some of these features.  At
least search and querying the currently installed packages are planned
in some fashion (someone has indicated the intent to do both these, I
believe).

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

From kaelin at acm.org  Sat Nov  7 01:01:04 2009
From: kaelin at acm.org (Kaelin Colclasure)
Date: Fri, 6 Nov 2009 16:01:04 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>
	<94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>
Message-ID: <79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>


On Nov 6, 2009, at 3:54 PM, Tarek Ziad? wrote:

> 2009/11/7 Kaelin Colclasure <kaelin at acm.org>:
> [...]
>> As both a CPAN contributor and a recent PyPI contributor I think I  
>> can speak
>> to this:
>>
>> When I packaged my first Perl module for CPAN distribution, there  
>> was no
>> question about how to go about it. There was a clear recipe and it  
>> worked
>> without a hitch the first time I tried.
>>
>> When I recently packaged my first Python module for PyPI I was a bit
>> dismayed to learn that easy_install is just one of several ways to
>> disseminate Python code. The documentation is fragmented... The  
>> stuff I read
>> on easy_install basically assumed you already knew all about  
>> distutils, and
>> I eventually had to reorganize my code from a simple module to a  
>> package
>> just to get a few non-code files to be pulled in by the install. A  
>> less
>> persistent would-be contributor might have given up in frustration.
>>
>> I joined this mailing list instead, and got my package working with  
>> some
>> kind assistance. :-) But CPAN was just easier to contribute to.
>
> What was in particular, the pieces missing in Distutils (doc+code)
> that made you use Setuptools (doc+code) ?

Since I bootstrapped the environment I used to learn Python with  
easy_install, I naturally went straight to the easy_install docs to  
learn how to give back. I wasn't even aware of this separate thing  
called distutils until I read about it in the easy_install (err  
setuptools) documentation (sic).

-- Kaelin


From david.lyon at preisshare.net  Sat Nov  7 01:20:23 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 06 Nov 2009 19:20:23 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <675CD838-1DA9-412F-BBDC-9E7ADE18E375@gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>
	<e0ee771d9e908f207e9dba75a4182beb@preisshare.net>
	<675CD838-1DA9-412F-BBDC-9E7ADE18E375@gmail.com>
Message-ID: <4bd010f886b0698383baf73189deb5c0@preisshare.net>

On Fri, 6 Nov 2009 19:15:52 -0500, "ssteinerX at gmail.com"
<ssteinerx at gmail.com> wrote:
>> That's definitely one thing. We have nothing like that yet.
>
http://bitbucket.org/tarek/distribute/issue/88/create-an-equivalent-of-perl-mcpan-e-shell-for

So what... :-)

I have my own shell to too.... 

But my shell is built into a standard setup.py. It has less command line
parameters to
remember than perl. So it's even better than perl.

http://bitbucket.org/djlyon/original-setup.py-with-metadata/src/tip/setup.py

C:\>python setup.py
setup - WARNING - Configuration file setup.conf not found
Usage: setup.py [options]

Options:
  -h, --help            show this help message and exit
  -i NEW_PACKAGE, --install=NEW_PACKAGE
                        Install a Package/Application
  -r PACKAGE_TO_REMOVE, --remove=PACKAGE_TO_REMOVE
                        Install a Package/Application
  -l, --list            List installed Packages
  -d, --debug           Show debug messages
  -u, --user            Use the User Package Directory
  -s, --site            Use the Site Package Directory
  -q, --quiet           don't print status messages to stdout
  -v, --comparepkgsetup
                        Compares package installation against Packages own
                        internal unverified setup.py
  -c, --cleanup         Cleanup workarea

David



From zedobject at gmail.com  Sat Nov  7 01:35:04 2009
From: zedobject at gmail.com (Milind Khadilkar)
Date: Sat, 7 Nov 2009 06:05:04 +0530
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
Message-ID: <116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>

QUOTE
imho,

PyPI + PEP 381 + PEP 381 compliant clients == CPAN
UNQUOTE
In addition to PyPI, PEP381, PEP381 compliant clients, we need a dash of
changed outlook to get CPAN. The Perl community is --  by hearsay --
oriented sympathetically towards people who are not primarily programmers.
In particular, I have heard scientists and finance persons developing web
applications say this when introduced to python.

But the same people were more satisfied with the python stuff when somebody
helped them through the technicalities.

When someone encounters PyPI for the first time, is it easy to learn about
it from the PyPI page itself? Hopeful clicks on the "tutorial" link brings
up a bewildering  CheeseShop Tutorial, without an explanation of what
CheeseShop is. Reading through the options detailed in the page makes sense
only if you already know about them. Definitely not the material for the
beginner or for the technologically-just-adequate.

By the way, the PyPI tutorial is just an example. The python community,
while excellent in the support it provides to the technically competent,
needs to be more friendly towards mere end-users. The attitude should be
more charitable than "GOD helps them, and only them, who help themselves".

Or, rather than between commuinities, is it actually the difference between
Python and Perl as languages themselves?

Regards
Milind Khadilkar


On Sat, Nov 7, 2009 at 5:16 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> On Fri, Nov 6, 2009 at 11:53 PM,  <exarkun at twistedmatrix.com> wrote:
> [..]
> >
> > PyPI *does* let you download packages easily, it does let you upload new
> > ones and organize them without having to worry about integrating them
> into
> > existing packages.  So the features of CPAN that are described as
> desirable
> > are already available in PyPI.
>
> imho,
>
> PyPI + PEP 381 + PEP 381 compliant clients == CPAN
>
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091107/4e5b948c/attachment-0001.htm>

From ziade.tarek at gmail.com  Sat Nov  7 02:06:39 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 02:06:39 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>
	<94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>
	<79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>
Message-ID: <94bdd2610911061706q5bb932d5y94206c2c3483def7@mail.gmail.com>

On Sat, Nov 7, 2009 at 1:01 AM, Kaelin Colclasure <kaelin at acm.org> wrote:
>
> On Nov 6, 2009, at 3:54 PM, Tarek Ziad? wrote:
>
>> 2009/11/7 Kaelin Colclasure <kaelin at acm.org>:
>> [...]
>>>
>>> As both a CPAN contributor and a recent PyPI contributor I think I can
>>> speak
>>> to this:
>>>
>>> When I packaged my first Perl module for CPAN distribution, there was no
>>> question about how to go about it. There was a clear recipe and it worked
>>> without a hitch the first time I tried.
>>>
>>> When I recently packaged my first Python module for PyPI I was a bit
>>> dismayed to learn that easy_install is just one of several ways to
>>> disseminate Python code. The documentation is fragmented... The stuff I
>>> read
>>> on easy_install basically assumed you already knew all about distutils,
>>> and
>>> I eventually had to reorganize my code from a simple module to a package
>>> just to get a few non-code files to be pulled in by the install. A less
>>> persistent would-be contributor might have given up in frustration.
>>>
>>> I joined this mailing list instead, and got my package working with some
>>> kind assistance. :-) But CPAN was just easier to contribute to.
>>
>> What was in particular, the pieces missing in Distutils (doc+code)
>> that made you use Setuptools (doc+code) ?
>
> Since I bootstrapped the environment I used to learn Python with
> easy_install, I naturally went straight to the easy_install docs to learn
> how to give back. I wasn't even aware of this separate thing called
> distutils until I read about it in the easy_install (err setuptools)
> documentation (sic).

Yes, that the great thing *and* the bad thing about Setuptools. It
provided missing features and boostraps
what people needed on the top of Distutils. But in the meantime, it
makes it fuzzy for the end-user that there are two projects, and for
him to understand what's from the Distutils project and what's from
the Setuptools project.

What is clear for us though, is that we need to change Distutils
w.r.t. the Setuptools experience feedback.

For the code part, PEP 376 is one important work.
(http://www.python.org/dev/peps/pep-0376/), it enhances Distutils
with a lot of ideas that were taken from Setuptools (with its author
help), and also adds missing features
we need (like an uninstall command).

For the documentation part I am afraid it will be messy for the end
users trying to package apps in Python *until* all PEPs
have made it into Python.

Although, as Ian Bicking says: we could write today some kind of
all-in-one tutorial so end-users can work out without having to run
after the documenation in several places.

Tarek

From david.lyon at preisshare.net  Sat Nov  7 02:06:09 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 06 Nov 2009 20:06:09 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <b654cd2e0911061550t2621ddcw9fc8e86accb4bf50@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<6a36e7290911061456r297ec847s46b5ad01a57ab83d@mail.gmail.com>
	<e0ee771d9e908f207e9dba75a4182beb@preisshare.net>
	<b654cd2e0911061550t2621ddcw9fc8e86accb4bf50@mail.gmail.com>
Message-ID: <c4f1755bb645ba56d4ca49230c98459f@preisshare.net>

On Fri, 6 Nov 2009 17:50:46 -0600, Ian Bicking <ianb at colorstudy.com> wrote:
> My understanding is that they have all the same problems we do --
> version management, dependencies, isolation, varying quality, etc.

Same problems. But CPAN does it a lot better.

> When I've tried to install things from CPAN it's been surprisingly
> obtuse.  This is not to say there aren't good things, but "we need
> CPAN" doesn't really get to the heart of it.

Works for me - it's like chalk and cheese - but python being the
chalk in this case. Corporate contractor developers know the 
difference. Scientific dudes can obviously spot the difference.

More than anything, it is a culture difference. CPAN is taken
really seriously by those involved. There's so much passion to
get things right and make things work smoothly.

The "heart of it" is the CPAN culture. 

I never experienced the 'we don't support your platform' issue
ever in perl. To CPAN, it is all just perl.

> Testing is a bit off; "setup.py test" hasn't caught on that 
> well..

In perl there is more incentive to provide this.

In Python we don't have any place where this can be automatically
activated on a package. So there is no incentive.

The real world relevance would be some tests to ensure that
the package can work on a number of different platforms and
python versions before being accepted, and that the package
has some tests that can be automatically run.

> In terms of how a setup.py should be structured, and other developer
> packaging issues, I think we're getting closer but we *really* need
> someone to document the conventions and lessons that are usually just
> spread through copy-and-paste and developer feedback...

The individual components that we have are improving and we are
getting closer to having something. But CPAN has an energy level
behind it that we are yet to duplicate.

David


From ziade.tarek at gmail.com  Sat Nov  7 02:19:02 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 02:19:02 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
Message-ID: <94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>

On Sat, Nov 7, 2009 at 1:35 AM, Milind Khadilkar <zedobject at gmail.com> wrote:
> QUOTE
> imho,
>
> PyPI + PEP 381 + PEP 381 compliant clients == CPAN
> UNQUOTE
> In addition to PyPI, PEP381, PEP381 compliant clients, we need a dash of
> changed outlook to get CPAN. The Perl community is --? by hearsay --
> oriented sympathetically towards people who are not primarily programmers.
> In particular, I have heard scientists and finance persons developing web
> applications say this when introduced to python.
>
> But the same people were more satisfied with the python stuff when somebody
> helped them through the technicalities.
>
> When someone encounters PyPI for the first time, is it easy to learn about
> it from the PyPI page itself? Hopeful clicks on the "tutorial" link brings
> up a bewildering? CheeseShop Tutorial, without an explanation of what
> CheeseShop is. Reading through the options detailed in the page makes sense
> only if you already know about them. Definitely not the material for the
> beginner or for the technologically-just-adequate.
>
> By the way, the PyPI tutorial is just an example. The python community,
> while excellent in the support it provides to the technically competent,
> needs to be more friendly towards mere end-users. The attitude should be
> more charitable than "GOD helps them, and only them, who help themselves".
>
> Or, rather than between commuinities, is it actually the difference between
> Python and Perl as languages themselves?
>

I don't know about the difference between Perl and Python, but I do
know this about me
(and this applies to other people in this room I believe)

As a "packaging tools" developer I am totally plunged into the
technical aspects of the project,
and I unfortunately don't have all the time I wish I had to work on this more.
Especially right now because there's a lot of work going on.

So, for the documentation part, I am hiding behind the agile
manifesto, which says,

"working software over comprehensive documentation"

but that's a shame :)

Now, if you do want to help in the documentation so it's better for
newcomers, (in PyPI, Distutils, etc),
I think we would all be happy on such contributions

Tarek

From chris at simplistix.co.uk  Sat Nov  7 02:52:42 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Sat, 07 Nov 2009 01:52:42 +0000
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <4AF4D2EA.4000102@simplistix.co.uk>

This feels like a post relative to the catalog-sig too, or maybe even 
moreso than distutils-sig...

Chris

Guido van Rossum wrote:
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?
> 
> --Guido
> 
> ---------- Forwarded message ----------
> From: dalloliogm <noreply-comment at blogger.com>
> Date: Fri, Nov 6, 2009 at 8:01 AM
> Subject: [Neopythonic] New comment on Python in the Scientific World.
> To: gvanrossum at gmail.com
> 
> 
> dalloliogm has left a new comment on your post "Python in the Scientific World":
> 
> Python is suffering a lot in the scientific word, because it has not a
> CPAN-like repository.
> 
> PyPI is fine, but it is still far from the level of CPAN, CRAN,
> Bioconductor, etc..
> 
> Scientists who use programming usually have a lot of different
> interests and approaches, therefore it is really difficult to write a
> package that can be useful to everyone.
> Other programming language like Perl and R have repository-like
> structure which enable people to download packages easily, and to
> upload new ones and organize them withouth having to worry about
> having to integrate them into existing packages.
> 
> This is what is happening to biopython now: it is a monolitic package
> that it is supposed to work for any bioinformatic problem; but this is
> so general that to accomplish that you would need to add a lot of
> dependencies, to numpy, networkx, suds, any kind of library.
> However, since easy_install is not as ready yet as the counterparts in
> other languages, if the biopython developers add too many
> dependencies, nobody will be able to install it properly, and nobody
> will use it.
> 
> 
> 
> Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM
> 
> 

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From kevin at bud.ca  Sat Nov  7 02:56:25 2009
From: kevin at bud.ca (Kevin Teague)
Date: Fri, 6 Nov 2009 17:56:25 -0800 (PST)
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com> 
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com> 
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com> 
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
Message-ID: <6746db94-976a-45ef-942c-9c35169d634c@v15g2000prn.googlegroups.com>

Since part of my job includes assisting bioinformaticians with
installing packages and providing libraries which they can use, I'll
chime in with a couple of major thrusts behind the statement "CPAN >
PyPI":

1) Automatic dependency handling.

Python still doesn't officially support automatic dependency handling.
The addition of the 'Requires' field specifying import names and not
the project distribution names was unfortunate. Thankfully PJE gave us
'install_requires' and Tarek and others are helping to PEP that field
into the official package metadata.

So now it's possible to install many Python projects today with
automatic dependency handling. However, for scientific python
projects, there has been little movement towards this. Many projects
still have setup.py scripts that are written such that they can only
be installed with manual hand-holding. BioPython is one such example
where running 'python setup.py install' asks a series of interactive
questions on the command prompt ... which I imagine was written that
way because Python didn't have any dependency-handling tools at they
time the setup.py was created, so they essentially wrote a mini-
dependency handler in setup.py. This is a common pattern in many
scientific python projects (and it sucks!). This means that you can't
write a python project, state that you require BioPython, and
automatically install that dependency when your project is installed.

With the CPAN client being ubiquitous in Perl, any author who wrote a
Makefile.PL that behaved like that would be harangued until they fixed
it. But things are moving the right direction in Python, so we'll get
there, just a decade or so later than Perl. But better late than
never!

Client-wise, I think Python is starting to do really well (if you are
fortunate enough to only need to install projects with well written
setup.py scripts). Our bioinformaticians use Buildout or pip
+virtualenv, depending up which tool is better suited for them (e.g.
they just need some libraries to run against or a team collaborating
on a more complex web application or a project that includes some non-
python parts as well). In this area I think CPAN's first mover
advantage is working against them, as inertia and the "CPAN is better"
mantra has meant that tools that allow repeatable installations like
Buildout or Pip either don't exist or are very immature and rarely
used. I'd much rather declaratively state up-front (and store in
version control) what packages and the configuration those packages
may need, then grab a coffee while the tool does it's job, than sit
there guiding an interactive session along like the CPAN client
requires.

Being able to run private PyPI servers is a nice PyPI advantage on the
server side. Bioinformaticians need to share packages where the code
either can't be pulbically released, or they simply don't spend enough
time coding to want to release code which "anyone could see".


2) The python packaging "morass" as a scapegoat

Yeah, so installing, creating and distributing Python packages is a
bit of a morass. The fact that Python has a much larger Standard
Library than Perl means that out-of-the box Python is easier, but
folks using Perl are forced to learn how to install libraries much
earlier on, so they get over that initial hump of "how to deal with
3rd party libraries" a bit earlier. And as many have noted the
documentation surrounding Python packaging is not the greatest.

However, in my experience scientists can be notorious for not really
wanting to think about what code they are running or what version it
is or how they should be deploying it. Their brains are already quite
full trying to deal with the complex scientific problems they're
working on, and there is no glamour or kudos for being "good at
installating software" for a scientist. They often take the same
attitude to automated testing, but at least the scientist who says,
"ah, automated testing is a waste of time and adds little value" may
feel quite a bit of peer pressure to change this opinion. But a
scientist who says, "ah, packaging and installation is a morass and a
waste of time." tends only be met with, "amen! You're better of not
even trying you'll just be wasting your time". So it doesn't take much
for them to throw their arms up and say, "augh! It's hopeless!" and
they have no shortage of scapegoats in Python packaging right now! But
our organization has Perl using-scientists and Ruby using-scientists
and they love to say the same thing, "Library management is too hard
with Perl" or "Ruby gems suck" and "Java CLASSPATH management is a
nightmare".



> Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?

I think the current works-in-progress will eventually get us equal to
or greater than CPAN. But given the history and design of Distutils,
it is sometimes tempting to say, "Ah, let's just chuck the whole thing
out and start with something clean and well-designed." Of course,
pragmatically I don't think such an effort would do much more than
generate a lot of hand-wringing.

However, if we really had the interests of the scientific community in
mind, we'd be thinking about the fact that scientists write code in
Python, Ruby, Perl, R and whatever language happens to be handy and
gets the job done. And from their perspective the fact that if they
happen to use two different programming languages, they need to learn
two different packaging formats and two different package management
tools. If packaging formats and installation formats were standardized
enough that they worked for multiple languages, that would be
something that many might consider jumping ship for. And it would
provide a level playing fields for all F/OSS languages. It's a
terribly ambitious project, but I can dream can't I?

(and yes, we do use system package managers such as RPM+Yum and Dpkg
+apt-get where it makes sense - but those tools and formats are really
more geared towards sysadmins and I've never seen them used in a way
that integrates well with the requirements and workflow of a developer
or scientist)

From setuptools at bugs.python.org  Sat Nov  7 03:32:16 2009
From: setuptools at bugs.python.org (Aaron Gallagher)
Date: Sat, 07 Nov 2009 02:32:16 +0000
Subject: [Distutils] [issue93] SandboxViolation when opening os.devnull
In-Reply-To: <1257561136.31.0.181610180572.issue93@psf.upfronthosting.co.za>
Message-ID: <1257561136.31.0.181610180572.issue93@psf.upfronthosting.co.za>


New submission from Aaron Gallagher <habnabit at gmail.com>:

This is a bug in setuptools 0.6c9. Console output follows.

ender:test habnabit$ easy_install oursql
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
packages/Pyrex/Compiler/Errors.py:17: DeprecationWarning: BaseException.message 
has been deprecated as of Python 2.6
  self.message = message
Searching for oursql
Reading http://pypi.python.org/simple/oursql/
Reading http://launchpad.net/oursql
Best match: oursql 0.1
Downloading http://launchpad.net/oursql/trunk/0.1/+download/oursql-0.1.zip
Processing oursql-0.1.zip
Running oursql-0.1/setup.py -q bdist_egg --dist-dir 
/var/folders/vt/vtsTQLkkFNCqREdN+o2BqE+++TI/-Tmp-/easy_install-H7Ob5k/oursql-
0.1/egg-dist-tmp-xuwDCz
error: Setup script exited with error: SandboxViolation: open('/dev/null', 'w') 
{}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

----------
messages: 459
nosy: habnabit
priority: bug
status: unread
title: SandboxViolation when opening os.devnull

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue93>
_______________________________________________

From david.lyon at preisshare.net  Sat Nov  7 05:07:31 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 06 Nov 2009 23:07:31 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
Message-ID: <b486a619993864ed75dca0bbb94a04f2@preisshare.net>

On Sat, 7 Nov 2009 02:19:02 +0100, Tarek Ziad? <ziade.tarek at gmail.com>
wrote:

> I unfortunately don't have all the time I wish I had to work on this
> more.

We all know that. 

It needs somebody new to take on the challenge.

> Now, if you do want to help in the documentation so it's better for
> newcomers, (in PyPI, Distutils, etc),
> I think we would all be happy on such contributions

The problem isn't only about documentation. We need the following:

 - Overhaul the build system (simplify)

 - implement a simple install system (metadata install?)

 - specify an internal test framework for packages.

 - setup a buildbot testing framework on pypi

After that, we could modify the documentation.

>> The Perl community is --  by hearsay --
>> oriented sympathetically towards people who are not primarily
>> programmers.

That's what we don't have.... and need.

I think this is just too big for distutils alone. Distutils
and the building of packages is only one part of what needs
to change to bring about the changes that Guido is requesting.

David



From david.lyon at preisshare.net  Sat Nov  7 05:38:06 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 06 Nov 2009 23:38:06 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
Message-ID: <1159be1d3410aac7677bb57b997a304c@preisshare.net>

On Sat, 7 Nov 2009 00:46:01 +0100, Tarek Ziad? <ziade.tarek at gmail.com>
wrote:

> PyPI + PEP 381 + PEP 381 compliant clients == CPAN

Not sure about that..

You mustn't have used CPAN much.

And if we try to pretend they are or cobble together a solution it's
very much like adding a turbo-charger to a lada-niva in an attempt
to come up with a competition rally car like a subaru-wrx.

> PyPI + PEP 381 + PEP 381 compliant clients == still imaginary
                                                solution

CPAN == seriously good solution that's been working for years.

Even if you implement PyPI + PEP 381 + PEP 381 tomorrow I promise
you that you won't be anywhere close to CPAN.

It's a much more serious challenge than perphaps you realise..

David


From david at ar.media.kyoto-u.ac.jp  Sat Nov  7 08:49:12 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 07 Nov 2009 16:49:12 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
Message-ID: <4AF52678.6040602@ar.media.kyoto-u.ac.jp>

Hi Guido,

Guido van Rossum wrote:
> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net> wrote:
>   
>> So the packages on CPAN are typically of a higher quality, simply
>> because they've been machine checked. I like that.
>>     
>
> Speaking purely on hearsay, I don't believe that. In fact, I've heard
> plenty of laments about the complete lack of quality control on CPAN.
>   

I cannot speak for CPAN, as I have never used it. But CRAN (which is
CPAN for R) works much better that PyPI today in practice. I am not sure
what exactly makes it work better, but it has the following properties,
both technical and more 'social':
    - R is a niche language, and targets mostly scientists. It is a
smaller community, more focused. They can push solutions more easily.
    - There is an extensive doc on how to develop R extensions (you can
download a 130 pages pdf).
    - R packages are much more constraints: there is a standard source
organization, which makes for a more consistant experience
    - There are regular checks of the packages (all the packages are
daily checked on a build farm on fedora and debian). It also has a
machine to check windows.

http://cran.r-project.org/web/checks/check_summary.html
http://cran.r-project.org/bin/windows/contrib/checkSummaryWin.html

I am obviously quite excited by Snakebite potential here.

Concerning distutils, I think it is important to improve it, but I think
it is inherently  flawed for serious and repeatable packaging. I have
written a quite extensive article on it from my point of view as a
numpy/scipy core developer and release manager
(http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/),
I won't rehearse it here, but basically:
    - distutils is too complex for simple packages, and too inflexible
for complex ones. Adding new features to distutils is a painful
experience. Even autotools with its mix of 100 000 lines autogenerated
shell code, perl, m4 is more pleasant.
    - Most simple packages could be "buildable" from purely declarative
description. This is important IMHO because it means they are simple to
package by OS vendors, and you can more easily automate building and
testing.
    - it is hard to interact with other build/distribution tools, which
is sometimes needed.
    - distutils is too flexible for some things where it should not
be(like specifying the content of sdist tarballs), and that makes it
very difficult to repeat things in different environments.

Contrary to other people, I don't think a successor to distutils is that
hard to develop, especially since good designs in other languages
already exist. It would take time, especially to get a transition story
right, but I think it would be worthwhile.

cheers,

David

From ben+python at benfinney.id.au  Sat Nov  7 09:20:54 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 07 Nov 2009 19:20:54 +1100
Subject: [Distutils] People want CPAN :-)
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
Message-ID: <87iqdm7nkp.fsf@benfinney.id.au>

Guido van Rossum <guido at python.org> writes:

> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net> wrote:
> > So the packages on CPAN are typically of a higher quality, simply
> > because they've been machine checked. I like that.
>
> Speaking purely on hearsay, I don't believe that. In fact, I've heard
> plenty of laments about the complete lack of quality control on CPAN.

That's not inconsistent with CPAN having higher average quality than the
average quality of PyPI packages. It could merely mean that, as a result
of the awareness that CPAN packages are tested on upload, CPAN users
*expect* higher quality, and complain louder than PyPI users when they
find a low-quality package :-)

I think buildbot-style test runs for PyPI packages would raise average
package quality on PyPI.

-- 
 \       ?Everything you read in newspapers is absolutely true, except |
  `\        for that rare story of which you happen to have first-hand |
_o__)                                         knowledge.? ?Erwin Knoll |
Ben Finney


From regebro at gmail.com  Sat Nov  7 09:35:24 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Sat, 7 Nov 2009 09:35:24 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <87iqdm7nkp.fsf@benfinney.id.au>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<87iqdm7nkp.fsf@benfinney.id.au>
Message-ID: <319e029f0911070035i18b4dd01pb5fbfa51f30c55d7@mail.gmail.com>

I love how this created a flurry of speculation on what people who say
"Python doens't have a CPAN" mean. Wouldn't it be easer to *ask* them?
:-)

Just-wondering-ly
-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From regebro at gmail.com  Sat Nov  7 09:39:34 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Sat, 7 Nov 2009 09:39:34 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>
	<94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>
	<79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>
Message-ID: <319e029f0911070039m375c6ce7ub1e1421f629b8866@mail.gmail.com>

2009/11/7 Kaelin Colclasure <kaelin at acm.org>:
> Since I bootstrapped the environment I used to learn Python with
> easy_install, I naturally went straight to the easy_install docs to learn
> how to give back. I wasn't even aware of this separate thing called
> distutils until I read about it in the easy_install (err setuptools)
> documentation (sic).

Yes, this seems to be a reasonably way to realize that packaging is
done with distutils. What was then complicated with the distutils
docs? It's so long ago I did this the first time I don't remember if I
found it difficult.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From glyph at twistedmatrix.com  Sat Nov  7 10:11:57 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Sat, 7 Nov 2009 04:11:57 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <7B0BB044-61BA-4204-9963-069277F95919@twistedmatrix.com>


On Nov 6, 2009, at 12:53 PM, Guido van Rossum wrote:

> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?

In my experience, when users say this, they just mean "I tried  
easy_install and it broke".

PyPI doesn't have some deep, fundamental, architectural issue that  
prevents it from working.  The user experience of it is just buggy.   
Consider the difference between these two pages:

     http://docs.webfaction.com/software/perl.html
     http://docs.webfaction.com/software/python.html

Note that the 'python' page is more than twice as long, lists a ton of  
different installation options, and includes a gigantic  
"troubleshooting" section that apparently isn't necessary for perl.   
Note also that the Perl page is just a series of steps describing how  
to invoke the one installation mechanism, but the Python page is a  
hodgepodge of qualified instructions describing different possible  
mechanisms you can try.  It also appears that webfaction has modified  
the default environment's configuration to make their  
"troubleshooting" section *shorter* than it would have to be for more  
general Python software installation instructions.

The default behavior of most Python installation mechanisms - to my  
knowledge, 'python setup.py install', 'easy_install', and 'pip', will  
all raise exceptions by default on UNIX-y platforms, unless you're  
root.  On Windows (since a higher percentage of the user population  
walks around with admin rights all the time), the default invocations  
described by many project web pages will work if the installation is  
pure-python or if the author remembered to provide a Windows binary  
egg, but a common failure mode is "you don't have a compiler".   
Similarly, on a Mac, you have to have Xcode installed, although Python  
itself works fine if you don't, so it seems like you don't.

Many of these tools *would* work by default with a small amount of  
configuration, a couple of environment variables, and clearer error  
messages that indicate (A) *that* you need to install a C compiler and  
(B) *where* you need to go to get a C compiler.

One project that would help a lot is just a "easy python setup"  
documentation project that describes, as simply as possible, in large  
fonts, how to get a working Python setup that adheres to a few  
conventions.  Just include the 2 lines of .bashrc and explain how to  
add them; don't debate the merits of ~/bin vs. ~/.local/bin vs. ~/opt/ 
bin (although come on, ~/.local/bin/ is _clearly_ the right name for  
it), just pick one for each platform and provide clear step-by-step  
instructions for getting it to work. "put this in your ~/.bashrc:  
<really big PRE tag with shell setup in it>.  restart your shell."   
Anybody who has selected an alternate shell or done some other prior  
configuration will need to adjust their expectations, but we can worry  
about supporting unusual configurations when the community has a good  
answer for the default configuration.  (Although this is a good reason  
to do this as documentation and not attempt to write an  
autoconfigurating tool: an autoconfigurating tool needs to understand  
every possible nuance of the environment, but advanced users can  
notice which parts of the short document might not apply to them.)

I feel like I might be repeating this a bit much, but it's a really  
important point: many of the things I'm talking here are *not* about  
getting the code right as part of a Python tool, but in providing an  
easy, manageable way to integrate with _other_ tools that are outside  
of our collective control as Python package authors: the dynamic  
linker, the shell, the file manager and the C compiler (or lack  
thereof).  By providing a default user-home-directory installation  
location, Python itself is already doing almost as much as it can; if  
easy_install started installing things into that location by default  
*without* any of this bootstrapping documentation (or a very, very  
carefully written tool to do the bootstrapping for you) then importing  
pure Python packages might work great but scripts would be broken and  
any external shared libraries required by Python modules (even if they  
built correctly) would just give you an ImportError.

Once we have some kind of working consensus on this setup, the tools  
can change to support it: easy_install can default to installing  
things in the user's home directory in the case that (A) the  
environment is set up for it and (B) the user isn't an administrator.   
If the environment *isn't* set up, instead of spitting out twelve  
paragraphs explaining how really you should have read-write access to  
the location where your pth files are stored and a link into the  
middle of some dense technical reference documentation, it can just  
say "read this page" and link to something that says "HERE IS A .REG  
FILE THAT WILL FIX YOUR PYTHON.  CLICK ON IT AND DO NOT ASK QUESTIONS."

If such a document existed, it would also be good for PyPI to link to  
it prominently so that if a user comes across a package by way of a  
Google search that ends at PyPI, there is a clear set of instructions  
as to how to get it on their computer.

There's also something that everyone on this list can do today: every  
python package author should have a clean user account, OS install,  
virtual machine, or whatever; some otherwise pristine environment  
where they try to follow their own installation instructions *without*  
any of their usually available tools or tricks for setting up a python  
development environment, and without administrator access.  One thing  
that always strikes me about Python hackers is that we all have  
existing setups to support our own personal style of development, and  
so we are somewhat insulated from the new-user-experience pain.  I  
think if everyone on this list did this regularly on a package with a  
dozen or so dependencies, the situation would rapidly improve  
regardless of my other recommendations :).


From ziade.tarek at gmail.com  Sat Nov  7 12:12:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 12:12:44 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <b486a619993864ed75dca0bbb94a04f2@preisshare.net>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
	<b486a619993864ed75dca0bbb94a04f2@preisshare.net>
Message-ID: <94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>

On Sat, Nov 7, 2009 at 5:07 AM, David Lyon <david.lyon at preisshare.net> wrote:
> On Sat, 7 Nov 2009 02:19:02 +0100, Tarek Ziad? <ziade.tarek at gmail.com>
> wrote:
>
>> I unfortunately don't have all the time I wish I had to work on this
>> more.
>
> We all know that.
>
> It needs somebody new to take on the challenge.

David,

You don't understand at all what is going on I think. You have been
keeping sending passive-agressive
emails in python-dev and in distutils-sig everytime we mention the
work that's going on in Distutils.

This is happening since we said we didn't want to integrate your GUI
system in Python.

When I say "I unfortunately don't have all the time I wish I had to
work on this more", it doesn't mean
that we need to find somebody "new" to take on the challenge, it means
that we can take
more help from more people (and that includes you of course).

Wether you like it or not, I am maintaining Distutils, so I am trying
to coordinate the work that is going
on for that matter.  And this work is currently organized in several
PEPs we are changing everytime
someone provides some new feedback and insights.

You have started some kind of "counter-PEP" to push another build
system, and I really doubt
this is useful in any ways. Discussing new ideas is useful of course,
but changing Distutils for another build system
in the stdlib will not happen, unless its proven to be sucessfully
used by the community for some time.

I will do at this point what other have been doing at python-dev and
just ignore your mails for now on.

Tarek

From p.f.moore at gmail.com  Sat Nov  7 12:27:16 2009
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 7 Nov 2009 11:27:16 +0000
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <7B0BB044-61BA-4204-9963-069277F95919@twistedmatrix.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<7B0BB044-61BA-4204-9963-069277F95919@twistedmatrix.com>
Message-ID: <79990c6b0911070327p60125404t2598d2954359f5c0@mail.gmail.com>

2009/11/7 Glyph Lefkowitz <glyph at twistedmatrix.com>:
> One project that would help a lot is just a "easy python setup"
> documentation project that describes, as simply as possible, in large fonts,
> how to get a working Python setup that adheres to a few conventions.
[...]

+1 Strong conventions would help a lot here, IMHO.

At least as useful would be a similar document explaining how to write
a Python package (contents of setup.py, doc/test subdirectories,
registering on PyPI, what to upload - bdist_wininst, egg, source, ...)
This would make the build/install experience much more consistent for
new packages.

Paul.

From jeff at taupro.com  Sat Nov  7 13:04:15 2009
From: jeff at taupro.com (Jeff Rush)
Date: Sat, 07 Nov 2009 06:04:15 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <1159be1d3410aac7677bb57b997a304c@preisshare.net>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF49FCC.7050403@nextday.fi>	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<1159be1d3410aac7677bb57b997a304c@preisshare.net>
Message-ID: <4AF5623F.2070403@taupro.com>

David Lyon wrote:
> 
> Even if you implement PyPI + PEP 381 + PEP 381 tomorrow I promise
> you that you won't be anywhere close to CPAN.
> 
> It's a much more serious challenge than perphaps you realise..

I keep reading and I keep hearing you and others saying this, but as
someone who has never used CPAN, I'm not seeing the large number of
specific implementable tasks that CPAN clearly has and PyPI clearly does
not.

Without handwaving please, in a technical sense what does CPAN have that
is so wonderful other than the items mentioned so far of:

 - buildbot on the server
 - mirrored. hierarchical servers
 - one widely known and accepted way of doing packaging

If we implement those, if we implement PEP 381, in what specific way
will we still fall *far short* as you suggest?  I can't fix the social
aspects so those are not of use to me.  Only architecture.

BTW at the recent pyTexas regional conference we had a good group
discussion about packaging, with people offering analysis from the Perl,
Java, Ruby and Haskell communities.  It seems each language still only
covers part of the solution, albeit different parts.

-Jeff

From david.lyon at preisshare.net  Sat Nov  7 13:27:55 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 07 Nov 2009 07:27:55 -0500
Subject: [Distutils] People want CPAN :-) / New Build system
In-Reply-To: <4AF52678.6040602@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp>
Message-ID: <0ae5e10c241ce8bf324641f112982d57@preisshare.net>


Hi David,

I have to agree with you on every point.

On Sat, 07 Nov 2009 16:49:12 +0900, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:

> Contrary to other people, I don't think a successor to distutils is that
> hard to develop, especially since good designs in other languages
> already exist. It would take time, especially to get a transition story
> right, but I think it would be worthwhile.

That's just so controversial..

But since Guido has asked, lets just have some good old fashioned pragmatic
software engineering discussion.

Let's toss it around as pure numbers (for what you are suggesting):

 - how many lines of code would it take ?

   (500, 5000, 10000, 20000) ?

 - how much of the existing library could be used ?

   (10%, 25%, 50%,75%, 90%) ?

 - how many months would it take ?

   (3,6,9,12,18,36,72) ?

 - how many people would it require ?

   (1,2,3,5,10) ?

 - Could a PEP be started ? or co-author with somebody
   else ? Are there existing PEPs? Would a PEP for this
   have any chance of success? Could Tarek help with a
   PEP ? He must have some ideas on this..

 - What's the risk ?

I'd suggest we not call it a distutils replacement system.

Lets call it a stripped down simplified package builder.

To me, it's either the existing code from distutils that 
builds what we call a source package. Or it's a new
piece of code that builds a 'standard' python package.

The end result needs to be oriented to a non-technical
audience - so we need to keep it pretty simple - not
too many options.

>From my current understanding, the keys files in any
python package seem to be sources.txt and PKG_INFO.

So it could be possible, imo, to prompt for metadata
on the console, write it to a PKG_INFO, scan for
files and write the sources.txt, and then archive
the whole lot up using the standard library.

That's ultra simplistic... perphaps too much
so.

But imo it's worth doing estimations on.

David




From bradallen137 at gmail.com  Sat Nov  7 13:53:17 2009
From: bradallen137 at gmail.com (Brad Allen)
Date: Sat, 7 Nov 2009 06:53:17 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF5623F.2070403@taupro.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<1159be1d3410aac7677bb57b997a304c@preisshare.net>
	<4AF5623F.2070403@taupro.com>
Message-ID: <4957f1ef0911070453k57f7471fs20eada4982a8c004@mail.gmail.com>

On Sat, Nov 7, 2009 at 6:04 AM, Jeff Rush <jeff at taupro.com> wrote:
> BTW at the recent pyTexas regional conference we had a good group
> discussion about packaging, with people offering analysis from the Perl,
> Java, Ruby and Haskell communities. ?It seems each language still only
> covers part of the solution, albeit different parts.

During that conversation Walker mentioned that he thought that Java's
Maven packaging system had been really successful with the idea of
having a metadata file stored outside each package. That made it
easier for the metadata format to evolve; there is no need change
every package in a repository every time the metadata format is
improved. It also allows convenient inspection of dependencies between
packages without having to download/decompress those packages. (Maven
also stores a copy of the metadata inside the package, but the
authoritative/complete copy is considered the one outside the
package).

I've cc'd Walker hoping he will comment more about the other
advantages of externally stored metadata.

From g.brandl at gmx.net  Sat Nov  7 14:54:09 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 07 Nov 2009 14:54:09 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF52678.6040602@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp>
Message-ID: <hd3qr2$cc2$1@ger.gmane.org>

David Cournapeau schrieb:
> Hi Guido,
> 
> Guido van Rossum wrote:
>> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net> wrote:
>>   
>>> So the packages on CPAN are typically of a higher quality, simply
>>> because they've been machine checked. I like that.
>>>     
>>
>> Speaking purely on hearsay, I don't believe that. In fact, I've heard
>> plenty of laments about the complete lack of quality control on CPAN.

One thing about CPAN (and Haskell's libraries on hackage) that I think
many people see favorably, even though it's only superficial, is the
more-or-less consistent hierarchical naming for the individual packages
(or the contained modules in Haskell).  Compared with that, the Python
package namespace looks untidy.

> I cannot speak for CPAN, as I have never used it. But CRAN (which is
> CPAN for R) works much better that PyPI today in practice. I am not sure
> what exactly makes it work better, but it has the following properties,
> both technical and more 'social':
>     - R is a niche language, and targets mostly scientists. It is a
> smaller community, more focused. They can push solutions more easily.
>     - There is an extensive doc on how to develop R extensions (you can
> download a 130 pages pdf).

Note that the downloadable distutils manual has 94 pages and *should* be
enough to explain the basics of packaging.  It has to be updated, of
course, once the more advanced mechanisms are part of the core.

>     - R packages are much more constraints: there is a standard source
> organization, which makes for a more consistant experience
>     - There are regular checks of the packages (all the packages are
> daily checked on a build farm on fedora and debian). It also has a
> machine to check windows.
> 
> http://cran.r-project.org/web/checks/check_summary.html
> http://cran.r-project.org/bin/windows/contrib/checkSummaryWin.html
> 
> I am obviously quite excited by Snakebite potential here.

Me too.  Though it would be Snakebite + serious sandboxing.

> Concerning distutils, I think it is important to improve it, but I think
> it is inherently  flawed for serious and repeatable packaging. I have
> written a quite extensive article on it from my point of view as a
> numpy/scipy core developer and release manager
> (http://cournape.wordpress.com/2009/04/01/python-packaging-a-few-observations-cabal-for-a-solution/),

What you're saying there about Cabal is exactly my experience.  It is very
nice to work with, and I've not yet seen a conceptual failure.

But we're not far from that, with a static metadata file.

> I won't rehearse it here, but basically:
>     - distutils is too complex for simple packages, and too inflexible
> for complex ones. Adding new features to distutils is a painful
> experience. Even autotools with its mix of 100 000 lines autogenerated
> shell code, perl, m4 is more pleasant.

Really?  I would have assumed that even writing a whole new distutils
command/build step can't be more painful than adding the equivalent to
an autotools-style build system, being Python after all.  However, I've
never done such a thing, so I have to believe you.

Coming back to Cabal, do you know how easy it is to customize its build
steps?

>     - Most simple packages could be "buildable" from purely declarative
> description. This is important IMHO because it means they are simple to
> package by OS vendors, and you can more easily automate building and
> testing.

That's what we're heading towards, I think.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From g.brandl at gmx.net  Sat Nov  7 14:56:41 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 07 Nov 2009 14:56:41 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911061706q5bb932d5y94206c2c3483def7@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF49FCC.7050403@nextday.fi>	<7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>	<94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>	<79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>
	<94bdd2610911061706q5bb932d5y94206c2c3483def7@mail.gmail.com>
Message-ID: <hd3qvq$cc2$2@ger.gmane.org>

Tarek Ziad? schrieb:

> For the documentation part I am afraid it will be messy for the end
> users trying to package apps in Python *until* all PEPs
> have made it into Python.
> 
> Although, as Ian Bicking says: we could write today some kind of
> all-in-one tutorial so end-users can work out without having to run
> after the documenation in several places.

Big +1, and I would be more than willing to include it in the standard
Python documentation, *even* if it mostly describes setuptools/distribute/pip.

When people want to package a library, they *will* look for docs in Python,
and at the moment they only find the distutils reference.  While the latter
is necessary, a more howto-like standard document is much needed.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From pav at iki.fi  Sat Nov  7 13:03:14 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Sat, 07 Nov 2009 14:03:14 +0200
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF49FCC.7050403@nextday.fi>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
Message-ID: <1257595394.4334.23.camel@idol>

la, 2009-11-07 kello 00:14 +0200, Alex Gr?nholm kirjoitti:
[clip: problems in distributing scientific Python packages]   
> I for one did not understand the problem. What does CPAN have that
PyPI 
> doesn't?
> It is natural for packages (distributions, in distutils terms) to
have 
> dependencies on each other. Why is this a problem?

Personally, I've not so much had trouble with PyPi, but with the rest of
the toolchain.

What's special with scientific software is that

- They're usually not pure-Python
- Need support for not only for C, but e.g. Fortran compilers
- It may be necessary to build them on platforms
  where libraries etc. are in non-standard places
- It may be useful to be able to build them with non-gcc compilers
- They may need to ship more data files etc. than plain Python modules
- Python is a newcomer on the scientific scene.
  Not all people want to spend time to spend on installation problems.
  Not all people are experienced Python users.

So it may be more likely that the following things hurt in distributing
these Python modules:

1. Incomplete documentation for distutils.

   For example, where can you find out what `package_data` option
   of setup() wants as the input? What if you have your package in
   src/packagename and data files under data/? What are the paths
   given to it relative to?

   The Distribute documentation is starting to look quite
   reasonable -- so documentation is becoming less of a problem.
   But it seems still to assume that the reader is familiar with
   distutils.

2. Magic.

   For example, what decides which files are included by sdist?
   It appears this depends on (i) what's in the autogenerated
   *.egg-info/SOURCES.txt (ii) whether you are using SVN and are
   using setuptools (iii) possible package_data etc. options,
   (iv) MANIFEST or maybe MANIFEST.in.

   IMHO, the system is too byzantine in ordinary matters,
   which increases the number of things you need to learn.

3. Many layers: distutils, setuptools, numpy.distutils.

   Numpy has its own distutils extensions, primarily for Fortran
   support.

4. Inflexibility.

   The toolchain is a bit inflexible: suppose you need to do
   something "custom" during the build, say, detect sizeof(long double)
   and add a #define to build options according to it. Finding out how
   to do this properly again takes time.

5. Distutils, and tools derived from it have bad failure modes.

   This hurts most with building extension modules. Given the many
   layers, and the fact that the build is driven by software that few
   really understand, it's difficult to understand and fix even simple
   errors encountered.

   Suppose a build fails, because your C or Fortran compiler gets passed
   a flag it doesn't like. How do you work around this?

   Suppose you have a library installed in a non-standard location.
   How do you tell distutils to look for it in the correct place?
   (The answer is to use "build_ext" command separately and pass it -L,
   but this is difficult to find out, as "build" does not accept -L.)


The last one is in practice quite annoying, as given the heterogenous
environments, it's not easy to make your package buildable on all
possible platforms where people might want to use it. When people run
into problems, they are stumped by the complexity of distutils.


The above concerns only building packages -- perhaps there is more to
say also about other parts. Also, I don't really have much experience
with CPAN or CRAN, so I can't say how much Python is better or worse off
here.

-- 
Pauli Virtanen




From david.lyon at preisshare.net  Sat Nov  7 14:22:35 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 07 Nov 2009 08:22:35 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF5623F.2070403@taupro.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF49FCC.7050403@nextday.fi>	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<1159be1d3410aac7677bb57b997a304c@preisshare.net>
	<4AF5623F.2070403@taupro.com>
Message-ID: <d07fe9ab5c8fb1b904546f33fd1720d3@preisshare.net>


Hi Jeff,

On Sat, 07 Nov 2009 06:04:15 -0600, Jeff Rush <jeff at taupro.com> wrote:

> I keep reading and I keep hearing you and others saying this, but as
> someone who has never used CPAN, I'm not seeing the large number of
> specific implementable tasks that CPAN clearly has and PyPI clearly does
> not.

ok. Fair enough. Let me clear up the terminology.

In perl, CPAN means packages + local perl + repository

In python, pypi means repository.

If people complain about pypi versus cpan, the complaint actually
has nothing to do with pypi. What they are actually complaining
about is the package management support (or lack thereof) that
is built into python.

Let me say it a different way...

"Where can I find the pypi module in python?"

Of course there is no pypi module in python.

That's why this whole thing is just so confusing.

If there was such a thing as a pypi module, then users would
automatically use that to access pypi and download and install
packages.

But what python offers is:

 - setuptools
 - distribute
 - pip
 - distutils

In perl, there's only one word for everything, cpan. Even though
it's components exist as libraries and command interfaces on the
local system.

> Without handwaving please, in a technical sense what does CPAN have that
> is so wonderful other than the items mentioned so far of:
> 
>  - buildbot on the server
>  - mirrored. hierarchical servers
>  - one widely known and accepted way of doing packaging

+ Easy to find Modules and command interfaces to interface to the
repository 
  that exist on the local machine. All carrying the matching cpan name.

> If we implement those, if we implement PEP 381, in what specific way
> will we still fall *far short* as you suggest?  

Pypi itself isn't a problem.

Buildbots for packages shouldn't go on pypi itself in any case. They
should go on an entirely seperate server.

So, let me sum up. Pypi itself has no major external problem. It's
short on a package buildbot and local python has no pypi module.

Add a package buildbot and add a pypi package support and we're
pretty much caught up.

David










From david.lyon at preisshare.net  Sat Nov  7 14:38:25 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 07 Nov 2009 08:38:25 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4957f1ef0911070453k57f7471fs20eada4982a8c004@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<1159be1d3410aac7677bb57b997a304c@preisshare.net>
	<4AF5623F.2070403@taupro.com>
	<4957f1ef0911070453k57f7471fs20eada4982a8c004@mail.gmail.com>
Message-ID: <33ade18e1d4153d556ffd368f6586dc1@preisshare.net>

On Sat, 7 Nov 2009 06:53:17 -0600, Brad Allen <bradallen137 at gmail.com>
wrote:
> During that conversation Walker mentioned that he thought that Java's
> Maven packaging system had been really successful with the idea of
> having a metadata file stored outside each package. That made it
> easier for the metadata format to evolve; there is no need change
> every package in a repository every time the metadata format is
> improved. It also allows convenient inspection of dependencies between
> packages without having to download/decompress those packages. (Maven
> also stores a copy of the metadata inside the package, but the
> authoritative/complete copy is considered the one outside the
> package).

So too in python and pypi.

I have used it and it works fine.

http://wiki.python.org/moin/PyPiXmlRpc?action=show&redirect=CheeseShopXmlRpc

The problem is not that the functionality doesn't exist, just that
users are expected to do an xmlrpc call to get it...

(more logical would be to have a pypi module)

David



From ziade.tarek at gmail.com  Sat Nov  7 14:44:21 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 14:44:21 +0100
Subject: [Distutils] A Python Packaging Tutorial
Message-ID: <94bdd2610911070544q4648e73fnc483905ad3da97d3@mail.gmail.com>

As Ian suggested, and since others thinks its a good idea, I suggest
that we start writing a "python packaging guide" using
reStructuredText and DVCS,
explaining how to handle packaging today. (with *all* existing tools)

When it's done, we could add it in python's Doc/ with the help of Georg.

I think Pauli provided a good starting list of what is not obvious for
a new comer.

I propose to start a bitbucket repo, and people can contribute some writing.

Regards,
Tarek

From ssteinerx at gmail.com  Sat Nov  7 15:30:45 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Sat, 7 Nov 2009 09:30:45 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <87iqdm7nkp.fsf@benfinney.id.au>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<87iqdm7nkp.fsf@benfinney.id.au>
Message-ID: <B8D9CAC5-DA6D-47FC-AD1D-1FC388FBBC6B@gmail.com>

On Nov 7, 2009, at 3:20 AM, Ben Finney wrote:

> Guido van Rossum <guido at python.org> writes:
>
>> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon  
>> <david.lyon at preisshare.net> wrote:
>>>
> I think buildbot-style test runs for PyPI packages would raise average
> package quality on PyPI.

Please excuse the cross-post but I wanted to make sure that all these  
"CPAN for Python" discussions got this message and I've lost track of  
which list which part of what discussion had occurred on.

We are currently extending our distutils/Distribute test system to  
include installation of a broad range of packages as part of the pre- 
release process for a future release of Distribute and as part of our  
"smoke" test for distutils/Distribute.    Eventually, the goal is to  
integrate this with our buildbot system but that's a ways off.

Our goal is to install a range of packages and, where practicable,  
actually run and record any errors with the packages' individual test  
suites.

Right now, our "smoke" test only does Twisted and numpy.  We've  
discussed how to collect test results from Twisted trial and we'll be  
working on similar things for other test runners (nose et al.).  For  
Twisted, we're going to install and test both the current release  
version and an svn checkout from trunk.

It would be an extension of that concept to install and test *all*  
packages from PyPI but would, obviously, take considerable horsepower  
(and time) to run such an exhaustive test (especially if we're talking  
about 2.4?, 2.5, 2.6, 2.7, and 3.1+.

Right now I'm extending the configuration file for our smoke test to  
allow for various test runners (e.g. nose, twisted trial, etc.) so we  
can "smoke out" more installation problems and/or failed tests after  
installation.

For the first pass, I'm just focusing on Twisted and trial, then  
numpy, then finding packages that support nose so that I can collect  
the data on what ran, what passed, and what didn't.  I'm planning on  
collecting this all in a database and making some simple API so that  
it can be mined by very simple apps later.

At the point where that infrastructure is in place, we could pretty  
easily mine the data to do all kinds of crazy things people have  
mentioned like:

	* A ranking system of test coverage
	* Complexity analysis
	* Test coverage
	* Run pylint, pyflakes, 2to3, whatever automated measurement tools  
over the code
	* Send test failure messages to maintainers (maybe with opt-in in the  
new meta-data).
	* Whatever!

We're actively working on this right now; anyone who wants to lend a  
hand is welcome to contact me off-list and we can talk about what  
types of things we are needing and where we could use a hand.

All in all, I think this could be a big leap forward for the Python  
distribution ecosystem whether or not we eventually write the PyPan I  
wished for as a new Perl refugee.

Thanks,

S


From david.lyon at preisshare.net  Sat Nov  7 15:46:01 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 07 Nov 2009 09:46:01 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
	<b486a619993864ed75dca0bbb94a04f2@preisshare.net>
	<94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>
Message-ID: <521d7f8ddb2c38bd39a86a1367ed1fe2@preisshare.net>


Hi Tarek,

On Sat, 7 Nov 2009 12:12:44 +0100, Tarek Ziad? <ziade.tarek at gmail.com>
wrote:
> You don't understand at all what is going on I think. 

I guess so.

> .. it means that we can take
> more help from more people (and that includes you of course).

I really do accept that.

What's hard for me to understand exactly is what you will actually allow 
me to do. 

> Wether you like it or not, I am maintaining Distutils, so I am trying
> to coordinate the work that is going on for that matter.  And this work 
> is currently organized in several PEPs we are changing everytime
> someone provides some new feedback and insights.

Why would I mind that you're maintaining distutils ? I don't pretend
to know it as well as you do.

I've been working in software for over twenty years and I've
seen many things but nothing really like distutils. I certainly
couldn't maintain it the way you are doing.

I have real trouble understanding it.

> You have started some kind of "counter-PEP" to push another build
> system, and I really doubt this is useful in any ways. 

You shouldn't take it personally that another person tried to submit
a PEP. So many people helped me with my proposal on the mailing
list - and I owe it to them to at least get their efforts taken
into consideration as a PEP.

In any case, if you feel strongly about this then I can revise
my proposal so that it doesn't look like a proposal for a build
system (it was for a metadata setup - not a build system).

> Discussing new ideas is useful of course, ..

The implication there is you don't want me to submit PEPs because
they somehow counter your work. Well if that's the case I can
be more careful in the future. There's plenty of things
that I can write PEPs for now...

A metadata install was just the first..

And to my knowledge, there still isn't a PEP about that. Right?

David








From lists at zopyx.com  Sat Nov  7 15:57:07 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sat, 07 Nov 2009 15:57:07 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <4AF58AC3.6070207@zopyx.com>

Am 06.11.09 18:53, schrieb Guido van Rossum:
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?
My 2 cents after reading and ignoring the whole thread:

- PyPI provides a good functionality so far

What is annoying about PyPI:

 - some package maintainers have a certain ignorance
   and arrogance by misusing PyPI

   - for uploading packages without or broken metadata
   - for uploading packages of doubtful quality
   - for uploading packages to PyPI as a replacement
     for a private egg server

 - supports too much different versioning schemas. Both
   schema supported by setuptools and the one proposed
   by Tarek in some PEP are totally over-engineered.
   A simple and *enforced* versioning schema is what
   I want to see.

 - no more external hosting of packages. If people
   want their packages listed on Pypi, they should
   be required to upload their packages on PyPI
   (no more issues with non-available external server,
   no more issues with mirroring external servers,
   no more issues with wrong download URLs within
   package metadata)

 - better checks on uploaded packages. A source code
   release should be made using the 'sdist' command.
   We don't need source eggs of a package for
   Python 2.4-2.6 containing Python source code
   only.

The solution for a better PyPI:

 - more checks, more restrictions

 - every package maintainer uploading something to PyPI
   should have a certain attitude that PyPI is a public
   resource where the content should met certain
   quality criteria and where each package has
   a certain responsibility to Python community.

Andreas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091107/df7f9f22/attachment.vcf>

From ziade.tarek at gmail.com  Sat Nov  7 16:08:46 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 16:08:46 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <521d7f8ddb2c38bd39a86a1367ed1fe2@preisshare.net>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
	<b486a619993864ed75dca0bbb94a04f2@preisshare.net>
	<94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>
	<521d7f8ddb2c38bd39a86a1367ed1fe2@preisshare.net>
Message-ID: <94bdd2610911070708m1df68210j6ae8f61ef4a68119@mail.gmail.com>

On Sat, Nov 7, 2009 at 3:46 PM, David Lyon <david.lyon at preisshare.net> wrote:
[..]
>
>> .. it means that we can take
>> more help from more people (and that includes you of course).
>
> I really do accept that.
>
> What's hard for me to understand exactly is what you will actually allow
> me to do.

Gosh. I am not your boss, and I am not telling you what to do. This is
open source,
but this is also a community... You can't just come around and ignore
some parts
of what is being done.

I am just telling you that if you want to help in the static metadata field,
and if your goal is to help *Distutils* improve, you have to work with
what has been started!

[..]
>> You have started some kind of "counter-PEP" to push another build
>> system, and I really doubt this is useful in any ways.
>
> You shouldn't take it personally that another person tried to submit
> a PEP. So many people helped me with my proposal on the mailing
> list - and I owe it to them to at least get their efforts taken
> into consideration as a PEP.
>
> In any case, if you feel strongly about this then I can revise
> my proposal so that it doesn't look like a proposal for a build
> system (it was for a metadata setup - not a build system).
[..]
> The implication there is you don't want me to submit PEPs because
> they somehow counter your work. Well if that's the case I can
> be more careful in the future. There's plenty of things
> that I can write PEPs for now...

Neither Guido, neither Brett neither I are taking this personally. But at some
point, if your goal is to help improving Distutils, you have to work
with what has been
started.

> A metadata install was just the first..
>
> And to my knowledge, there still isn't a PEP about that. Right?

If you work on a PEP that is related to any PEP started in the same
area, I will strongly oppose against adding your PEP
because it is a non-sense. Take and read the existing PEPs and help us
improving them.

We don't add new PEPs for the heck of it. If it's a credit issue, your
name will be added in any PEP you will provide
a valuable help on.

Tarek

From ssteinerx at gmail.com  Sat Nov  7 16:10:59 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Sat, 7 Nov 2009 10:10:59 -0500
Subject: [Distutils] [Python-Dev] People want CPAN :-)
In-Reply-To: <4222a8490911070708j70d06be0j19a598fdddcee459@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<87iqdm7nkp.fsf@benfinney.id.au>
	<B8D9CAC5-DA6D-47FC-AD1D-1FC388FBBC6B@gmail.com>
	<4222a8490911070708j70d06be0j19a598fdddcee459@mail.gmail.com>
Message-ID: <1293335C-B422-472E-ACD8-4E46A756A9C8@gmail.com>


On Nov 7, 2009, at 10:08 AM, Jesse Noller wrote:

> On Sat, Nov 7, 2009 at 9:30 AM, ssteinerX at gmail.com <ssteinerx at gmail.com 
> > wrote:
>> On Nov 7, 2009, at 3:20 AM, Ben Finney wrote:
>>
>>> Guido van Rossum <guido at python.org> writes:
>>>
>>>> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net 
>>>> >
>>>> wrote:
>>>>>
>>> I think buildbot-style test runs for PyPI packages would raise  
>>> average
>>> package quality on PyPI.
>>
>> Please excuse the cross-post but I wanted to make sure that all  
>> these "CPAN
>> for Python" discussions got this message and I've lost track of  
>> which list
>> which part of what discussion had occurred on.
>>
>> We are currently extending our distutils/Distribute test system to  
>> include
>> installation of a broad range of packages as part of the pre- 
>> release process
>> for a future release of Distribute and as part of our "smoke" test  
>> for
>> distutils/Distribute.    Eventually, the goal is to integrate this  
>> with our
>> buildbot system but that's a ways off.
>
> Who is "we"?


We is the people working on Distribute/distutils.

S



From alex.gronholm at nextday.fi  Sat Nov  7 16:13:17 2009
From: alex.gronholm at nextday.fi (=?ISO-8859-1?Q?Alex_Gr=F6nholm?=)
Date: Sat, 07 Nov 2009 17:13:17 +0200
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF58AC3.6070207@zopyx.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
Message-ID: <4AF58E8D.3090404@nextday.fi>

Andreas Jung kirjoitti:
> Am 06.11.09 18:53, schrieb Guido van Rossum:
>   
>> I just found this comment on my blog. People have told me this in
>> person too, so I believe it is real pain (even if the solution may be
>> elusive and the suggested solutions may not work). But I don't know
>> how to improve the world. Is the work on distutils-sig going to be
>> enough? Or do we need some other kind of work in addition? Do we need
>> more than PyPI?
>>     
> My 2 cents after reading and ignoring the whole thread:
>
> - PyPI provides a good functionality so far
>
> What is annoying about PyPI:
>
>  - some package maintainers have a certain ignorance
>    and arrogance by misusing PyPI
>
>    - for uploading packages without or broken metadata
>    - for uploading packages of doubtful quality
>    - for uploading packages to PyPI as a replacement
>      for a private egg server
>
>  - supports too much different versioning schemas. Both
>    schema supported by setuptools and the one proposed
>    by Tarek in some PEP are totally over-engineered.
>    A simple and *enforced* versioning schema is what
>    I want to see.
>
>  - no more external hosting of packages. If people
>    want their packages listed on Pypi, they should
>    be required to upload their packages on PyPI
>    (no more issues with non-available external server,
>    no more issues with mirroring external servers,
>    no more issues with wrong download URLs within
>    package metadata)
>
>  - better checks on uploaded packages. A source code
>    release should be made using the 'sdist' command.
>    We don't need source eggs of a package for
>    Python 2.4-2.6 containing Python source code
>    only.
>   
+5
> The solution for a better PyPI:
>
>  - more checks, more restrictions
>
>  - every package maintainer uploading something to PyPI
>    should have a certain attitude that PyPI is a public
>    resource where the content should met certain
>    quality criteria and where each package has
>    a certain responsibility to Python community.
>
>   
+2
> Andreas
>
>   
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>   


From ziade.tarek at gmail.com  Sat Nov  7 16:37:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 16:37:37 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF58AC3.6070207@zopyx.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
Message-ID: <94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>

On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung <lists at zopyx.com> wrote:
[..]
> ?- supports too much different versioning schemas. Both
> ? schema supported by setuptools and the one proposed
> ? by Tarek in some PEP are totally over-engineered.
> ? A simple and *enforced* versioning schema is what
> ? I want to see.

Unfortunately, as long as we have release candidates, and development versions,
we need a more complex scheme that MAJOR.MINOR.MICRO.

If you think of any ways to simplify PEP 386, please help us,

Last, we can encourage people to use it, but we can't enforced it:
I know people that are happily using dates for their versions, and we
can't forbid
them to push their work on pypi just because of that.

We can try to educate then, but that's their pick at the end I think.

An enterprise PyPI could do enforce it, but not our community PyPI imho

[...]
>
> The solution for a better PyPI:
>
> ?- more checks, more restrictions
>  - every package maintainer uploading something to PyPI
>   should have a certain attitude that PyPI is a public
>   resource where the content should met certain
>   quality criteria and where each package has
>   a certain responsibility to Python community.

More checks would be nice, so we can provide QA rates or something similar.

I don't think we should enforce any policy whatsoever though at PyPI.
We can't force people that upload distributions to
comply with some strict QA rules imho (no binary distro allowed if no
sdist is present for example).

Tarek

From pje at telecommunity.com  Sat Nov  7 16:39:18 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 07 Nov 2009 10:39:18 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF58AC3.6070207@zopyx.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
Message-ID: <20091107153925.D901F3A40DF@sparrow.telecommunity.com>

At 03:57 PM 11/7/2009 +0100, Andreas Jung wrote:
>  - supports too much different versioning schemas. Both
>    schema supported by setuptools and the one proposed
>    by Tarek in some PEP are totally over-engineered.
>    A simple and *enforced* versioning schema is what
>    I want to see.
>
>  - no more external hosting of packages. If people
>    want their packages listed on Pypi, they should
>    be required to upload their packages on PyPI
>    (no more issues with non-available external server,
>    no more issues with mirroring external servers,
>    no more issues with wrong download URLs within
>    package metadata)

Do note that at least these two requirements of yours are likely to 
be opposed by some with at least as much force (if not more so) than 
you are proposing them with.


From jnoller at gmail.com  Sat Nov  7 16:08:55 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Sat, 7 Nov 2009 10:08:55 -0500
Subject: [Distutils] [Python-Dev] People want CPAN :-)
In-Reply-To: <B8D9CAC5-DA6D-47FC-AD1D-1FC388FBBC6B@gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<87iqdm7nkp.fsf@benfinney.id.au>
	<B8D9CAC5-DA6D-47FC-AD1D-1FC388FBBC6B@gmail.com>
Message-ID: <4222a8490911070708j70d06be0j19a598fdddcee459@mail.gmail.com>

On Sat, Nov 7, 2009 at 9:30 AM, ssteinerX at gmail.com <ssteinerx at gmail.com> wrote:
> On Nov 7, 2009, at 3:20 AM, Ben Finney wrote:
>
>> Guido van Rossum <guido at python.org> writes:
>>
>>> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net>
>>> wrote:
>>>>
>> I think buildbot-style test runs for PyPI packages would raise average
>> package quality on PyPI.
>
> Please excuse the cross-post but I wanted to make sure that all these "CPAN
> for Python" discussions got this message and I've lost track of which list
> which part of what discussion had occurred on.
>
> We are currently extending our distutils/Distribute test system to include
> installation of a broad range of packages as part of the pre-release process
> for a future release of Distribute and as part of our "smoke" test for
> distutils/Distribute. ? ?Eventually, the goal is to integrate this with our
> buildbot system but that's a ways off.

Who is "we"?

From lists at zopyx.com  Sat Nov  7 16:56:17 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sat, 07 Nov 2009 16:56:17 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	
	<4AF58AC3.6070207@zopyx.com>
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
Message-ID: <4AF598A1.2000009@zopyx.com>

Am 07.11.09 16:37, schrieb Tarek Ziad?:
> On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung <lists at zopyx.com> wrote:
> [..]
>   
>>  - supports too much different versioning schemas. Both
>>   schema supported by setuptools and the one proposed
>>   by Tarek in some PEP are totally over-engineered.
>>   A simple and *enforced* versioning schema is what
>>   I want to see.
>>     
> Unfortunately, as long as we have release candidates, and development versions,
> we need a more complex scheme that MAJOR.MINOR.MICRO.
>
>   
Do we need/want development on PyPI? At least not me.

MAJOR.MINOR.MICRO.PICO + |a-c]1..N

should be good enough.


> Last, we can encourage people to use it, but we can't enforced it:
>   
Of course we can..
> I know people that are happily using dates for their versions, and we
> can't forbid
> them to push their work on pypi just because of that.
>   
..one must not accept and support a whole zoo of private numbering
schemes. Agree on a common and minimal standard and enforce
the standard.
> We can try to educate then, but that's their pick at the end I think.
>   
Teaching is a good thing...
> An enterprise PyPI could do enforce it, but not our community PyPI imho

"community" does not imply that we can not agree on certain rules and
standards
for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed
package toilet. Python as a quality programming language needs a package
repository with some minimum standards - I completely disagree with
"community"
as a synonym for "we must make everyone happy".

Andreas


-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091107/2fb91785/attachment.vcf>

From david.lyon at preisshare.net  Sat Nov  7 16:56:24 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sat, 07 Nov 2009 10:56:24 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911070708m1df68210j6ae8f61ef4a68119@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
	<b486a619993864ed75dca0bbb94a04f2@preisshare.net>
	<94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>
	<521d7f8ddb2c38bd39a86a1367ed1fe2@preisshare.net>
	<94bdd2610911070708m1df68210j6ae8f61ef4a68119@mail.gmail.com>
Message-ID: <18008f3f68ff4544b22991059670822b@preisshare.net>

On Sat, 7 Nov 2009 16:08:46 +0100, Tarek Ziad? <ziade.tarek at gmail.com>
wrote:

> Gosh. I am not your boss, and I am not telling you what to do. 

otoh you're the boss of distutils. So you can direct people to
work on certain things to help you along. That would have kept
me much quieter with work.

> I am just telling you that if you want to help in the static metadata
> field,  and if your goal is to help *Distutils* improve, you have to 
> work with what has been started!

Well talk about static metadata installation was started at pycon
last year. Nothing happened till I started working on it. And I
believe that I'm the only one working on it now.

> If you work on a PEP that is related to any PEP started in the same
> area, I will strongly oppose against adding your PEP
> because it is a non-sense. 

Same area? as in any packaging PEP? any distutils PEP? There are
PEPs in every area. And PEPs are inter-related.

Does rejection of PEPs on the above grounds apply to everyone? or just
me? Other people seem to have PEPs in related areas.

> We don't add new PEPs for the heck of it. 

And people don't write them for the heck of it either.

If I see something I can do to assist existing PEPs I'll try to
do so.

David






From exarkun at twistedmatrix.com  Sat Nov  7 17:13:11 2009
From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com)
Date: Sat, 07 Nov 2009 16:13:11 -0000
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF598A1.2000009@zopyx.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
	<4AF598A1.2000009@zopyx.com>
Message-ID: <20091107161311.3229.1828836307.divmod.xquotient.294@localhost.localdomain>

On 03:56 pm, lists at zopyx.com wrote:
>Am 07.11.09 16:37, schrieb Tarek Ziad?:
>>On Sat, Nov 7, 2009 at 3:57 PM, Andreas Jung <lists at zopyx.com> wrote:
>>[..]
>>>  - supports too much different versioning schemas. Both
>>>   schema supported by setuptools and the one proposed
>>>   by Tarek in some PEP are totally over-engineered.
>>>   A simple and *enforced* versioning schema is what
>>>   I want to see.
>>Unfortunately, as long as we have release candidates, and development 
>>versions,
>>we need a more complex scheme that MAJOR.MINOR.MICRO.
>Do we need/want development on PyPI? At least not me.
>
>MAJOR.MINOR.MICRO.PICO + |a-c]1..N
>
>should be good enough.

Please be considerate of the time of other people and read the previous 
threads on this list about versioning schemes before proposing a new 
one.  If you have done this, then please make it evident by describing 
how your proposed scheme addresses the problems raised in previous 
discussions, or how those problems are not important/real/whatever.

Thanks.

Jean-Paul

From ziade.tarek at gmail.com  Sat Nov  7 17:13:38 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 7 Nov 2009 17:13:38 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF598A1.2000009@zopyx.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
	<4AF598A1.2000009@zopyx.com>
Message-ID: <94bdd2610911070813k4c3a7c55sf41504f437d4da94@mail.gmail.com>

On Sat, Nov 7, 2009 at 4:56 PM, Andreas Jung <lists at zopyx.com> wrote:
[..]
>
> Do we need/want development on PyPI? At least not me.
>
> MAJOR.MINOR.MICRO.PICO + |a-c]1..N
>
> should be good enough.

PEP 386 is about providing the version scheme so we can compare versions
in Distutils when we want to know if a dependency is met (like what
setuptools does).

So its wider than PyPI : people need to be able to compare development
versions as well.
So for example, zc.buildout can rely on it for your daily work.

[...]
>
> "community" does not imply that we can not agree on certain rules and
> standards
> for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed
> package toilet. Python as a quality programming language needs a package
> repository with some minimum standards - I completely disagree with
> "community"
> as a synonym for "we must make everyone happy".
>

But the philosophy of Python is to provide a multi-paradigm language I think,
without forcing any strong rule like this. (unlike Java I guess)

My mother (sorry that's the example I have in my mind) is using Python
in her university
math /statistics lab, and they don't really care about QA.

But she might push her software at PyPI one day. She won't if its
rejected because
she doesn't follow a version scheme, or push a binary release rather
than a source one.

Its good too have industrial-strength conventions, so we can build
industrial-level applications,
but I think we need to be careful about the ticket entry for PyPI.

Wouldn't be better to set these enforcements in subcommunity like
plone.org where it would
make a lot of sense to enforce QA for plone packages ?
(plone.org has PyPI support)

Tarek

From kaelin at acm.org  Sat Nov  7 17:42:29 2009
From: kaelin at acm.org (Kaelin Colclasure)
Date: Sat, 7 Nov 2009 08:42:29 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <319e029f0911070039m375c6ce7ub1e1421f629b8866@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF49FCC.7050403@nextday.fi>
	<7F7403B4-19C9-4265-BC0D-ACDE42D8FB6D@acm.org>
	<94bdd2610911061554p6f3b4cao5309572fe6ff402b@mail.gmail.com>
	<79C27C9D-82BA-4729-8B12-117D6A5C2B54@acm.org>
	<319e029f0911070039m375c6ce7ub1e1421f629b8866@mail.gmail.com>
Message-ID: <70C4A10E-5F32-4052-A26A-A0D389ECDB5B@acm.org>


On Nov 7, 2009, at 12:39 AM, Lennart Regebro wrote:

> 2009/11/7 Kaelin Colclasure <kaelin at acm.org>:
>> Since I bootstrapped the environment I used to learn Python with
>> easy_install, I naturally went straight to the easy_install docs to  
>> learn
>> how to give back. I wasn't even aware of this separate thing called
>> distutils until I read about it in the easy_install (err setuptools)
>> documentation (sic).
>
> Yes, this seems to be a reasonably way to realize that packaging is
> done with distutils. What was then complicated with the distutils
> docs? It's so long ago I did this the first time I don't remember if I
> found it difficult.


The setuptools docs I read left me with the impression that distutils  
was more about building C extensions and that if my package was pure  
Python source (which it was) I should not need anything more than  
setuptools. And this did prove true, eventually -- but only after I  
restructured my source into a package to be more like setuptools  
wanted it.

What wasted quite a bit of time along the way was that I found other  
examples of setup.py files that were using setuptools but were also  
using some underlying distutils conventions for packaging a single  
module. This approach wasn't mentioned in the setuptools docs but it  
*looked* more like what I was trying to do. But then, it turned out  
there was no simple way to use the single-module spec *and* get my  
static data installed along with the code.

HTH,

-- Kaelin


>
> -- 
> Lennart Regebro: Python, Zope, Plone, Grok
> http://regebro.wordpress.com/
> +33 661 58 14 64


From lists at zopyx.com  Sat Nov  7 17:45:40 2009
From: lists at zopyx.com (Andreas Jung)
Date: Sat, 07 Nov 2009 17:45:40 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911070813k4c3a7c55sf41504f437d4da94@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	
	<4AF58AC3.6070207@zopyx.com>	
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>	
	<4AF598A1.2000009@zopyx.com>
	<94bdd2610911070813k4c3a7c55sf41504f437d4da94@mail.gmail.com>
Message-ID: <4AF5A434.3050805@zopyx.com>

Am 07.11.09 17:13, schrieb Tarek Ziad?:
> On Sat, Nov 7, 2009 at 4:56 PM, Andreas Jung <lists at zopyx.com> wrote:
> [..]
>   
>> Do we need/want development on PyPI? At least not me.
>>
>> MAJOR.MINOR.MICRO.PICO + |a-c]1..N
>>
>> should be good enough.
>>     
> PEP 386 is about providing the version scheme so we can compare versions
> in Distutils when we want to know if a dependency is met (like what
> setuptools does).
>
> So its wider than PyPI : people need to be able to compare development
> versions as well.
> So for example, zc.buildout can rely on it for your daily work.
>   

ACK for a more necessity of one more complex versioning schema in general
(but we don't need to support all variants) but we don't need to support
dev packages on PyPI - that's why a stronger version check should be
enforced.
> [...]
>   
>> "community" does not imply that we can not agree on certain rules and
>> standards
>> for PyPI - otherwise PyPI remains as it sometimes appears - an unflashed
>> package toilet. Python as a quality programming language needs a package
>> repository with some minimum standards - I completely disagree with
>> "community"
>> as a synonym for "we must make everyone happy".
>>
>>     
> But the philosophy of Python is to provide a multi-paradigm language I think,
> without forcing any strong rule like this. (unlike Java I guess)
>
> My mother (sorry that's the example I have in my mind) is using Python
> in her university
> math /statistics lab, and they don't really care about QA.
>
> But she might push her software at PyPI one day. She won't if its
> rejected because
> she doesn't follow a version scheme, or push a binary release rather
> than a source one.
>
>   
I think your mother (and most others) are smart enough to understand
and support a simple versioning schema. Bringing it to the point in
different way: "community" does not mean anarchy. Package maintainers
have a lot of freedom but as said also a responsibility for their
software  - otherwise redeclare PyPI as package t***** (I mentioned the
word already)


> Its good too have industrial-strength conventions, so we can build
> industrial-level applications,
> but I think we need to be careful about the ticket entry for PyPI.
>
> Wouldn't be better to set these enforcements in subcommunity like
> plone.org where it would
> make a lot of sense to enforce QA for plone packages ?
> (plone.org has PyPI support)
>   
I don't care about subcommunities at this point. PyPI is a central
resource to Python. It is essentional for my daily work. It is essential
for me that the packages having reasonable metadata. It is essential
for me that the packages are available all the time. A certain quality
and standards are especially essential to non-professional Python
users and developers - nothing is more frustrating for those
people than dealing with non-functional packages, undocumented packages or
packages of pre-alpha quality.

Andreas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091107/cf20a8d9/attachment.vcf>

From kevin at bud.ca  Sat Nov  7 19:12:28 2009
From: kevin at bud.ca (Kevin Teague)
Date: Sat, 7 Nov 2009 10:12:28 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF58AC3.6070207@zopyx.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
Message-ID: <5ED7E483-7008-4A3C-9EF4-D963726C810F@bud.ca>


On Nov 7, 2009, at 6:57 AM, Andreas Jung wrote:

>
> - no more external hosting of packages. If people
>   want their packages listed on Pypi, they should
>   be required to upload their packages on PyPI
>   (no more issues with non-available external server,
>   no more issues with mirroring external servers,
>   no more issues with wrong download URLs within
>   package metadata)
>


Amen!

Externally hosted packages are a royal PITA and it makes it so much  
harder to automatically install dependencies. This field should at the  
least be marked in the Distutils docs with a big "DEPRECATED: DO NOT  
USE" note.


From kevin at bud.ca  Sun Nov  8 00:18:12 2009
From: kevin at bud.ca (Kevin Teague)
Date: Sat, 7 Nov 2009 15:18:12 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <5ED7E483-7008-4A3C-9EF4-D963726C810F@bud.ca>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<5ED7E483-7008-4A3C-9EF4-D963726C810F@bud.ca>
Message-ID: <8B69462B-9E73-48B5-BD00-C19A1B5886E0@bud.ca>

One other really nice thing about the Perl packaging ecosystem is that  
their standard library is packaged!

If there is a bug found in the Perl standard library, it's trivial to  
upgrade it to a newer release with a bug fix. For example, the recent  
little distutils snafu would have been a lot less painful for the end  
user's if we'd been able to get the bug fix with a simple:

  $ pip install --upgrade distutils

In this respect, from an end user perspective, it really feels like  
you're getting hit with a stick, "Need that distutils fix? Hmm, well,  
OK, but you're going to have wait another month until we do a full  
Python release, and then accept all these other unrelated changes if  
you want that ... "

CPAN even informs you if there's a newer release of itself available,  
and suggests you might like to upgrade:

   There's a new CPAN.pm version (v1.9402) available!
   [Current version is v1.7602]
   You might want to try
     install Bundle::CPAN
     reload cpan
   without quitting the current session. It should be a seamless upgrade
   while we are running...

Buildout has a similar upgrade notification feature, but I don't think  
pip or easy_install does? One nice thing about setuptools (or now  
distribute) not having been merged into the standard library is that  
they're easy to update to newer releases. As we push more of this code  
down into Distutils, we are making it more difficult to get updates :(



From jannis at leidel.info  Sun Nov  8 00:32:02 2009
From: jannis at leidel.info (Jannis Leidel)
Date: Sun, 8 Nov 2009 00:32:02 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <8B69462B-9E73-48B5-BD00-C19A1B5886E0@bud.ca>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<5ED7E483-7008-4A3C-9EF4-D963726C810F@bud.ca>
	<8B69462B-9E73-48B5-BD00-C19A1B5886E0@bud.ca>
Message-ID: <70A36C3D-4B59-44C3-950F-E96B9D39ECD5@leidel.info>

Am 08.11.2009 um 00:18 schrieb Kevin Teague:

> One other really nice thing about the Perl packaging ecosystem is  
> that their standard library is packaged!
>
> If there is a bug found in the Perl standard library, it's trivial  
> to upgrade it to a newer release with a bug fix. For example, the  
> recent little distutils snafu would have been a lot less painful for  
> the end user's if we'd been able to get the bug fix with a simple:
>
> $ pip install --upgrade distutils
>
> In this respect, from an end user perspective, it really feels like  
> you're getting hit with a stick, "Need that distutils fix? Hmm,  
> well, OK, but you're going to have wait another month until we do a  
> full Python release, and then accept all these other unrelated  
> changes if you want that ... "
>
> CPAN even informs you if there's a newer release of itself  
> available, and suggests you might like to upgrade:
>
>  There's a new CPAN.pm version (v1.9402) available!
>  [Current version is v1.7602]
>  You might want to try
>    install Bundle::CPAN
>    reload cpan
>  without quitting the current session. It should be a seamless upgrade
>  while we are running...
>
> Buildout has a similar upgrade notification feature, but I don't  
> think pip or easy_install does? One nice thing about setuptools (or  
> now distribute) not having been merged into the standard library is  
> that they're easy to update to newer releases. As we push more of  
> this code down into Distutils, we are making it more difficult to  
> get updates :(

Oh, intriguing idea, has moving distutils out of Python core been  
considered before?

Jannis

From kevin at bud.ca  Sun Nov  8 01:10:23 2009
From: kevin at bud.ca (Kevin Teague)
Date: Sat, 7 Nov 2009 16:10:23 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <70A36C3D-4B59-44C3-950F-E96B9D39ECD5@leidel.info>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<5ED7E483-7008-4A3C-9EF4-D963726C810F@bud.ca>
	<8B69462B-9E73-48B5-BD00-C19A1B5886E0@bud.ca>
	<70A36C3D-4B59-44C3-950F-E96B9D39ECD5@leidel.info>
Message-ID: <41BE3326-E3F2-43AE-B54F-B4F7DE994C44@bud.ca>


On Nov 7, 2009, at 3:32 PM, Jannis Leidel wrote:

>
> Oh, intriguing idea, has moving distutils out of Python core been  
> considered before?
>

To be clear, I'm not suggesting moving anything in or out of the  
standard library. Just taking what's in the standard library and  
packaging it up, and allowing for releases of these packages to happen  
on PyPI (or for people to more easily include VCS checkouts of these  
packages in their Buildouts or Pip-outs). The main Python interpreter  
download would still include the same standard library packages (or  
whatever the python-dev wants to be in there), and they would still be  
installed in the same way, just with the addition of .egg-info files  
to make them PEP 376 compliant (right now wsgiref in the standard  
library has an .egg-info but it's an exception).

It's been considered before (Chris Withers was recently asking for  
it), and perhaps before that. I was kvetching about this on the stdlib- 
sig last month, where I outlined some of the other problems that not  
having standard packaging for the standard library presents:

http://mail.python.org/pipermail/stdlib-sig/2009-October/000721.html


From ssteinerx at gmail.com  Sun Nov  8 01:23:17 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Sat, 7 Nov 2009 19:23:17 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <41BE3326-E3F2-43AE-B54F-B4F7DE994C44@bud.ca>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<5ED7E483-7008-4A3C-9EF4-D963726C810F@bud.ca>
	<8B69462B-9E73-48B5-BD00-C19A1B5886E0@bud.ca>
	<70A36C3D-4B59-44C3-950F-E96B9D39ECD5@leidel.info>
	<41BE3326-E3F2-43AE-B54F-B4F7DE994C44@bud.ca>
Message-ID: <10D6ACE5-D6DE-4F8C-A937-6A944030C6DA@gmail.com>


On Nov 7, 2009, at 7:10 PM, Kevin Teague wrote:

Being able to update individual modules within the standard library  
would absolutely rock my world and would have removed the urgency from  
2.6.3/4 updates.

Just update what's necessary.  What a concept.

Yes, please.  The sooner the better.

As long as the testing is at least as thorough, and reverting is very  
easy, why _wouldn't_ this be a good idea?

I gotta say, all this talk of CPAN reminds me of when I first started  
using Python and was mystified/mortified at the total lack of  
something similar.  I kind of got used to it but this is bringing back  
the itch in a major way...

S


From david at ar.media.kyoto-u.ac.jp  Sun Nov  8 14:22:07 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 08 Nov 2009 22:22:07 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <hd3qr2$cc2$1@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>	<4AF52678.6040602@ar.media.kyoto-u.ac.jp>
	<hd3qr2$cc2$1@ger.gmane.org>
Message-ID: <4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>

Georg Brandl wrote:
>
> One thing about CPAN (and Haskell's libraries on hackage) that I think
> many people see favorably, even though it's only superficial, is the
> more-or-less consistent hierarchical naming for the individual packages
> (or the contained modules in Haskell).  Compared with that, the Python
> package namespace looks untidy.

That's true, but there is not much we can do on this one, so I did not
mention it.

>
> Note that the downloadable distutils manual has 94 pages and *should* be
> enough to explain the basics of packaging.  It has to be updated, of
> course, once the more advanced mechanisms are part of the core.

The manual is too complicated for simple tasks, and not very useful for
complex
ones. Mostly because distutils does not follow the "only one way to do
things"
mantra. I can help to improve the distutils documentation for the build
part, which is mostly undocumented (things like how to create a new
command to build ctypes extensions, etc...).

>
> Me too.  Though it would be Snakebite + serious sandboxing.

Sandboxing is of course needed, but that's a known problem, and people
have already thought hard about it. The open suse build system, albeit
linux specific, works quite well, for example. For environment
sandboxing, chroot works on all unix I know (including mac os x) -
security is more challenging, I don't have any expertise there. Windows
is more difficult to handle, though (maybe windows people know good
sandboxing solutions outside full-blown vm).

> What you're saying there about Cabal is exactly my experience.  It is very
> nice to work with, and I've not yet seen a conceptual failure.
>
> But we're not far from that, with a static metadata file.

Several people have claimed this so far, but I don't understand why -
could you
expand on this ?  My impression is that the focus is mostly on version
specification and install/build requirements in the static data, but to me
that's a tiny detail. I want something like .cabal files, where you can
specify
documentation, data, source files, etc... Something like what I started to
prototype there:

http://github.com/cournape/toydist/

To take an example you are well familiar with, you can fully describe
sphinx with it, and the conversion is mostly automatic. This is not even
500 LOC. With this kind of design, you can use different build systems
on top of it (there is for example unpublished code in toydist to use a
scons-based build system instead  of distutils as currently done).

>
>> I won't rehearse it here, but basically:
>>     - distutils is too complex for simple packages, and too inflexible
>> for complex ones. Adding new features to distutils is a painful
>> experience. Even autotools with its mix of 100 000 lines autogenerated
>> shell code, perl, m4 is more pleasant.
>
> Really? 

Sure, the perl/shell/awk/m4 mix is painful, but at least the result is
reasonably robust, and can be extended.

>  I would have assumed that even writing a whole new distutils
> command/build step can't be more painful than adding the equivalent to
> an autotools-style build system, being Python after all.  However, I've
> never done such a thing, so I have to believe you.

I expand on that, because I think few people understand the problem
here, and
that's maybe the main source of frustration for core numpy developers as
far as
distutils is concerned. True, writing your own command is easy. But it
has many
failure modes:
 - if you extend an existing command, you have to take care whether you run
   under setuptools or distutils (and soon distribute will make this worse).
   Those commands will not work the same when you run them under paver
either.
 - the division in subcommands is painful, and the abstraction does not make
   much sense IMHO. Recently, I needed to access simple things like library
   filename (foo ->libfoo.a/foo.lib/etc..), install prefix. But those
are not
   accessible to each command. The install prefix was particularly
painful, it
   took me several hours to get it work right with distutils inplace,
develop
   mode on all platforms. All this is trvially easy to get with
autotools, scons or waf.
   Every new feature I needed to add to numpy.distutils was an unpleasant
   experience. I had to read the distutils sources (for every supported
python
   version), run it on several platforms, and got it working by trial an
error.
 - if you want to add a new source file extension, you have to rewrite the
   build_ext or build_src command and  you often cannot reuse the base class
   methods.
 - etc...

Also, the distutils code is horrible: you don't really know what's
public and
what's not, most attributes are added at runtime (and sometimes differ
depending on the platform). Often, you get strange errors with the exception
swallowed, and that happens only on some platforms for some users; in that
case, the only way to debug it is to be able to run their platform. When you
write extensions to distutils, this contributes to the whole unpleasant
experience.

>
> Coming back to Cabal, do you know how easy it is to customize its build
> steps?

No, I don't. I know you have to use makefile/autoconf for complex
packages (for example,
gtk wrapper for haskell does not use cabal AFAIK).

But I think the only thing which matters is to have a basic simple with
which
you can interoperate with. It would not make sense to require from standard
build/packaging system to support fortran or most of what we need in
numpy. For example,
I have written numscons to get away from distutils: it enables the use
of scons for all the
building part, and can build complicated packages such as numpy and
scipy on many platforms (including
windows) but we cannot use it as our main tool because you can't easily
interoperate with distutils (to get sdist, bdist_wininst, etc... working).
A system which would make this possible would already be great - such a
system
would be both simpler and more reliable than current distutils IMHO.

>
> That's what we're heading towards, I think.

Guido wanted to know how scientific python people feel about the whole
situation, and my own impression is that we are going further from what we
need. I don't think anything based on distutils can help us. This is not to
criticize Tarek, PJE and other people's work: I understand that
distutils and
setuptools solve a lot of problems for many people, and I may just be a
minority.

David

From ziade.tarek at gmail.com  Sun Nov  8 19:01:06 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 8 Nov 2009 19:01:06 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp> <hd3qr2$cc2$1@ger.gmane.org>
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>

On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau <david at ar.media.kyoto-u.ac.jp>
[..]
> ?- if you extend an existing command, you have to take care whether you run
> ? under setuptools or distutils (and soon distribute will make this worse).

No, because unlike setuptools, we want to remove the patching that is done on
the Command class and on the DIstribution class, and make Distribute a good
Distutils citizen in that matter.

IOW you won't suffer anymore from what you've described.

> ? Those commands will not work the same when you run them under paver
> either.
> ?- the division in subcommands is painful, and the abstraction does not make
> ? much sense IMHO. Recently, I needed to access simple things like library
> ? filename (foo ->libfoo.a/foo.lib/etc..), install prefix. But those
> are not
> ? accessible to each command.
[...]
>  - if you want to add a new source file extension, you have to rewrite the
>   build_ext or build_src command and  you often cannot reuse the base class
>   methods.
>  - etc...

This is going to be changed because I am currently refactoring the
build_ext command
so such methods are in a base "compiler" class and eventually in util.py

You are welcome to help in this refactoring,

> Also, the distutils code is horrible: you don't really know what's
> public and
> what's not, most attributes are added at runtime (and sometimes differ
> depending on the platform).
> Often, you get strange errors with the exception
> swallowed, and that happens only on some platforms for some users; in that
> case, the only way to debug it is to be able to run their platform. When you
> write extensions to distutils, this contributes to the whole unpleasant
> experience.

I agree the code is not modern, but things are getting changed through
small steps.

Although, I don't buy the "strange errors" part and things getting swallowed :)

[..]
>
> Guido wanted to know how scientific python people feel about the whole
> situation, and my own impression is that we are going further from what we
> need. I don't think anything based on distutils can help us. This is not to
> criticize Tarek, PJE and other people's work: I understand that
> distutils and
> setuptools solve a lot of problems for many people, and I may just be a
> minority.

My opinion is that you've build something else when Distutils was not
evolving anymore. This is not true anymore. It's moving again. And I
think that the work that is going on is heading in the right
direction, even for your use cases imho.

If projects that maintain distutils patched versions push those
patches now in the Python issue tracker, diffing against the current
trunk, and if those patches make sense and are with tests, that's by
far the *easiest* way to help improving Distutils.

And that's the easiest work for me : I'll just review them and commit
them if they do improve Distutils.

Tarek

From floris.bruynooghe at gmail.com  Sun Nov  8 20:19:44 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sun, 8 Nov 2009 19:19:44 +0000
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <18008f3f68ff4544b22991059670822b@preisshare.net>
References: <4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
	<b486a619993864ed75dca0bbb94a04f2@preisshare.net>
	<94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>
	<521d7f8ddb2c38bd39a86a1367ed1fe2@preisshare.net>
	<94bdd2610911070708m1df68210j6ae8f61ef4a68119@mail.gmail.com>
	<18008f3f68ff4544b22991059670822b@preisshare.net>
Message-ID: <20091108191944.GA5380@laurie.devork>

On Sat, Nov 07, 2009 at 10:56:24AM -0500, David Lyon wrote:
> On Sat, 7 Nov 2009 16:08:46 +0100, Tarek Ziad? <ziade.tarek at gmail.com>
> wrote:
> 
> > Gosh. I am not your boss, and I am not telling you what to do. 
> 
> otoh you're the boss of distutils. So you can direct people to
> work on certain things to help you along. That would have kept
> me much quieter with work.

If you have time to burn and need to be told how to spend it: on top
of my head a useful contribution would be to improve the PEP 386
reference implementation for example (I pick that one as I know most
about it from all the PEP proposals currently).  This is a PEP that
should maybe be finished first of the bunch, so that's definitely
useful.  Useful things:

* Read up on the PEP and all the documentation in the reference
  implementation

* Check out previous discussions and make sure nothing is missed.

* Add more test!  The core class could be tested better, and
  suggest_rational_version is massively undertested.  For this you
  need to read up on the old distutils version scheme as well as
  setuptools scheme and build lots of test cases for it.  Also looking
  at all versions on the cheesshop to figure out tricky version
  numbers and test the function.  All the "reasonable" version numbers
  used on PyPI should be tested really.

* Check the history of the implementation.  Maybe there have been
  checkins that correct things in one area but left similar bugs in
  other areas (e.g. a bug fixed in the class but not in
  suggest_rational_version).  If so add tests.

* If you managed to create tests that fail, see if you can figure out
  ways to make them pass.

* Improve the documentation of the reference implementation, after
  you've done many of the above things you'll definitely have found
  some places where it could be improved.

* Create patches out of the above work and submit it to the refrence
  implementation, if they're useful and good they'll get accepted.
  But don't be discouraged if a re-work is asked for initially.


I'm not telling you the above things because I want to be your (or
anyone's) boss and tell other people what to do.  I'm telling you them
as an example of how to contribute to the current work.  The important
thing is that I don't know anything more then you do, I haven't had
secret conversations with a cabal or so (I'm not even on IRC usually).
The only thing I do is read the mailing list and look at proposals I'm
interested in.  Finding and carrying out work like this is
contributing.


Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From david.lyon at preisshare.net  Sun Nov  8 23:03:04 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Sun, 08 Nov 2009 17:03:04 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <20091108191944.GA5380@laurie.devork>
References: <4AF49FCC.7050403@nextday.fi>
	<20091106225332.3229.621319926.divmod.xquotient.264@localhost.localdomain>
	<94bdd2610911061546g32052943x45883f6de62011f9@mail.gmail.com>
	<116b13090911061635j500bc82asd7fa666669f71479@mail.gmail.com>
	<94bdd2610911061719i29e755fcy66e716289cbc74c2@mail.gmail.com>
	<b486a619993864ed75dca0bbb94a04f2@preisshare.net>
	<94bdd2610911070312k24da051fy3e649724cbc3d734@mail.gmail.com>
	<521d7f8ddb2c38bd39a86a1367ed1fe2@preisshare.net>
	<94bdd2610911070708m1df68210j6ae8f61ef4a68119@mail.gmail.com>
	<18008f3f68ff4544b22991059670822b@preisshare.net>
	<20091108191944.GA5380@laurie.devork>
Message-ID: <7375f4367476d299ba7e5936cf0282c3@preisshare.net>

On Sun, 8 Nov 2009 19:19:44 +0000, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
> .. improve the PEP 386
> reference implementation for example (I pick that one as I know most
> about it from all the PEP proposals currently). 

I'm +1 on PEP-386. It makes sense to me.

> This is a PEP that should maybe be finished first of the bunch

PEP-345 is perphaps more important. More depends on that.

I think/hope Guido's post has changed things a little. I agree with
those who say everybody should work together to some PEPs closed off
and I also agree with the push to offer something that is of a
more comparable standard to CPAN.

To get to where Guido is asking for, I think there are some gaps
in the PEP coverage. So we need to cover those bases also.

David







From david at ar.media.kyoto-u.ac.jp  Mon Nov  9 03:03:56 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Nov 2009 11:03:56 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>	
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>	
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp>
	<hd3qr2$cc2$1@ger.gmane.org>	
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
Message-ID: <4AF7788C.5090501@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
> On Sun, Nov 8, 2009 at 2:22 PM, David Cournapeau <david at ar.media.kyoto-u.ac.jp>
> [..]
>>  - if you extend an existing command, you have to take care whether you run
>>   under setuptools or distutils (and soon distribute will make this worse).
>
> No, because unlike setuptools, we want to remove the patching that is done on
> the Command class and on the DIstribution class, and make Distribute a good
> Distutils citizen in that matter.

I don't think that's possible, not without major changes of the
distutils design at least. Right now, in numpy.distutils, we extend the
Distribution class to hold a few additional data. We inherit from
setuptools or distutils depending on whether setuptools has been
imported. The fundamental problem is not so much that setuptools does
monkey-patching, but that you need to inherit to extend. Once you have
several possible packages which inherit independently from those
classes, you have to special case each of them if you want to support
all of them.

>
>
>
> This is going to be changed because I am currently refactoring the
> build_ext command
> so such methods are in a base "compiler" class and eventually in util.py

Refactoring cannot help, unless you fundamentally change how it works. I
gave an example of a simple feature which is hard to do with current
distutils: obtaining the install prefix early in the build scheme. When
you do:

python setup.py install --prefix=foo

the build_* commands don't know prefix, only install knows it. In numpy,
I added various hack each more ugly than the other to fake running the
install command before running the build commands to obtain this
information. I would be surprised if subsequent refactor of distutils
will not break it.

> I agree the code is not modern, but things are getting changed through
> small steps.
>
> Although, I don't buy the "strange errors" part and things getting swallowed :)

I don't understand what's there to buy. Several people reported
distutils errors without any backtrace, though a fair shared of those
were caused by our own extensions.

Concerning changing into small steps: I understand that changing
gradually is often better than starting from scratch, but my own
experience with numpy.distutils and distutils convinced me that there is
not much to save from distutils code. I consider the API, the UI and the
implementation deeply flawed. For me, an  "improved backward compatible"
distutils is an oxymoron. More practically, I think the recent
distutils-related changes in 2.6.3 and 2.6.4  will keep happening. In
numpy.distutils, we depend a lot on internal details of distutils,
because we need things not available in the API, and I don't think we
are alone.

I would mention that numpy is one of the package which has the most
invested in distutils in terms of code (sloccount tells me that
numpy.distutils is 10315 LOC, whereas python 2.6.2 distutils is 10648
LOC), so I am certainly aware of the cost of breaking backward
compatibility.

> My opinion is that you've build something else when Distutils was not
> evolving anymore.

It has nothing to do with distutils evolving. We have had our own
extensions for years (I am a relatively newcommer, but some
numpy.distutils code goes back to 2001), so we could have done pretty
much what we wanted. We now have two build systems: one based on
distutils, one based on scons. Every-time I added a build feature to
numpy, it took me much more time to add it to distutils than to the
scons build. The code needed to support scons build is ~ 2000 lines of
code, is more robust, handle dependencies automatically, support
reliable parallel builds, and is more maintainable.

cheers,

David

From robert.kern at gmail.com  Mon Nov  9 05:28:39 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Sun, 08 Nov 2009 22:28:39 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>		<a21398cfb92acc7d47f8151c249291c4@preisshare.net>		<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>		<4AF52678.6040602@ar.media.kyoto-u.ac.jp>	<hd3qr2$cc2$1@ger.gmane.org>		<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
Message-ID: <hd85pn$4as$2@ger.gmane.org>

David Cournapeau wrote:

> I don't understand what's there to buy. Several people reported
> distutils errors without any backtrace, though a fair shared of those
> were caused by our own extensions.

distutils specifically swallows exceptions and formats them for users by 
default. After all, it is trying to behave like a regular command line program 
that interacts with users, not developers. This is easily overridable for 
developers who are trying debug problems by setting the environment variable 
DISTUTILS_DEBUG=1. This will make distutils just give the traceback.

Is this what you are referring to?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Mon Nov  9 05:51:46 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Nov 2009 13:51:46 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <hd85pn$4as$2@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>		<a21398cfb92acc7d47f8151c249291c4@preisshare.net>		<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>		<4AF52678.6040602@ar.media.kyoto-u.ac.jp>	<hd3qr2$cc2$1@ger.gmane.org>		<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>
Message-ID: <4AF79FE2.20102@ar.media.kyoto-u.ac.jp>

Robert Kern wrote:
> After all, it is trying to behave like a regular command line program
> that interacts with users, not developers. This is easily overridable
> for developers who are trying debug problems by setting the
> environment variable DISTUTILS_DEBUG=1. This will make distutils just
> give the traceback.

I did not know about that option, thanks.

The last example I remember was python 2.6 breaking our mingw support:
the raised exception was an empty message, so you got a "error:" as your
error message when the exception was caught in setup. I am not sure it
is better than a traceback: maybe distutils could separate between
'expected' and 'unexpected' exceptions to make this clearer.

David

From ianb at colorstudy.com  Mon Nov  9 06:28:42 2009
From: ianb at colorstudy.com (Ian Bicking)
Date: Sun, 8 Nov 2009 23:28:42 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <hd85pn$4as$2@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com> 
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com> 
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp> <hd3qr2$cc2$1@ger.gmane.org> 
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com> 
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org>
Message-ID: <b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>

On Sun, Nov 8, 2009 at 10:28 PM, Robert Kern <robert.kern at gmail.com> wrote:
> David Cournapeau wrote:
>
>> I don't understand what's there to buy. Several people reported
>> distutils errors without any backtrace, though a fair shared of those
>> were caused by our own extensions.
>
> distutils specifically swallows exceptions and formats them for users by
> default. After all, it is trying to behave like a regular command line
> program that interacts with users, not developers. This is easily
> overridable for developers who are trying debug problems by setting the
> environment variable DISTUTILS_DEBUG=1. This will make distutils just give
> the traceback.

In the tools I've written, I generally give the traceback if the
verbosity is turned up, and in a case like this (an unexpected
exception -- for distutils that's any exception except a few that
distutils defines) I include "use -v for more".

I think distutils (or Distribute) could do the same.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

From david.lyon at preisshare.net  Mon Nov  9 07:19:38 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 09 Nov 2009 01:19:38 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp>
	<hd3qr2$cc2$1@ger.gmane.org>
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
Message-ID: <69d337f7d047877fc48710cb7d17ac14@preisshare.net>

On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking <ianb at colorstudy.com> wrote:

> In the tools I've written, I generally give the traceback if the
> verbosity is turned up, and in a case like this (an unexpected
> exception -- for distutils that's any exception except a few that
> distutils defines) I include "use -v for more".
> 
> I think distutils (or Distribute) could do the same.

Hi Ian,

I think PIP is quite an accomplishment.

But don't you think that its a big ask to refactor distutils/distribute 
to redo their error messages for package building?

imo the basic problem in setup.py and package building comes from 
ambiguity in the file specification section. Everything else is 
pretty much ok.

With the declarative format David C is hinting at (from Haskell) it
should be a lot easier to collect those files up and put them
in a .tar.gz archive.

I'm just wondering if you've ever thought about the tool chain
on the other side from pip. Like how hard is it to create archive
files of source with all the right stuff. So I guess what's your
take on how close to being right is David C?

Best Regards

David  





From mihamina at gulfsat.mg  Mon Nov  9 07:37:11 2009
From: mihamina at gulfsat.mg (Rakotomandimby Mihamina)
Date: Mon, 09 Nov 2009 09:37:11 +0300
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <4AF7B897.6000107@gulfsat.mg>

11/06/2009 08:53 PM, Guido van Rossum:
> Python is suffering a lot in the scientific word, because it has not a
> CPAN-like repository.

One thing I think is important too:
Perl has "helpers" that help to distribution-package easily.
Python must have too.

For example, debian packaging what is on pypi must be straighforward.
(same for RPM based distribution and other systems)

-- 
       Architecte Informatique chez Blueline/Gulfsat:
    Administration Systeme, Recherche & Developpement
                                    +261 33 11 207 36

From ben+python at benfinney.id.au  Mon Nov  9 07:57:25 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 09 Nov 2009 17:57:25 +1100
Subject: [Distutils] People want CPAN :-)
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7B897.6000107@gulfsat.mg>
Message-ID: <87my2w2nje.fsf@benfinney.id.au>

Rakotomandimby Mihamina <mihamina at gulfsat.mg> writes:

> For example, debian packaging what is on pypi must be straighforward.
> (same for RPM based distribution and other systems)

Packaging for Debian is much more about following the policy, which
deliberately involves human intervention and judgement, and can't very
well be automated. RPM, as I understand it, is more lax and a simple RPM
can be generated quite automatically.

Can you give an example of what you mean:

  * How straightforward do you find performing the Debian packaging for
    a Perl package, and what tools do you use to do it?

  * Would you consider it sufficient for the same (or equivalent)
    process to apply for Debian packaging of a Python package?

-- 
 \       ?If consumers even know there's a DRM, what it is, and how it |
  `\     works, we've already failed.? ?Peter Lee, Disney corporation, |
_o__)                                                             2005 |
Ben Finney


From ben+python at benfinney.id.au  Mon Nov  9 08:09:22 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 09 Nov 2009 18:09:22 +1100
Subject: [Distutils] People want CPAN :-)
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp> <hd3qr2$cc2$1@ger.gmane.org>
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
Message-ID: <87eio82mzh.fsf@benfinney.id.au>

David Lyon <david.lyon at preisshare.net> writes:

> On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking <ianb at colorstudy.com> wrote:
>
> > In the tools I've written, I generally give the traceback if the
> > verbosity is turned up, and in a case like this (an unexpected
> > exception -- for distutils that's any exception except a few that
> > distutils defines) I include "use -v for more".
> > 
> > I think distutils (or Distribute) could do the same.
>
> Hi Ian,
>
> I think PIP is quite an accomplishment.
>
> But don't you think that its a big ask to refactor
> distutils/distribute to redo their error messages for package
> building?

I've just had a read through the code for ?pip?; AFAICT, the ?redo the
error messages for package building? essentially amounts to using the
?logging? module. Is that a big ask?

-- 
 \        ?We have to go forth and crush every world view that doesn't |
  `\                believe in tolerance and free speech.? ?David Brin |
_o__)                                                                  |
Ben Finney


From david at ar.media.kyoto-u.ac.jp  Mon Nov  9 08:06:30 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Nov 2009 16:06:30 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <87my2w2nje.fsf@benfinney.id.au>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF7B897.6000107@gulfsat.mg>
	<87my2w2nje.fsf@benfinney.id.au>
Message-ID: <4AF7BF76.9080709@ar.media.kyoto-u.ac.jp>

Ben Finney wrote:
> Rakotomandimby Mihamina <mihamina at gulfsat.mg> writes:
>
>   
>> For example, debian packaging what is on pypi must be straighforward.
>> (same for RPM based distribution and other systems)
>>     
>
> Packaging for Debian is much more about following the policy, which
> deliberately involves human intervention and judgement, and can't very
> well be automated. RPM, as I understand it, is more lax and a simple RPM
> can be generated quite automatically.
>   

A .deb can also be automated up to a certain level. Even for packages
like numpy, there is not much needed. Also, the official debian policy
is mandatory for adoption into official repositories - but if I need to
push a simple .deb to colleagues for a new package of mine, I don't
want/need to spend that much time. For example, if distutils were to
support the --*dir options of autoconf plus all the related metadata to
tag files accordingly, it would make the task quite simple.

Even for complex package, this would make packaging and package update
simpler for package maintainer. If I look at the debian dir for the
numpy package (in Ubuntu), there is really not much needed: the .install
files to tag which files to install for which package, adding debian
specific files (README.debian and co). Most of it if not all could be
removed with the corresponding distutils support (which would not be
.deb specific in any way, and would help rpm, .pkg and even  windows
packaging as well though .e.g. nsis files).

David


From mihamina at gulfsat.mg  Mon Nov  9 08:26:40 2009
From: mihamina at gulfsat.mg (Rakotomandimby Mihamina)
Date: Mon, 09 Nov 2009 10:26:40 +0300
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <87my2w2nje.fsf@benfinney.id.au>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF7B897.6000107@gulfsat.mg>
	<87my2w2nje.fsf@benfinney.id.au>
Message-ID: <4AF7C430.2060100@gulfsat.mg>

11/09/2009 09:57 AM, Ben Finney:
>> For example, debian packaging what is on pypi must be straighforward.
>> (same for RPM based distribution and other systems)
> Packaging for Debian is much more about following the policy, which
> deliberately involves human intervention and judgement, and can't very
> well be automated.

As far as I know (I might know nothing;)), and as far as I follow
the debian-mentors ML, most of the "problems" is about filesystem hierarchy
Python packages are pretty clear on their FS hierarchy, normalizing it
should not be that hard.
Once it's clear about that, Debian packaging straighforwardness is 80% done.

> Can you give an example of what you mean:
>    * How straightforward do you find performing the Debian packaging for
>      a Perl package, and what tools do you use to do it?
>    * Would you consider it sufficient for the same (or equivalent)
>      process to apply for Debian packaging of a Python package?

Give me just some time to give a try on an examples.

My tries might invalidate my assumptions, but I need to test that
on a bunch of Perl modules to confirm. Not just on 2-3.

-- 
       Architecte Informatique chez Blueline/Gulfsat:
    Administration Systeme, Recherche & Developpement
                                    +261 33 11 207 36

From ianb at colorstudy.com  Mon Nov  9 08:50:25 2009
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 9 Nov 2009 01:50:25 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <87eio82mzh.fsf@benfinney.id.au>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com> 
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp> <hd3qr2$cc2$1@ger.gmane.org> 
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com> 
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org> 
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com> 
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
Message-ID: <b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>

On Mon, Nov 9, 2009 at 1:09 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> David Lyon <david.lyon at preisshare.net> writes:
>
>> On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking <ianb at colorstudy.com> wrote:
>>
>> > In the tools I've written, I generally give the traceback if the
>> > verbosity is turned up, and in a case like this (an unexpected
>> > exception -- for distutils that's any exception except a few that
>> > distutils defines) I include "use -v for more".
>> >
>> > I think distutils (or Distribute) could do the same.
>>
>> Hi Ian,
>>
>> I think PIP is quite an accomplishment.
>>
>> But don't you think that its a big ask to refactor
>> distutils/distribute to redo their error messages for package
>> building?
>
> I've just had a read through the code for ?pip?; AFAICT, the ?redo the
> error messages for package building? essentially amounts to using the
> ?logging? module. Is that a big ask?

pip doesn't use the logging module, it uses its own logger, which is
intended more for managing the output of a command-line program and
not just post-mortem debugging.

I don't think changing distutils to improve error output would be hard
at all.  It looks like there's a line in distutils.core that catches
these exceptions (and doesn't look like it actually catches all
exceptions?), and that can just be fixed.

Another topic that has come up: I do agree subclassing makes it really
hard to have multiple lines of development (specifically something
like Setuptools or Distribute, along with ad hoc development in
setup.py files).  But I think that can be resolved.  Perhaps, for
instance, Distribute can have implementations of commands like build,
that can easily be customized or extended without subclassing (e.g.,
by pre-build or post-build functions).  I'd really be shocked if a
rewrite of distutils was necessary, or even necessary to simplify
things.


-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

From david at ar.media.kyoto-u.ac.jp  Mon Nov  9 09:17:35 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 09 Nov 2009 17:17:35 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF52678.6040602@ar.media.kyoto-u.ac.jp>
	<hd3qr2$cc2$1@ger.gmane.org>
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
Message-ID: <4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>

Ian Bicking wrote:
> I don't think changing distutils to improve error output would be hard
> at all.  It looks like there's a line in distutils.core that catches
> these exceptions (and doesn't look like it actually catches all
> exceptions?), and that can just be fixed.

I agree this is one of the thing which can be improved without
unintended consequences.

>
> Another topic that has come up: I do agree subclassing makes it really
> hard to have multiple lines of development (specifically something
> like Setuptools or Distribute, along with ad hoc development in
> setup.py files).  But I think that can be resolved.  Perhaps, for
> instance, Distribute can have implementations of commands like build,
> that can easily be customized or extended without subclassing (e.g.,
> by pre-build or post-build functions).

In numpy's case, we subclass the Distribution class mainly to add new
data which are shared within commands (for example, to build pure C
libraries made available to 3rd parties, or to keep track of the scons
scripts).

I have myself tried the pre/post hooks for commands (for
numpy.distutils), but I did not go very far: the problem was almost
always linked to commands which need to share knowledge from each other.
OTOH, I tried this when I started pocking into numpy.distutils 2 years
ago, maybe I could go further today.

Some things are fixable in distutils: for example, to build things, you
should be able to get rid of the imperative operations, and have instead
of registry of extension -> action (ala scons/waf). This would make
adding new tools (cython, assembly, etc...) easier, as you would add
tools through a registry instead of overriding build_ext as currently
done by e.g. numpy.distutils or cython.distutils. Doing so while keeping
backward compatibility would be hard, though.

>   I'd really be shocked if a
> rewrite of distutils was necessary, or even necessary to simplify
> things.

That's the opposite of my own experience. I think I have given several
reasonable examples of shortcomings of distutils: I would be glad to
hear that each of them has a simple solution which is
backward-compatible in distutils.

But another way to look at it is to ask: What is there in distutils that
you would consider important to keep ? It is important that people who
maintain packages do not have to rewrite their setup to a new
hypothetical system: asking thousand of developers to rewrite their
setup would be insane. But this can be done without being tied to
distutils API.

David


From ozarow at gmail.com  Mon Nov  9 16:48:15 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Mon, 9 Nov 2009 16:48:15 +0100
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <op.u2xx3kxx1q4jlt@whymac.activestate.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
Message-ID: <20091109154815.GJ2966@piotro.eu>

[Sridhar Ratnakumar, 2009-11-05]
> On Thu, 05 Nov 2009 13:03:45 -0800, Tarek Ziad? <ziade.tarek at gmail.com>  
> wrote:
>
>> If I understand correctly, ActiveState has its own releases of
>> python-ldap repackaged in pypm files,
>> and has several releases per python-ldap versions.
>
> Correct. Much like what Linux distributions (like debian) do.

Debian revision is used only in .deb, .egg-info or __version__ is not
changed.
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From sridharr at activestate.com  Mon Nov  9 22:32:55 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Mon, 09 Nov 2009 13:32:55 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
Message-ID: <op.u25cs5zb1q4jlt@whymac.activestate.com>

On Sat, 07 Nov 2009 07:37:37 -0800, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

>>
>> The solution for a better PyPI:
>>
>>  - more checks, more restrictions
>>  - every package maintainer uploading something to PyPI
>>   should have a certain attitude that PyPI is a public
>>   resource where the content should met certain
>>   quality criteria and where each package has
>>   a certain responsibility to Python community.
> More checks would be nice, so we can provide QA rates or something  
> similar.
> I don't think we should enforce any policy whatsoever though at PyPI.
> We can't force people that upload distributions to
> comply with some strict QA rules imho (no binary distro allowed if no
> sdist is present for example).

I suggest that we check for valid metadata on the uploaded sdists at the  
least. If you visit http://pypm.activestate.com/ - most failed packages  
are due to the fact the sdist uploaded by the author misses certain files  
such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself.

Without such quality policing, I can't see how tools like pip/easy_install  
could even install the package (let alone doing it in an user-friendly  
way).

-srid

From sridharr at activestate.com  Mon Nov  9 22:45:29 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Mon, 09 Nov 2009 13:45:29 -0800
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <20091109154815.GJ2966@piotro.eu>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<20091109154815.GJ2966@piotro.eu>
Message-ID: <op.u25dd3j51q4jlt@whymac.activestate.com>

On Mon, 09 Nov 2009 07:48:15 -0800, Piotr Ozarowski <ozarow at gmail.com>  
wrote:

> [Sridhar Ratnakumar, 2009-11-05]
>> On Thu, 05 Nov 2009 13:03:45 -0800, Tarek Ziad? <ziade.tarek at gmail.com>
>> wrote:
>>
>>> If I understand correctly, ActiveState has its own releases of
>>> python-ldap repackaged in pypm files,
>>> and has several releases per python-ldap versions.
>>
>> Correct. Much like what Linux distributions (like debian) do.
> Debian revision is used only in .deb, .egg-info or __version__ is not
> changed.

Same with PyPM. Which is why release numbers need to be stored elsewhere.  
Given the current design/proposals - what makes sense to me now is to  
store it in .egg-info/PyPM.txt.

-srid

From david.lyon at preisshare.net  Mon Nov  9 22:48:42 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 09 Nov 2009 16:48:42 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <op.u25cs5zb1q4jlt@whymac.activestate.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
	<op.u25cs5zb1q4jlt@whymac.activestate.com>
Message-ID: <eb4e09a2e847817ad943bb938547a061@preisshare.net>

On Mon, 09 Nov 2009 13:32:55 -0800, "Sridhar Ratnakumar"
<sridharr at activestate.com> wrote:

> I suggest that we check for valid metadata on the uploaded sdists at the 

> least. If you visit http://pypm.activestate.com/ - most failed packages  
> are due to the fact the sdist uploaded by the author misses certain files
 
> such as README.txt (that is read by setup.py) or setup.py/PKG-INFO
itself.

That may be out of the scope of distutils. Not sure.

CPAN does do checks and we've taken it upon ourselves to start writing
something similar. Our target is to set up to run daily and package
validate
run on a server farm.

It's located at : 

    http://bitbucket.org/djlyon/pypi-package-testbot/

> Without such quality policing, I can't see how tools like
pip/easy_install 
> could even install the package (let alone doing it in an user-friendly  
> way).

It's a community issue, sure. And it affects everyone when something
breaks because of some very minor fault.

David


From david.lyon at preisshare.net  Mon Nov  9 23:28:18 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 09 Nov 2009 17:28:18 -0500
Subject: [Distutils]
	=?utf-8?q?PEP_376_--_support_for_release_versions=3F?=
In-Reply-To: <op.u25dd3j51q4jlt@whymac.activestate.com>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>
	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>
	<op.u2xx3kxx1q4jlt@whymac.activestate.com>
	<20091109154815.GJ2966@piotro.eu>
	<op.u25dd3j51q4jlt@whymac.activestate.com>
Message-ID: <aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>

On Mon, 09 Nov 2009 13:45:29 -0800, "Sridhar Ratnakumar"
<sridharr at activestate.com> wrote:
> Same with PyPM. Which is why release numbers need to be stored elsewhere.
 
> Given the current design/proposals - what makes sense to me now is to  
> store it in .egg-info/PyPM.txt.

But release numbers are already stored in the PKG_INFO metadata
file. Why store it somewhere else?

David

From robert.kern at gmail.com  Mon Nov  9 23:39:40 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 09 Nov 2009 16:39:40 -0600
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>	<20091109154815.GJ2966@piotro.eu>	<op.u25dd3j51q4jlt@whymac.activestate.com>
	<aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>
Message-ID: <hda5nc$gos$1@ger.gmane.org>

On 2009-11-09 16:28 PM, David Lyon wrote:
> On Mon, 09 Nov 2009 13:45:29 -0800, "Sridhar Ratnakumar"
> <sridharr at activestate.com>  wrote:
>> Same with PyPM. Which is why release numbers need to be stored elsewhere.
>
>> Given the current design/proposals - what makes sense to me now is to
>> store it in .egg-info/PyPM.txt.
>
> But release numbers are already stored in the PKG_INFO metadata
> file. Why store it somewhere else?

No, they're not.

http://www.python.org/dev/peps/pep-0241/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ziade.tarek at gmail.com  Mon Nov  9 23:42:58 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 9 Nov 2009 23:42:58 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <op.u25cs5zb1q4jlt@whymac.activestate.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF58AC3.6070207@zopyx.com>
	<94bdd2610911070737o677ac572i9ac08316c1ffe5f4@mail.gmail.com>
	<op.u25cs5zb1q4jlt@whymac.activestate.com>
Message-ID: <94bdd2610911091442g86b313h7e0aec6f172b71c1@mail.gmail.com>

> I suggest that we check for valid metadata on the uploaded sdists at the least. If you visit
> http://pypm.activestate.com/ - most failed packages are due to the fact the sdist
> uploaded by the author misses certain files such as README.txt (that is read by setup.py)
> or setup.py/PKG-INFO itself.

Unfortunately we can't run arbitrary code on PyPI. So if someone ships
a broken setup.py, there's nothing much we can do unless we are able
to run it in some kind of jail.
Some work was started with Steve Steiner on that topic, but we're
using a buildbot.
It's still experimental because running an arbitrary setup.py can fail
for many reasons.

Another thing: once PEP 345 has the required changes (having metadata
fields with platform conditions) we will be able to do some checks
without having to run any code
for any field located in PKG-INFO

In any case, I am still not convinced that these checks should be
forced on PyPI side when the sdist is uploaded. I see this as a QA
rating, because even if a project's setup.py is great, other things
can be wrong in the project's code itself.

Tarek




On 11/9/09, Sridhar Ratnakumar <sridharr at activestate.com> wrote:
> On Sat, 07 Nov 2009 07:37:37 -0800, Tarek Ziad? <ziade.tarek at gmail.com>
> wrote:
>
>>>
>>> The solution for a better PyPI:
>>>
>>>  - more checks, more restrictions
>>>  - every package maintainer uploading something to PyPI
>>>   should have a certain attitude that PyPI is a public
>>>   resource where the content should met certain
>>>   quality criteria and where each package has
>>>   a certain responsibility to Python community.
>> More checks would be nice, so we can provide QA rates or something
>> similar.
>> I don't think we should enforce any policy whatsoever though at PyPI.
>> We can't force people that upload distributions to
>> comply with some strict QA rules imho (no binary distro allowed if no
>> sdist is present for example).
>
> I suggest that we check for valid metadata on the uploaded sdists at the
> least. If you visit http://pypm.activestate.com/ - most failed packages
> are due to the fact the sdist uploaded by the author misses certain files
> such as README.txt (that is read by setup.py) or setup.py/PKG-INFO itself.
>
> Without such quality policing, I can't see how tools like pip/easy_install
> could even install the package (let alone doing it in an user-friendly
> way).
>
> -srid
>


-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From ziade.tarek at gmail.com  Mon Nov  9 23:51:10 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 9 Nov 2009 23:51:10 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>

On 11/9/09, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
[...]
> Some things are fixable in distutils: for example, to build things, you
> should be able to get rid of the imperative operations, and have instead
> of registry of extension -> action (ala scons/waf).

What is a registery of extension exactly ? Distutils let you register your own
commands, you can use through the CLI.

Can you provide more details ?


>>   I'd really be shocked if a
>> rewrite of distutils was necessary, or even necessary to simplify
>> things.
>
> That's the opposite of my own experience. I think I have given several
> reasonable examples of shortcomings of distutils: I would be glad to
> hear that each of them has a simple solution which is
> backward-compatible in distutils.

I am in for making Distutils evolve, but I need very precise real
world use cases
not saying that Distutils shouldn't do imperative operations).

Last, I am not sure why you want only backward-compatible changes in distutils.

There's no plan to keep backward-compatibility if breaking it makes
DIstutils better. We will have pending deprecation warnings, that's
all.

Tarek

From david.lyon at preisshare.net  Mon Nov  9 23:53:29 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 09 Nov 2009 17:53:29 -0500
Subject: [Distutils]
	=?utf-8?q?PEP_376_--_support_for_release_versions=3F?=
In-Reply-To: <hda5nc$gos$1@ger.gmane.org>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>	<20091109154815.GJ2966@piotro.eu>	<op.u25dd3j51q4jlt@whymac.activestate.com>
	<aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>
	<hda5nc$gos$1@ger.gmane.org>
Message-ID: <545fee41121d4422fb80f62a983f3e0b@preisshare.net>

On Mon, 09 Nov 2009 16:39:40 -0600, Robert Kern <robert.kern at gmail.com>
wrote:
>> But release numbers are already stored in the PKG_INFO metadata
>> file. Why store it somewhere else?
> 
> No, they're not.
> 
> http://www.python.org/dev/peps/pep-0241/

Ok, well they're part of http://www.python.org/dev/peps/pep-0345/

I'll revise my statement back to saying that doing it according
to PEP-345 makes the most sense imho. 

PEP-376 specifies the simplest way of storing a version number
in a package that I can see.

David


From robert.kern at gmail.com  Tue Nov 10 00:01:53 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 09 Nov 2009 17:01:53 -0600
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <545fee41121d4422fb80f62a983f3e0b@preisshare.net>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>	<20091109154815.GJ2966@piotro.eu>	<op.u25dd3j51q4jlt@whymac.activestate.com>	<aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>	<hda5nc$gos$1@ger.gmane.org>
	<545fee41121d4422fb80f62a983f3e0b@preisshare.net>
Message-ID: <hda711$kst$1@ger.gmane.org>

On 2009-11-09 16:53 PM, David Lyon wrote:
> On Mon, 09 Nov 2009 16:39:40 -0600, Robert Kern<robert.kern at gmail.com>
> wrote:
>>> But release numbers are already stored in the PKG_INFO metadata
>>> file. Why store it somewhere else?
>>
>> No, they're not.
>>
>> http://www.python.org/dev/peps/pep-0241/
>
> Ok, well they're part of http://www.python.org/dev/peps/pep-0345/
>
> I'll revise my statement back to saying that doing it according
> to PEP-345 makes the most sense imho.

Where? I don't see it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From david.lyon at preisshare.net  Tue Nov 10 00:17:41 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 09 Nov 2009 18:17:41 -0500
Subject: [Distutils]
	=?utf-8?q?PEP_376_--_support_for_release_versions=3F?=
In-Reply-To: <hda711$kst$1@ger.gmane.org>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>	<20091109154815.GJ2966@piotro.eu>	<op.u25dd3j51q4jlt@whymac.activestate.com>	<aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>	<hda5nc$gos$1@ger.gmane.org>
	<545fee41121d4422fb80f62a983f3e0b@preisshare.net>
	<hda711$kst$1@ger.gmane.org>
Message-ID: <29d760255bffc2c8d6c7d9869a1511c6@preisshare.net>

On Mon, 09 Nov 2009 17:01:53 -0600, Robert Kern <robert.kern at gmail.com>
wrote:
>> Ok, well they're part of http://www.python.org/dev/peps/pep-0345/
>>
>> I'll revise my statement back to saying that doing it according
>> to PEP-345 makes the most sense imho.
> 
> Where? I don't see it.

Right at the top, 3rd field, after Metadata-Version and Name:

Version

    A string containing the package's version number. This field must be in
the format specified in PEP 386 [3].

    Example:

    Version: 1.0a2



From robert.kern at gmail.com  Tue Nov 10 00:25:45 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 09 Nov 2009 17:25:45 -0600
Subject: [Distutils] PEP 376 -- support for release versions?
In-Reply-To: <29d760255bffc2c8d6c7d9869a1511c6@preisshare.net>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>	<20091109154815.GJ2966@piotro.eu>	<op.u25dd3j51q4jlt@whymac.activestate.com>	<aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>	<hda5nc$gos$1@ger.gmane.org>	<545fee41121d4422fb80f62a983f3e0b@preisshare.net>	<hda711$kst$1@ger.gmane.org>
	<29d760255bffc2c8d6c7d9869a1511c6@preisshare.net>
Message-ID: <hda8dq$obe$1@ger.gmane.org>

On 2009-11-09 17:17 PM, David Lyon wrote:
> On Mon, 09 Nov 2009 17:01:53 -0600, Robert Kern<robert.kern at gmail.com>
> wrote:
>>> Ok, well they're part of http://www.python.org/dev/peps/pep-0345/
>>>
>>> I'll revise my statement back to saying that doing it according
>>> to PEP-345 makes the most sense imho.
>>
>> Where? I don't see it.
>
> Right at the top, 3rd field, after Metadata-Version and Name:
>
> Version
>
>      A string containing the package's version number. This field must be in
> the format specified in PEP 386 [3].
>
>      Example:
>
>      Version: 1.0a2

That's not what we're talking about. That is the version of the upstream code 
itself. We're talking about release numbers for the binary package made by third 
parties like ActiveState or Enthought. As we fix bugs in our build process for 
each package, we may need to release multiple binary packages for each main 
version of the upstream code. Please read the rest of the thread for the reasons 
why we need to do this and why we should not simply munge the main package 
version to do this.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From david.lyon at preisshare.net  Tue Nov 10 01:43:05 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Mon, 09 Nov 2009 19:43:05 -0500
Subject: [Distutils]
	=?utf-8?q?PEP_376_--_support_for_release_versions=3F?=
In-Reply-To: <hda8dq$obe$1@ger.gmane.org>
References: <op.u2xvbld41q4jlt@whymac.activestate.com>	<94bdd2610911051303n2d10bfebtab37dcb7b81419ec@mail.gmail.com>	<op.u2xx3kxx1q4jlt@whymac.activestate.com>	<20091109154815.GJ2966@piotro.eu>	<op.u25dd3j51q4jlt@whymac.activestate.com>	<aafa8256c991a6877a7ebd8d25dd5141@preisshare.net>	<hda5nc$gos$1@ger.gmane.org>	<545fee41121d4422fb80f62a983f3e0b@preisshare.net>	<hda711$kst$1@ger.gmane.org>
	<29d760255bffc2c8d6c7d9869a1511c6@preisshare.net>
	<hda8dq$obe$1@ger.gmane.org>
Message-ID: <4c6d40b8b7b8d720ab98de89153de0d1@preisshare.net>

On Mon, 09 Nov 2009 17:25:45 -0600, Robert Kern <robert.kern at gmail.com>
wrote:

> We're talking about release numbers for the binary package made by
> third parties like ActiveState or Enthought. As we fix bugs in our 
> build process for each package, we may need to release multiple 
> binary packages for each main

Thanks for clarifying.

Ok, well if it hidden way inside the package where nobody really
bothers to look I guess that it won't make much difference.

I have to say that my opinion is that we should be cleaning
up the crap that is inside packages and not adding to it.

Presently we have all sorts of duplication of information in
there, much of which isn't used.

Lets look:

 1) dependency_links : imo obsolete, usually left empty

 2) requires : imo made obsolete soon by PEP-345

 3) sources : Works fine

 4) top_level : wt.. is this? What program uses this?

 5) not-zip-safe : couldn't this be a setting in setup.cfg?

 6) PKG-INFO : The metadata file itself

So from the above list, 4 out 6 files serve no real purpose.

I'm actually ok with new files and directories created after
installation. But that should be made a little clearer in the 
PEP. It's ambiguous as to whether the files are carried in the 
package or created at installation time.

David




From cool-rr at cool-rr.com  Tue Nov 10 02:39:09 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Tue, 10 Nov 2009 01:39:09 +0000 (UTC)
Subject: [Distutils] People want CPAN
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <loom.20091110T023324-84@post.gmane.org>

Guido van Rossum <guido <at> python.org> writes:

> 
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?
> 
> --Guido


I didn't see this mentioned in the other comments: Sometimes
easy_install just gets stuck with an IncompleteRead error or
something like that. (I may be butchering the error's name, 
sorry.) Then some part of the html of the page gets printed
up to a seemingly arbitrary point, and the install fails.
I'm not sure what causes that, but I've seen it happening at
random for at least a couple of months. Sometimes it works,
sometimes it fails with the incomplete read thing, and then
if you try again it will fail at the same point of the html
page. Try it a few hours later, it might work then. Not nice.

(I've seen it happen on both Linux and Windows.)

Ram.



From setuptools at bugs.python.org  Tue Nov 10 02:41:41 2009
From: setuptools at bugs.python.org (jerry)
Date: Tue, 10 Nov 2009 01:41:41 +0000
Subject: [Distutils] [issue94] is ok),
Message-ID: <1257817301.09.0.801740068334.issue94@psf.upfronthosting.co.za>


Changes by jerry <jerryregal at ymail.com>:


----------
files: unnamed
status: unread
title: is ok),
Added file: http://bugs.python.org/setuptools/file64/unnamed

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue94>
_______________________________________________

From david at ar.media.kyoto-u.ac.jp  Tue Nov 10 02:31:39 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 10 Nov 2009 10:31:39 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	
	<4AF6C5FF.8050109@ar.media.kyoto-u.ac.jp>	
	<94bdd2610911081001q744ab751wde2d9a78b9a2eeeb@mail.gmail.com>	
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>	
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>	
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>	
	<87eio82mzh.fsf@benfinney.id.au>	
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>	
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
Message-ID: <4AF8C27B.20807@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
>
> What is a registery of extension exactly ? Distutils let you register your own
> commands, you can use through the CLI.
>
> Can you provide more details ?

Sure. Right now, if you want to deal with a new source or a new target,
you need to create a new command or override one. For example, cython
has distutils extension which subclass build_ext, we in numpy do the
same. This causes several issues:
    - cython build_ext subclass distutils command, we do the same. How
to use both at the same time ? That's one typical example of the
subclassing issue Ian Bicking mentioned,
    - in the case of compiled extensions, the basic structure is to
build all object files for each extension from the compiler class
(compile method). The compile method structure is:

for obj in objects:
     if src == "bla":
         do_bla()
     elif src == "blabla":
         do_blabla()
     ...
     else:
          raise some exception for unrecognied extension.

If you want to support a new source/target (say assembler), you need to
deal at this stage (or use hacks to deal with the new  file extension
you want to deal with, and remove it later...). So you need to copy
build_ext, you cannot extend it.

If you look at msvccompiler vs msvc9compiler, both compile methods are
almost the same and copied from each other.

Now, if instead you have a dictionary {src_extension: callable}, you
would be able to add a new tool and extend existing code without
touching or copying anything. You could have syntactic sugar to have
something like:

@extension(".asm")
def assemble(....):
    do_assembler_stuff()

All the assembler-related thing would be in a separate module, and you
would only need to register for the tool explicitly when you need it
(every tool could be the same, and current distutils tools would then be
registered automatically).

> I am in for making Distutils evolve, but I need very precise real
> world use cases
> not saying that Distutils shouldn't do imperative operations).

I have given very explicit examples in this discussion - I have written
them on the wiki last time it was discussed as requested:

http://wiki.python.org/moin/Distutils/PluginSystem

I don't think it is accurate to summarize my critic as a vague "do
imperative operations".

>
> Last, I am not sure why you want only backward-compatible changes in distutils.
>
> There's no plan to keep backward-compatibility if breaking it makes
> DIstutils better. We will have pending deprecation warnings, that's
> all.

What is the intended policy: deprecate something in v1 and break it in
v1+1 ? I am not sure this works well when you need to correct deep
design issues which impact a lot of code (and this will be required).
For example, I would prefer not changing numpy.distutils several times
just to make a few API cleaner.

I guess I don't see the point of breaking things while keeping the
current distutils codebase. I would rather do the exact contrary: throw
away the codebase, but keep current setup.py working through a
conversion script. Things like sandboxing pypi and other things become
much easier that way.

cheers,

David

From wentland at cl.uni-heidelberg.de  Tue Nov 10 09:33:50 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Tue, 10 Nov 2009 09:33:50 +0100
Subject: [Distutils] A Python Packaging Tutorial
In-Reply-To: <94bdd2610911070544q4648e73fnc483905ad3da97d3@mail.gmail.com>
References: <94bdd2610911070544q4648e73fnc483905ad3da97d3@mail.gmail.com>
Message-ID: <20091110083350.GE20601@kinakuta.local>

On Sat, Nov 07, 2009 at 14:44 +0100, Tarek Ziad? wrote:
> As Ian suggested, and since others thinks its a good idea, I suggest
> that we start writing a "python packaging guide" using
> reStructuredText and DVCS, explaining how to handle packaging today.
> (with *all* existing tools)

I think this is a great idea, but what is meant by "today" ? Do you
intend to describe packaging with the 0.6 or 0.7 branch of Distribute?

> When it's done, we could add it in python's Doc/ with the help of Georg.

This is indeed of uttermost importance as the natural way to start
packaging today is to read the "Distribute Python Modules" document in
the Python documentation.

One problem I was wasting my time on was trying to get the "Requires" and
"Provides" arguments to *distutils* setup right, just to learn that
these are not used at all and all the effort was wasted. Things like
this should not happen. I know the reasons for that now, but it was
confusing nonetheless.

> I think Pauli provided a good starting list of what is not obvious for
> a new comer.

As a new comer I might want to add some points:

1. Distribute 0.6 / 0.7

* Should I use Distribute 0.6 features in my software right now? 

I have the feeling that using Distribute 0.6 and learning the setuptools
API is not a wise decision right now, given all the changes that are
happening.

* Will there be an upgrade path?

2. Data Files

It looks like this *important* part of packaging software is neglected
in the ongoing discussion and development. What are the plans regarding
packaging data files? pkgutil.get_data() is absolutely useless for
retrieving data files IMHO. The alternative (subclassing distutil
commands) should *not* be the *standard* way to handle this.

kind regards and thanks for all the work!

    Wolodja
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091110/7c382bfb/attachment.pgp>

From reinout at vanrees.org  Tue Nov 10 15:00:56 2009
From: reinout at vanrees.org (Reinout van Rees)
Date: Tue, 10 Nov 2009 14:00:56 +0000 (UTC)
Subject: [Distutils] buildout/distribute recursion error fixed
References: <slrnhf53cf.tdr.reinout@isengrim.local>
	<94bdd2610911050358n4cb88665wfcf4825301182bc2@mail.gmail.com>
Message-ID: <slrnhfisgo.1k22.reinout@isengrim.local>

On 2009-11-05, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Nov 5, 2009 at 9:32 AM, Reinout van Rees <reinout at vanrees.org> wrote:
>> Hi all,
>>
>> Several people have seen a hard-to-reproduce recursion error when running
>> buildout in combination with distribute. ?You'd get 200 lines of
>> "distribute".
>>
>> Good news: the problem has been identified. ?And I've fixed it on a
>> zc.buildout branch (branches/reinout_distribute_upgrade_check). ?Tarek will
>> test it and probably/hopefully release it real soon, wink, wink.
>
> Yes, thanks for the fix/work ! It looks nice and also simplifies the code.
>
> Before it's merged in trunk and eventually released, we need to :
>
> - add a test demonstrating this bug in easy_install.txt

Buildout's test aren't setup to test this kind of things directly.  I *did*
add a test to easy_install.txt that tests whether the offending _get_dist()
(which starts the recursion) gets called.  This fails before my fix and
succeeds afterwards.

On Tarek's suggestion, I moved the test out of easy_install.txt to a separate
file to keep the corner case test out of the user documentation.

> - manually test the various scenarii (these are not fully covered with
> zc.buildout unittests because the assmption is made that python don't
> have setuptools installed):
>
>     - nude python, buildout without --distribute, plone buildout cfg file
>     - python with setuptools, buildout witout --distribute, plone
> buildout cfg file
>     - python with distribute < 0.6.8, buildout witout --distribute,
> plone buildout cfg file
>     - these 3 tests *with* --distribute

I did not test all the variants, that would cost me waaaay too much time for
such a small fix.  To be honest.

I did reproduce it with setuptools + distribute with and without the fix with
the development buildout itself.  That executes the changed code point enough.


Reinout


-- 
Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"


From ziade.tarek at gmail.com  Tue Nov 10 19:17:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 10 Nov 2009 19:17:11 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>

On Tue, Nov 10, 2009 at 2:31 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
[..]
>
> If you want to support a new source/target (say assembler), you need to
> deal at this stage (or use hacks to deal with the new ?file extension
> you want to deal with, and remove it later...). So you need to copy
> build_ext, you cannot extend it.
>
> If you look at msvccompiler vs msvc9compiler, both compile methods are
> almost the same and copied from each other.

So here's a refactoring proposal :

- build_ext doesn't handle a compiler instance anymore (therefore it
doesn't have a "compile_type" anymore)

- instead, the Extension class handles this compile type

- right now we have a dict in ccompiler (compiler_class) that has all
the compiler classes.
  let's make it extensible through a registery using distutils.cfg
(like the commands)

so: when build_ext is run, it compiles the Extensions by using this
registery of compilers, and the type of compiler contained in each
extension. It keeps instances in a pool to reuse them if needed/

[..]
>
> I have given very explicit examples in this discussion - I have written
> them on the wiki last time it was discussed as requested:
>
> http://wiki.python.org/moin/Distutils/PluginSystem
>
> I don't think it is accurate to summarize my critic as a vague "do
> imperative operations".

Sorry about that, I lost track of this.

>
>>
>> Last, I am not sure why you want only backward-compatible changes in distutils.
>>
>> There's no plan to keep backward-compatibility if breaking it makes
>> DIstutils better. We will have pending deprecation warnings, that's
>> all.
>
> What is the intended policy: deprecate something in v1 and break it in
> v1+1 ? I am not sure this works well when you need to correct deep
> design issues which impact a lot of code (and this will be required).
> For example, I would prefer not changing numpy.distutils several times
> just to make a few API cleaner.

You just don't replace a stdlib package with its v2 like that, there
is a deprecation process to have.

Distutils is not living only through setup.py. It has some public APIs
imported and used by people.

While some deep changes are required, if setting up a deprecation is
possible, I will do it.

>
> I guess I don't see the point of breaking things while keeping the
> current distutils codebase. I would rather do the exact contrary: throw
> away the codebase, but keep current setup.py working through a
> conversion script. Things like sandboxing pypi and other things become
> much easier that way.

-1

I'm hearing this sometimes. That's what I thought too when I started
to work with Distutils
a year and a half ago. And I couldn't understand Guido when he was saying that
redoing it from scratch would be a mistake. Now I agree with him.

There's a *lot* of knowledge in Distutils. Throwing it away means
entering a new cycle that will last
for several years. Small refactoring can be done today, as long as the
code is tested.

While the code is not yet as modern as other modules in the stdlib, I
invite you to compare Python 2.4's
distutils with the current trunk, and realize the amount of work I've
been doing in the past year.

My goal was to have a good coverage before starting any big
refactoring, and to fix the old pending bugs

2 years ago, the code was barely tested (the coverage was 18%) and now
it's well tested (80%+ depending on the platform). It's now almost
fully PEP 8, and I am now more confident when I start any refactoring.

I understand the pain with build_ext. let's work it out.

Tarek

From david at ar.media.kyoto-u.ac.jp  Wed Nov 11 03:08:32 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 11 Nov 2009 11:08:32 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>	
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>	
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>	
	<87eio82mzh.fsf@benfinney.id.au>	
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>	
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>	
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>	
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
Message-ID: <4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
> On Tue, Nov 10, 2009 at 2:31 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
> [..]
>> If you want to support a new source/target (say assembler), you need to
>> deal at this stage (or use hacks to deal with the new  file extension
>> you want to deal with, and remove it later...). So you need to copy
>> build_ext, you cannot extend it.
>>
>> If you look at msvccompiler vs msvc9compiler, both compile methods are
>> almost the same and copied from each other.
>
> So here's a refactoring proposal :
>
> - build_ext doesn't handle a compiler instance anymore (therefore it
> doesn't have a "compile_type" anymore)
>
> - instead, the Extension class handles this compile type
>
> - right now we have a dict in ccompiler (compiler_class) that has all
> the compiler classes.
>   let's make it extensible through a registery using distutils.cfg
> (like the commands)
>
> so: when build_ext is run, it compiles the Extensions by using this
> registery of compilers, and the type of compiler contained in each
> extension. It keeps instances in a pool to reuse them if needed/

What is important here is how to add new tools, without touching or
impacting distutils in many places. In particular, what distutils
expects from a tool should be clearly defined (right now it is
implementation defined).

Adding cython or assembly support are good use cases to refactor this I
think.

> You just don't replace a stdlib package with its v2 like that, there
> is a deprecation process to have.
>
> Distutils is not living only through setup.py. It has some public APIs
> imported and used by people.

I am aware about the usage of distutils: I don't think it has public
API, though. It has functions that people use, but it is far from clear
what is public from what is private. Many things we do numpy.distutils
are clearly dependent on implementation details of distutils.

The problem with moving forward distutils is not deprecation, the
problem is when you break/remove the old API. At this point, you will
break people's code who rely on the old API. If the changes are big
enough (as they should be to warrant breaking the old API in the first
place), it will require significant effort from distutils API users. So
people will see this as a trade-off: does it worth my time to spend time
to use the new version of distutils, what will it bring to me ? Without
significant new features, this will be difficult.

> -1
>
> I'm hearing this sometimes. That's what I thought too when I started
> to work with Distutils
> a year and a half ago. And I couldn't understand Guido when he was saying that
> redoing it from scratch would be a mistake. Now I agree with him.

IIRC, the main argument was that a new system would mean giving up
existing setup.py, but that can be worked around. I don't know whether
Guido's opinion has changed since, but he wondered last year wether
dropping backward compatibility was an option
(http://www.opensubscriber.com/message/distutils-sig at python.org/10291496.html).

Otherwise, a new system would look nothing like distutils. One of the
main argument to avoid rewrite is that you will end up doing the same
mistakes, and old code is more complicated because it has been tested.
But here, we know what a good design is like as other languages have
vastly superior solutions to this problem.

As far as compilation is concerned at least, the distutils knowledge is
vastly overblown. First, most of it comes from autoconf on unices. You
have the MSVC tools knowledge, but that can be easily reused (in
numscons, I added msvc support from the python 2.6 msvc9compiler, this
was not difficult). Most other tools are rather obsolete - and if you
break any API in distribute there, you will most likely lose them as
well anyway (I am thinking about OS/2, Metrowerk kind of tools).

Again, I don't mean to say that working on distribute is a mistake, or
criticize what you do in any way. I just don't think it will solve any
significant issue for the scientific python community. But enough
"stop-energy": at this point, I will just shut up, and continue working
on my own idea - if they make sense, the scientific community is big
enough so that the solution could be used there only, at least for some
time.

cheers,

David

From greg.ewing at canterbury.ac.nz  Wed Nov 11 07:14:42 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 11 Nov 2009 19:14:42 +1300
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
Message-ID: <4AFA5652.8080708@canterbury.ac.nz>

David Cournapeau wrote:
> One of the
> main argument to avoid rewrite is that you will end up doing the same
> mistakes, and old code is more complicated because it has been tested.
> But here, we know what a good design is like as other languages have
> vastly superior solutions to this problem.

Also, it seems to me that in this case, the basic
architecture of distutils is already so full of
mistakes that there just isn't an incremental way
of getting to a better place, especially given the
requirement of not breaking any existing setup.py
scripts together with the fact that the API of distutils
effectively consists of its *entire implementation*.

So while complete rewrites are best avoided *if possible*,
I don't think we have a choice in this case.

-- 
Greg

From david at ar.media.kyoto-u.ac.jp  Wed Nov 11 07:05:03 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 11 Nov 2009 15:05:03 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA5652.8080708@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>	<87eio82mzh.fsf@benfinney.id.au>	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
Message-ID: <4AFA540F.3050205@ar.media.kyoto-u.ac.jp>

Greg Ewing wrote:
>
> Also, it seems to me that in this case, the basic
> architecture of distutils is already so full of
> mistakes that there just isn't an incremental way
> of getting to a better place, especially given the
> requirement of not breaking any existing setup.py
> scripts together with the fact that the API of distutils
> effectively consists of its *entire implementation*.

Exactly. The fact that we in numpy consider distutils backward
compatibility not worth the cost, even though we are most likely the
most tied up with distutils, is quite telling about the state of affairs
IMHO.

David

From ziade.tarek at gmail.com  Wed Nov 11 09:59:23 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 09:59:23 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911110059j6ddb150bx57b528ecf806e0@mail.gmail.com>

On Wed, Nov 11, 2009 at 3:08 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
[..]
>
> What is important here is how to add new tools, without touching or
> impacting distutils in many places. In particular, what distutils
> expects from a tool should be clearly defined (right now it is
> implementation defined).

Yes, and in that case, it means writing a new compiler class.

[...]
>> Distutils is not living only through setup.py. It has some public APIs
>> imported and used by people.
>
> I am aware about the usage of distutils: I don't think it has public
> API, though. It has functions that people use, but it is far from clear
> what is public from what is private. Many things we do numpy.distutils
> are clearly dependent on implementation details of distutils.
>

A public api doesn't have a "_" prefix.

[..]
> Otherwise, a new system would look nothing like distutils. One of the
> main argument to avoid rewrite is that you will end up doing the same
> mistakes, and old code is more complicated because it has been tested.
> But here, we know what a good design is like as other languages have
> vastly superior solutions to this problem.
>
> As far as compilation is concerned at least, the distutils knowledge is
> vastly overblown. First, most of it comes from autoconf on unices. You
> have the MSVC tools knowledge, but that can be easily reused (in
> numscons, I added msvc support from the python 2.6 msvc9compiler, this
> was not difficult). Most other tools are rather obsolete - and if you
> break any API in distribute there, you will most likely lose them as
> well anyway (I am thinking about OS/2, Metrowerk kind of tools).
>
> Again, I don't mean to say that working on distribute is a mistake, or
> criticize what you do in any way. I just don't think it will solve any
> significant issue for the scientific python community. But enough
> "stop-energy": at this point, I will just shut up, and continue working
> on my own idea - if they make sense, the scientific community is big
> enough so that the solution could be used there only, at least for some
> time.

I asked you for your use cases so we could work on changing things,
but it's evident
at this point that you don't want to use Distutils or you don't think
it can evolve.

I don't think the scientific community is so different from any other Python
community either, in what they need.

And I don't think Distutils is a lost case, as you seem to think.

Tarek

From ziade.tarek at gmail.com  Wed Nov 11 10:09:23 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 10:09:23 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA5652.8080708@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
Message-ID: <94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>

On Wed, Nov 11, 2009 at 7:14 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> David Cournapeau wrote:
>>
>> One of the
>> main argument to avoid rewrite is that you will end up doing the same
>> mistakes, and old code is more complicated because it has been tested.
>> But here, we know what a good design is like as other languages have
>> vastly superior solutions to this problem.
>
> Also, it seems to me that in this case, the basic
> architecture of distutils is already so full of
> mistakes that there just isn't an incremental way
> of getting to a better place, especially given the
> requirement of not breaking any existing setup.py
> scripts together with the fact that the API of distutils
> effectively consists of its *entire implementation*.
>
> So while complete rewrites are best avoided *if possible*,
> I don't think we have a choice in this case.

While "build_ext" is not handy, I don't buy the fact that Distutils is
"full of mistakes".
We have to work with use cases. David gave a use case: being able to
compile cython
or assembly files. I proposed a solution based on being able to define
a compiler
at the Extension level, rather that for the entire build_ext command.

If the answer to that solution is just: "Distutils sucks anyways..",
it is not really helpfull imho..

I don't see the point to write Distutils from scratch, instead of
making it evolve.

Tarek

From ziade.tarek at gmail.com  Wed Nov 11 10:11:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 10:11:25 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>

On Wed, Nov 11, 2009 at 7:05 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
[..]
>
> Exactly. The fact that we in numpy consider distutils backward
> compatibility not worth the cost, even though we are most likely the
> most tied up with distutils, is quite telling about the state of affairs
> IMHO.

That doesn't prove Distutils can't evolve. That just shows that numpy
worked on its
side. If the Numpy projects made some refactoring/improvement, why the
project didn't try to push
it back in Distutils itself ?

From regebro at gmail.com  Wed Nov 11 10:39:30 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 11 Nov 2009 10:39:30 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
Message-ID: <319e029f0911110139i35ebee52t7579ee034b2cc4c@mail.gmail.com>

To the people who wants a rewrite, two things need to be asked:

1. Do you think the new PEPs in development should be followed? In
that case, what is the benefit of rewriting, instead of fixing?

2. When are you done? :-) I'm not being rude, but this is open source.
There is no "Someone" that can rewrite distutils from scratch, it must
be done by those who thinks it should be done. Those who think
distutils should be rewritten from scratch need to sit down and do it.
If nobody is willing to write the code, the code is not needed.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From david at ar.media.kyoto-u.ac.jp  Wed Nov 11 10:46:55 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 11 Nov 2009 18:46:55 +0900
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	
	<87eio82mzh.fsf@benfinney.id.au>	
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>	
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>	
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>	
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>	
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>	
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>	
	<4AFA5652.8080708@canterbury.ac.nz>	
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
Message-ID: <4AFA880F.8020801@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
> On Wed, Nov 11, 2009 at 7:05 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
> [..]
>> Exactly. The fact that we in numpy consider distutils backward
>> compatibility not worth the cost, even though we are most likely the
>> most tied up with distutils, is quite telling about the state of affairs
>> IMHO.
>
> That doesn't prove Distutils can't evolve.

No, but that's not the point I was trying to make: I meant that we
consider the distutils API not to be a significant asset, and would be
happy to throw away all numpy.distutils for a significantly better
system. We now have two build systems in numpy (one based on scons): I
think it takes me 5 to 10 times more effort on average to add a feature
to the distutils-based compared to the scons system. There are some
features I can not implement because I have not find a solution to it in
distutils. There are arbitrary limitations like the inability to call
commands directly from setup.py, retrieve global informations from
setup.py, classes which behave differently on different platforms.

Example: how to retrieve the install prefix in setup.py. You need a good
understanding of distutils to understand why it is so complicated, and
the example shows almost everything that's wrong in distutils design.
Many expectations are undocumented, like what method/class can be called
where and when. All this implicit behavior is part of the API, and
that's not documented anywhere that I know of.

Also, what happened in python 2.6.3/2.6.4 w.r.t. distribute has happened
quite often for numpy.distutils, and I consider it inherent to distutils
way of working.

> If the Numpy projects made some refactoring/improvement, why the
> project didn't try to push
> it back in Distutils itself ?

They are not improvements or refactoring for most part. They are things
quite specific to our needs: fortran support, support for our internal
code generator, swig, f2py, etc... A few things could be useful to
distutils, like manifest support for mingw for 2.6 and later, as well as
basic autoconf-like tests (checking for functions, type, type size,
etc...). We would be happy to contribute patches to Distribute if this
is considered useful.

cheers,

David


From ziade.tarek at gmail.com  Wed Nov 11 11:25:48 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 11:25:48 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>

On Wed, Nov 11, 2009 at 10:46 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
[..]
> Example: how to retrieve the install prefix in setup.py. You need a good
> understanding of distutils to understand why it is so complicated, and
> the example shows almost everything that's wrong in distutils design.
> Many expectations are undocumented, like what method/class can be called
> where and when. All this implicit behavior is part of the API, and
> that's not documented anywhere that I know of.

Ok, so I read this example as a lack of documentation and the lack of
clear APIs to get the installation paths.

Also, notice that we are in the process of adding a new python module
in the stdlib, called
"sysconfig", that will contains all installation paths scheme for all
supported platforms.

I have a branch in the svn going on for that.

Knowing where to install things and what is up with a given platform
is a wider problem that Distutils in fact, it concerns site.py as
well, and having a sysconfig module that handles this will help.

I also expect it to help the work people from Jython, PyPy, and
IronPython will do in the future.

>
> Also, what happened in python 2.6.3/2.6.4 w.r.t. distribute has happened
> quite often for numpy.distutils, and I consider it inherent to distutils
> way of working.

I fully agree that this particular example is demonstrating that
build_ext is suffering from a lack
of API documentation.

I am still waiting for you comment on the solution to remove the
compile code from build_ext
and have it at the Extension level, with an option to add new
compilers in a  more easier way.

>
>> If the Numpy projects made some refactoring/improvement, why the
>> project didn't try to push
>> it back in Distutils itself ?
>
> They are not improvements or refactoring for most part. They are things
> quite specific to our needs: fortran support, support for our internal
> code generator, swig, f2py, etc... A few things could be useful to
> distutils, like manifest support for mingw for 2.6 and later, as well as
> basic autoconf-like tests (checking for functions, type, type size,
> etc...). We would be happy to contribute patches to Distribute if this
> is considered useful.

If there's someone from the Numpy project that can help in isolating patches
againts Distutils trunk in our issue tracker, I'd be more than happy
to reduce the gap between
the two projects.

Notice that this may have been done in the past, I didn't manage to
review all the distutils bugs
yet (some are 5 years old)

Tarek

From sparks.m at gmail.com  Tue Nov 10 02:12:13 2009
From: sparks.m at gmail.com (Michael Sparks)
Date: Tue, 10 Nov 2009 01:12:13 +0000
Subject: [Distutils] [Python-Dev] People want CPAN :-)
In-Reply-To: <B8D9CAC5-DA6D-47FC-AD1D-1FC388FBBC6B@gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a21398cfb92acc7d47f8151c249291c4@preisshare.net>
	<ca471dc20911061514q642ad41br9fb02d675fe411db@mail.gmail.com>
	<87iqdm7nkp.fsf@benfinney.id.au>
	<B8D9CAC5-DA6D-47FC-AD1D-1FC388FBBC6B@gmail.com>
Message-ID: <20e5c9660911091712y3548145yf0b48a0fa075220f@mail.gmail.com>

[ I'm posting this comment in reply to seeing this thread:
    * http://thread.gmane.org/gmane.comp.python.distutils.devel/11359
Which has been reposted around - and I've read that thread. I lurk on
this list, in case anything comes up that I'd hope to be able to say
something useful to. I don't know if this will be, but that's my
reason for posting. If this is the wrong place, my apologies, I don't
sub to distutils-sig :-/ ]

On Sat, Nov 7, 2009 at 2:30 PM, ssteinerX at gmail.com <ssteinerx at gmail.com> wrote:
> On Nov 7, 2009, at 3:20 AM, Ben Finney wrote:
>
>> Guido van Rossum <guido at python.org> writes:
>>
>>> On Fri, Nov 6, 2009 at 2:52 PM, David Lyon <david.lyon at preisshare.net>
>>> wrote:
>>>>
[ lots of snippage ]
...
> All in all, I think this could be a big leap forward for the Python
> distribution ecosystem whether or not we eventually write the PyPan I wished
> for as a new Perl refugee.

Speaking as someone who left the perl world for the python world, many
years ago now, primarily due to working on one project, the thing I
really miss about Perl is CPAN. It's not the fact that you know you do
perl Makefile.PL && make && make test && make install. Nor the fact
that it's trivial to set up a skeleton package setup that makes that
work for you. It's not the fact that there's an installer than can
download & track dependencies.

The thing that makes the difference IMHO is two points:
    * In a language which has a core ethos "There's more than one way
to do it", packaging is the one place where there is one, and only one
obvious way to do it. (Oddly with python, with packaging this is
flipped - do I as a random project use distutils? pip? setuptools?
distribute? virtualenv?)
    * It has a managed namespace or perhaps better - a co-ordinated namespace.

CPAN may have lots of ills, and bad aspects about it (I've never
really trusted the auto installer due to seeing one too many people
having their perl installation as a whole upgraded due to a bug that
was squashed 6-8 years ago), but these two points are pretty much
killer.

All the other aspects like auto download, upload, dependency tracking,
auto doc extraction for the website etc, really follow from the
managed namespace really. I realise that various efforts like
easy_install & distribute & friends make that sort of step implicitly
- since there can only be one http://pypi.python.org/pypi/flibble .
But it's not quite the same - due to externally hosted packages.

For more detail about this aspect:
   * http://www.cpan.org/modules/04pause.html#namespace

I'm really mentioning this because I didn't see it listed, and I
really think that it's very easy to underestimate this aspect of CPAN.

IMHO, it is what matters the most about CPAN. The fact that they've
nabbed the CTAN idea of having an archive network for storing,
mirroring and grabbing stuff from is by comparison /almost/ irrelevant
IMHO. It is the sort of thing that leads to the DBI::DBD type stuff
that is being simple to use, because of the encouragement to talk and
share a namespace.

The biggest issue with this is retrofitting this to an existing world.

Personal opinion, I hope it's useful, and going back into lurk mode (I
hope :). If this annoys you, please just ignore it.


Michael.

From david.lyon at preisshare.net  Wed Nov 11 12:20:09 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 11 Nov 2009 06:20:09 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFA5652.8080708@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp>
	<hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
Message-ID: <c6b86a12ca0622d021303ba635577bfa@preisshare.net>

On Wed, 11 Nov 2009 19:14:42 +1300, Greg Ewing
<greg.ewing at canterbury.ac.nz>
wrote:

> Also, it seems to me that in this case, the basic
> architecture of distutils is already so full of
> mistakes that there just isn't an incremental way
> of getting to a better place, especially given the
> requirement of not breaking any existing setup.py
> scripts together with the fact that the API of distutils
> effectively consists of its *entire implementation*.
> 
> So while complete rewrites are best avoided *if possible*,
> I don't think we have a choice in this case.

Fwiw, I can't see any way to make that work.

Even if I said that I didn't like distutils, which btw isn't
true because my knowledge of it is so limited, it's just
not practical to rewrite such a complex tool within any
reasonable timeframe.

Anyway, moving towards CPAN isn't about rewriting all of
distutils, only a few key parts. Or restructuring.

The parts that shouldn't be rewritten (because the people
that wrote it were too clever - and the degree of difficulty
to high) is any C interface. That's an important part of
distutils that is likely to end up worse rather than better.

If there are candidates for 'evolution', it has to be in
the areas such as package creation (collecting all the
files and putting them in a .tar.gz or .egg or .zip
or .exe) for upload to pypi.

In CPAN, one wouldn't dream of there being so many 
possibilities.

As I've said on python-dev, we should just revise and
call a package an egg. In science, it wouldn't be tolerated
to call a hydrogen atom using 6 different synonyms. But
in python we do.

Then we wonder why people get confused...

There's a big difference between a rewrite and an
evolution. Rewrite isn't viable, but an evolution is.

>From science, we even know that it only takes two
generations to become nearly immune from strong radiation
as shown in chernoble. So when we talk about evolution
lets use it in the modern sense, not in the darwinian.

Evolution is better than revolution is better than nothing
happening at all.

David









From cournape at gmail.com  Wed Nov 11 13:31:44 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Nov 2009 12:31:44 +0000 (UTC)
Subject: [Distutils] People want CPAN
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
Message-ID: <loom.20091111T124450-168@post.gmane.org>

Tarek Ziad?? <ziade.tarek <at> gmail.com> writes:

> 
> Also, notice that we are in the process of adding a new python module
> in the stdlib, called
> "sysconfig", that will contains all installation paths scheme for all
> supported platforms.

I don't think we are talking about the same thing. If the pb was just missing
API, we could have just added it to numpy.distutils. We override pretty much
every command of distutils and setuptools anyway.

I needed to obtain the install prefix, be it the default one, or the one set
through --prefix, or the current directory for in place build/develop install.
I need to know it when building C libraries, in build_clib.

AFAIK, there is only one way to get the information: since the --prefix is only
known once install.finalize_options() has run, you need to call the method from
build_clib. 

Naive code could be something like

class MyBuildClib(build_clib):
    def run(self):
        install_cmd = self.distribution.get_command_obj("install")
        if not install_cmd.finalized == 1:
            install_cmd.finalize_options()

        if self.inplace == 1:
            top = "."
        else:
            top = install_cmd.install_libbase
        # top is the top directory where libraries should be installed
        build_clib.run(self)

It is already quite ugly, but it looks like it could work. It actually does in
the simple case, like python setup.py install. But if you call python setup.py
build_clib install --prefix=something, it does not. The error message is a
typical example of distutils style: "error: must supply either prefix/exec-
prefix/home or install-base/install-platbase -- not both". The only way I
managed to make this work is to replace:

    install_cmd = self.distribution.get_command_obj("install"))                                  

by 

    install_cmd = copy.copy(self.distribution.get_command_obj("install"))                                  

That's not an API problem, or a documentation problem. That's a fundamental
issue because --prefix is only known from install, and when you start running
commands in an order different than expected by distutils, you get weird errors
(the above error is actually almost sensical if you really know distutils
code). I passed over the fact that in some conditions that elude me ATM and are
platform specific, install_libbase does not exist and raises an attribute
error.

If you have a configure / build / install like the vast majority of build
systems out there, this problem disappears. I don't see how the problem can be
fixed without touching how commands work.

Moreover, that's a typical example where the only way to be robust is to check
that every attribute you are using actually exist before. At that point, you
really want to run back to m4, perl, autogenerated makefiles and shell
programming :)

> I am still waiting for you comment on the solution to remove the
> compile code from build_ext
> and have it at the Extension level, with an option to add new
> compilers in a  more easier way.

I will try to document how scons does it. I think the basic idea could be
reused in distutils.

> If there's someone from the Numpy project that can help in isolating patches
> againts Distutils trunk in our issue tracker, I'd be more than happy
> to reduce the gap between
> the two projects.

If that was not clear, I am that guy. I have been the main
maintainer of numpy.distutils and of numpy/scipy build infrastructure for some
time now,

David



From ssteinerx at gmail.com  Wed Nov 11 13:36:14 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Wed, 11 Nov 2009 07:36:14 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <319e029f0911110139i35ebee52t7579ee034b2cc4c@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<319e029f0911110139i35ebee52t7579ee034b2cc4c@mail.gmail.com>
Message-ID: <45BD89A1-B5CE-4B39-8BCD-27BC8E7AF57F@gmail.com>

On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote:

> If nobody is willing to write the code, the code is not needed.

I think it would be more accurate to say that nobody is deciding that  
the "need" is sufficient to invest the resources to fill it.

There are lots of things that many people would be willing and able to  
write, that would fill a great "need", and that many people would  
benefit from, but that aren't getting written because there's no one  
willing to invest the time or money.

In come cases people are both willing and able, but unable to invest  
the time due to the pressing need to make a living.  I'd love to work  
on open source all day, but my wife and kids get cranky if they don't  
eat for a couple of days.

People don't always agree on what is the "rational" need to fill, and  
sometimes what we, as programmers "need", and know would make us more  
productive, is not what the people controlling the pursestrings are  
willing to finance, even if it would ultimately benefit them  
financially.

S




From regebro at gmail.com  Wed Nov 11 13:38:42 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 11 Nov 2009 13:38:42 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <45BD89A1-B5CE-4B39-8BCD-27BC8E7AF57F@gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<319e029f0911110139i35ebee52t7579ee034b2cc4c@mail.gmail.com>
	<45BD89A1-B5CE-4B39-8BCD-27BC8E7AF57F@gmail.com>
Message-ID: <319e029f0911110438vfd86078h333e8f7fa140e21b@mail.gmail.com>

2009/11/11 ssteinerX at gmail.com <ssteinerx at gmail.com>:
> On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote:
>
>> If nobody is willing to write the code, the code is not needed.
>
> I think it would be more accurate to say that nobody is deciding that the
> "need" is sufficient to invest the resources to fill it.

More accurate, but longer and redundant. :)

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From david.lyon at preisshare.net  Wed Nov 11 14:07:16 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 11 Nov 2009 08:07:16 -0500
Subject: [Distutils] =?utf-8?q?Fwd=3A_installing_lxml_=3F_-_todays_user_ex?=
 =?utf-8?q?perience_from_python-list=40python=2Eorg?=
Message-ID: <0d75c556f2db0cd28c5316be635d7f8f@preisshare.net>


-------- Original Message --------
Subject: installing lxml ?
Date: Wed, 11 Nov 2009 04:49:48 -0800 (PST)
From: 7stud <bbxx789_05ss at yahoo.com>
To: python-list at python.org

I'm trying to install lxml, but I can't figure out the installation
instructions.  Here:

http://codespeak.net/lxml/installation.html

it says:

1) Get the easy_install tool.

Ok, I went to the easy_install website, downloaded, and installed it.
The last two lines of the output during installation said this:

Installing easy_install script to /Library/Frameworks/Python.framework/
Versions/2.6/bin
Installing easy_install-2.6 script to /Library/Frameworks/
Python.framework/Versions/2.6/bin


2) ...run the following as super-user (or administrator):

easy_install lxml

On MS Windows, the above will install the binary builds that we
provide. If there is no binary build of the latest release yet, please
search PyPI for the last release that has them and pass that version
to easy_install like this:
easy_install lxml==2.2.2

On Linux (and most other well-behaved operating systems), easy_install
will manage to build the source distribution as long as libxml2 and
libxslt are properly installed, including development packages, i.e.
header files, etc. Use your package management tool to look for
packages like libxml2-dev or libxslt-devel if the build fails, and
make sure they are installed.

On MacOS-X, use the following to build the source distribution, and
make sure you have a working Internet connection, as this will
download libxml2 and libxslt in order to build them:
STATIC_DEPS=true easy_install lxml

-----------

My os is mac os x 10.4.11.   But this:

STATIC_DEPS=true easy_install lxml

is not a valid command:

$ sudo STATIC_DEPS=true easy_install lxml
Password:
sudo: STATIC_DEPS=true: command not found

In any case, if I do this:

$ sudo easy_install lxml
sudo: easy_install: command not found

In other words, when I installed easy_install it did not add anything
to my PATH which points to the installation directory mentioned during
installation:

Installing easy_install script to /Library/Frameworks/Python.framework/
Versions/2.6/bin

Ok, so I need to use the full path to the easy_install program (which
is not mentioned ANYWHERE in the installation instructions), i.e.

/Library/Frameworks/Python.framework/Versions/2.6/bin/easy_install

...but this still isn't going to work:

$ sudo STATIC_DEPS=true /Library/Frameworks/Python.framework/Versions/
2.6/bin/easy_install lxml
Password:
sudo: STATIC_DEPS=true: command not found


So what the heck is going on??

Attention developers: you may be one of the best programmers in the
world, but if you can't convey how to use your software to the average
user, then you are the equivalent of one of the worst programmers on
the planet.






-- 
http://mail.python.org/mailman/listinfo/python-list

From ziade.tarek at gmail.com  Wed Nov 11 14:16:15 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 14:16:15 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <loom.20091111T124450-168@post.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
Message-ID: <94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>

On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau <cournape at gmail.com> wrote:
[..]
> AFAIK, there is only one way to get the information: since the --prefix is only
> known once install.finalize_options() has run, you need to call the method from
> build_clib.
>
> Naive code could be something like
>
> class MyBuildClib(build_clib):
> ? ?def run(self):
> ? ? ? ?install_cmd = self.distribution.get_command_obj("install")
> ? ? ? ?if not install_cmd.finalized == 1:
> ? ? ? ? ? ?install_cmd.finalize_options()
>
> ? ? ? ?if self.inplace == 1:
> ? ? ? ? ? ?top = "."
> ? ? ? ?else:
> ? ? ? ? ? ?top = install_cmd.install_libbase
> ? ? ? ?# top is the top directory where libraries should be installed
> ? ? ? ?build_clib.run(self)
>
> It is already quite ugly, but it looks like it could work. It actually does in
> the simple case, like python setup.py install. But if you call python setup.py
> build_clib install --prefix=something, it does not. The error message is a
> typical example of distutils style: "error: must supply either prefix/exec-
> prefix/home or install-base/install-platbase -- not both". The only way I
> managed to make this work is to replace:

Ouch, That's not to be done. Something is wrong with your build_clib
design here.
You are roughly calling "install" as a sub command.

If you want to have install options for your command, and if your
command is about "installing",
it means that your command has to be a subcommand of "install". Those
get called once the options
passed to install have been finalized.

IOW, build_clib is not a subcommand of install, so you have troubles.

subcommands are: install_lib, install_headers, install_scripts,
install_data, install_egg_info
(and I agree that it's not simple to extend this list)

But, why in the first place, do you need the install --prefix in a
build command ?

For build_clib, if you need to build something, it goes in the
build_dir, or in place., and this is not impacted
by the install command.

>
> If you have a configure / build / install like the vast majority of build
> systems out there, this problem disappears. I don't see how the problem can be
> fixed without touching how commands work.

I fail to understand your demonstration. Commands that are in charge
of the *build*
have nothing to do with commands that are in charge of *installing*
file in various places
in the target system. So I fail to understand why build_clib interacts
with install,
and why it has to impact it howsoever (or vice-versa)

Now, if we take the generic use case (even if I don't think it should
be used in your case):

"a simple way to share options amongst commands"

As a matter of fact, there's a hackish way to perform this, by using
the distribution instance
as a placeholder for these "common" options, so several command can
read/write it.
(as opposed to local options and global options) without having to get
the command that manage the option.

But, at the end, since an option is either global, either specific to
a command, I guess a simple API in the command class should be enough
to avoid this hack:

   get_option(command_name, option_name)

This is similar to getting the command, (instanciate it + finalize it
if it doesn't exists yet)
and return a finalized option.

>
> Moreover, that's a typical example where the only way to be robust is to check
> that every attribute you are using actually exist before. At that point, you
> really want to run back to m4, perl, autogenerated makefiles and shell
> programming :)
>
>> I am still waiting for you comment on the solution to remove the
>> compile code from build_ext
>> and have it at the Extension level, with an option to add new
>> compilers in a ?more easier way.
>
> I will try to document how scons does it. I think the basic idea could be
> reused in distutils.

But, you didn't answer: if we add the ability to define a different
compiler for each Extension,
will it solve your use case ?

>> If there's someone from the Numpy project that can help in isolating patches
>> againts Distutils trunk in our issue tracker, I'd be more than happy
>> to reduce the gap between
>> the two projects.
>
> If that was not clear, I am that guy. I have been the main
> maintainer of numpy.distutils and of numpy/scipy build infrastructure for some
> time now,

If you are willing to spend some time in that, I am the guy who can
commit your patches in Python.

Regards
Tarek

From ssteinerx at gmail.com  Wed Nov 11 14:46:07 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Wed, 11 Nov 2009 08:46:07 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <319e029f0911110438vfd86078h333e8f7fa140e21b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<319e029f0911110139i35ebee52t7579ee034b2cc4c@mail.gmail.com>
	<45BD89A1-B5CE-4B39-8BCD-27BC8E7AF57F@gmail.com>
	<319e029f0911110438vfd86078h333e8f7fa140e21b@mail.gmail.com>
Message-ID: <5AA77A81-BF08-4340-A5D0-4DA3583D087A@gmail.com>


On Nov 11, 2009, at 7:38 AM, Lennart Regebro wrote:

> 2009/11/11 ssteinerX at gmail.com <ssteinerx at gmail.com>:
>> On Nov 11, 2009, at 4:39 AM, Lennart Regebro wrote:
>>
>>> If nobody is willing to write the code, the code is not needed.
>>
>> I think it would be more accurate to say that nobody is deciding  
>> that the
>> "need" is sufficient to invest the resources to fill it.
>
> More accurate, but longer and redundant. :)

I don't think it's redundant because a lot of "needs" in Python go  
unmet, not due to people's inability or unwillingness, but due to a  
lack of time/funding which amount to the same thing.

S


From cournape at gmail.com  Wed Nov 11 14:48:17 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Nov 2009 22:48:17 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
Message-ID: <5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>

On Wed, Nov 11, 2009 at 10:16 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

>
> Ouch, That's not to be done. Something is wrong with your build_clib
> design here.
> You are roughly calling "install" as a sub command.

Yes, it is "wrong" from distutils POV, but there is no other solution
to get the prefix. The problem is that prefix is available in install,
not that I want to know the prefix when building.

>
> If you want to have install options for your command, and if your
> command is about "installing",

I did not say my command was about installing - it does not install
anything. To be complete, we do not use this in build_clib, but in
build_src , to generate pkg-config-like files (build_src is a
numpy.distutils-specific command to build source files, for example
automatically generated wrappers around fortran libraries, and is
typically run before any other build_ command)

>
> For build_clib, if you need to build something, it goes in the
> build_dir, or in place., and this is not impacted
> by the install command.

That's exactly the problem. build + install description is too
simplistic for complex builds: you often need to know install options
at build time. You need to know some build options when building
configuration files, you need to pass configuration options to other
subsequent steps, etc...

>
> This is similar to getting the command, (instanciate it + finalize it
> if it doesn't exists yet)
> and return a finalized option.

This does not solve the issue IMHO. Since we both seem to like
thinking about use-cases, consider this use-case: you have a python
package with a complex extension built with make (say you have a
build_make command which calls a makefile). How do you do it ? How to
communicate path informations, compiler options between make and
distutils ? How to handle relinking (changing rpath at install time) ?

>>
>> I will try to document how scons does it. I think the basic idea could be
>> reused in distutils.
>
> But, you didn't answer: if we add the ability to define a different
> compiler for each Extension,
> will it solve your use case ?

I did not answer because my answer will take time to accurately
summarize tools problems in build tools. We have discussed quite a bit
about those issues in scons as well and I want to be sure that my
answer takes this into account as well.

>
> If you are willing to spend some time in that, I am the guy who can
> commit your patches in Python.

I will try to prepare a couple of patches against the hg repo this WE,

David

From m.van.rees at zestsoftware.nl  Wed Nov 11 15:10:59 2009
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Wed, 11 Nov 2009 14:10:59 +0000 (UTC)
Subject: [Distutils] Fwd: installing lxml ? - todays user experience
 from python-list@python.org
References: <0d75c556f2db0cd28c5316be635d7f8f@preisshare.net>
Message-ID: <hdeglj$8r5$1@ger.gmane.org>

David Lyon, on 2009-11-11:
> $ sudo STATIC_DEPS=true /Library/Frameworks/Python.framework/Versions/
> 2.6/bin/easy_install lxml
> Password:
> sudo: STATIC_DEPS=true: command not found

On linux 'true' is a command:
$ whatis true
true (1)             - do nothing, successfully

Apparently that is not available on Mac OS X.

Can you try with STATIC_DEPS=1 ?

(Note that I have no Mac experience.)

-- 
Maurits van Rees | http://maurits.vanrees.org/
            Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]


From ziade.tarek at gmail.com  Wed Nov 11 15:13:40 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 15:13:40 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
Message-ID: <94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>

On Wed, Nov 11, 2009 at 2:48 PM, David Cournapeau <cournape at gmail.com> wrote:
>> If you want to have install options for your command, and if your
>> command is about "installing",
>
> I did not say my command was about installing - it does not install
> anything. To be complete, we do not use this in build_clib, but in
> build_src , to generate pkg-config-like files (build_src is a
> numpy.distutils-specific command to build source files, for example
> automatically generated wrappers around fortran libraries, and is
> typically run before any other build_ command)
>

But you call it with "install" in your example, meaning that is is
called at install time, right ?

Or it is just that you want to get the "--prefix" value finalized and
computed by the install
command. If it's the later, I guess you will be able to use the
upcoming "sysconfig" module,
that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.

And I will probably add a way to override those prefix, meaning that
you will be able to
get from your command all install paths depending on a root prefix.

If not, it means that you are doing a post or pre install step during
installation.
e.g. like RPM's pre/post commits hooks.


>>
>> For build_clib, if you need to build something, it goes in the
>> build_dir, or in place., and this is not impacted
>> by the install command.
>
> That's exactly the problem. build + install description is too
> simplistic for complex builds: you often need to know install options
> at build time. You need to know some build options when building
> configuration files, you need to pass configuration options to other
> subsequent steps, etc...

Sorry, I can see it yet, it still fuzzy. Does that mean your binary
distribution will not be relocatable ?
e.g. meaning that once you have done your build for a given --prefix,
it won't be installable anywhere else ?

In that case you would need to remove the --prefix option from
install, and have it on your build
command.

Sounds like a pre/post install hook but I am not sure.

>> This is similar to getting the command, (instanciate it + finalize it
>> if it doesn't exists yet)
>> and return a finalized option.
>
> This does not solve the issue IMHO. Since we both seem to like
> thinking about use-cases, consider this use-case: you have a python
> package with a complex extension built with make (say you have a
> build_make command which calls a makefile). How do you do it ? How to
> communicate path informations, compiler options between make and
> distutils ? How to handle relinking (changing rpath at install time) ?

I don't know for the first part. I have to try it out. Can you provide
me such an extension ?

For the relinking at installation time problem, it is obvious that
something can be done.
we could have a pre/post install option where an arbitrary command could be
launch, as an install subcommand.

pre: works on the build_dir built by install or a previous build.
post: works on the installed file list

[..]
>> If you are willing to spend some time in that, I am the guy who can
>> commit your patches in Python.
>
> I will try to prepare a couple of patches against the hg repo this WE,

yeah, thanks ! \o/

Tarek

From cournape at gmail.com  Wed Nov 11 15:47:20 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Nov 2009 23:47:20 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
Message-ID: <5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>

On Wed, Nov 11, 2009 at 11:13 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> Or it is just that you want to get the "--prefix" value finalized and
> computed by the install
> command.

Yes.

> If it's the later, I guess you will be able to use the
> upcoming "sysconfig" module,
> that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.

Where is the sysconfig sources ? I don't see it in bitbucket.

> Sorry, I can see it yet, it still fuzzy. Does that mean your binary
> distribution will not be relocatable ?

Relinking was just an example, but yes, the binary would not
relocatable in that case (although you can actually build relocatable
binaries through $ORIGIN, but we are not here to talk about advanced
deployment issues of binaries). Just to be clear, I am not advocating
distutils to do it or even implement it at all, just to make it
possible.

>
>>> This is similar to getting the command, (instanciate it + finalize it
>>> if it doesn't exists yet)
>>> and return a finalized option.
>>
>> This does not solve the issue IMHO. Since we both seem to like
>> thinking about use-cases, consider this use-case: you have a python
>> package with a complex extension built with make (say you have a
>> build_make command which calls a makefile). How do you do it ? How to
>> communicate path informations, compiler options between make and
>> distutils ? How to handle relinking (changing rpath at install time) ?
>
> I don't know for the first part. I have to try it out. Can you provide
> me such an extension ?

Not for make, but I can try to port numpy.distutils.command.scons to
distutils (or distribute). The current code is damn ugly (I did most
of it when I started digging into distutils), but you can get an idea
here:

http://github.com/cournape/numpy/blob/master/numpy/distutils/command/scons.py

It calls scons, and you can thus build any C extension using scons.
Now, both distutils and scons makes this difficult (in particular,
there is no way to call scons from distutils, you need to launch scons
executable).

For me, one of the core idea of an improved distutils would be to make
this much easier. All compilers options form distutils would be in
simple data files with simple API, no objects, no class with countless
methods and complex protocol. Distutils v2 would have a default "dumb"
build tool, but you could use whatever tool instead if desired.

David

From cournape at gmail.com  Wed Nov 11 15:58:16 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 11 Nov 2009 23:58:16 +0900
Subject: [Distutils] Fwd: installing lxml ? - todays user experience
	from python-list@python.org
In-Reply-To: <hdeglj$8r5$1@ger.gmane.org>
References: <0d75c556f2db0cd28c5316be635d7f8f@preisshare.net>
	<hdeglj$8r5$1@ger.gmane.org>
Message-ID: <5b8d13220911110658v4deae658r633cde9995df6d01@mail.gmail.com>

On Wed, Nov 11, 2009 at 11:10 PM, Maurits van Rees
<m.van.rees at zestsoftware.nl> wrote:
> David Lyon, on 2009-11-11:
>> $ sudo STATIC_DEPS=true /Library/Frameworks/Python.framework/Versions/
>> 2.6/bin/easy_install lxml
>> Password:
>> sudo: STATIC_DEPS=true: command not found
>
> On linux 'true' is a command:
> $ whatis true
> true (1) ? ? ? ? ? ? - do nothing, successfully

 I would advise against using sudo easy_install. This will cause
trouble by mixing the system python and the softwares it expects vs.
custom ones. If it were me, I would even raises an exception in
easy_install if run under root without any prefix option. That's an
ever-ending source of trouble.

Most of the OP problems have nothing to do with easy_install or even
python, but with sudo. sudo does not propagate env variables (more
exactly, it has strict rules about which variables to reuse). Do not
use sudo, and your problem will disappear. On python 2.6, a good way
to avoid sudo is to use --local to  install things in $HOME/.local (I
don't know how to do that from easy_install --local is an install
option to distutils). Otherwise, install things in a directory which
is in your PYTHONPATH. I for example use $HOME/local on mac os x, and
install everything there for python < 2.6.

David

From wentland at cl.uni-heidelberg.de  Wed Nov 11 16:32:41 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Wed, 11 Nov 2009 16:32:41 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
References: <4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
Message-ID: <20091111153241.GD7023@kinakuta.local>

On Wed, Nov 11, 2009 at 14:16 +0100, Tarek Ziad? wrote:
> On Wed, Nov 11, 2009 at 1:31 PM, David Cournapeau <cournape at gmail.com> wrote:
> But, at the end, since an option is either global, either specific to
> a command, I guess a simple API in the command class should be enough
> to avoid this hack:

>    get_option(command_name, option_name)

+1
-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091111/cc0e862b/attachment.pgp>

From floris.bruynooghe at gmail.com  Wed Nov 11 16:31:17 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Wed, 11 Nov 2009 15:31:17 +0000
Subject: [Distutils] Fwd: installing lxml ? - todays user experience
 from python-list@python.org
In-Reply-To: <hdeglj$8r5$1@ger.gmane.org>
References: <0d75c556f2db0cd28c5316be635d7f8f@preisshare.net>
	<hdeglj$8r5$1@ger.gmane.org>
Message-ID: <20091111153117.GB24614@laurie.devork>

On Wed, Nov 11, 2009 at 02:10:59PM +0000, Maurits van Rees wrote:
> David Lyon, on 2009-11-11:
> > $ sudo STATIC_DEPS=true /Library/Frameworks/Python.framework/Versions/
> > 2.6/bin/easy_install lxml
> > Password:
> > sudo: STATIC_DEPS=true: command not found
> 
> On linux 'true' is a command:

That's not the issue.  The point of STATIC_DEPS=true is to set an
environment variable in the environment that easy_install is running
in.  So the problem here is not knowing how to invoke sudo (something
like "ENV=val sudo -E cmd" would do the job on top of my head, not
sure though - just "sudo -i" and then you have a normal root shell
though).

But that doesn't seem to be the point of this email here.  If you want
to help out with that you should probably find the thread on
python-list and participate there.


Regards
Floris

From regebro at gmail.com  Wed Nov 11 16:45:16 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Wed, 11 Nov 2009 16:45:16 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <5AA77A81-BF08-4340-A5D0-4DA3583D087A@gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<319e029f0911110139i35ebee52t7579ee034b2cc4c@mail.gmail.com>
	<45BD89A1-B5CE-4B39-8BCD-27BC8E7AF57F@gmail.com>
	<319e029f0911110438vfd86078h333e8f7fa140e21b@mail.gmail.com>
	<5AA77A81-BF08-4340-A5D0-4DA3583D087A@gmail.com>
Message-ID: <319e029f0911110745w9e9767fn18624f643c725cda@mail.gmail.com>

2009/11/11 ssteinerX at gmail.com <ssteinerx at gmail.com>:
> I don't think it's redundant because a lot of "needs" in Python go unmet,
> not due to people's inability or unwillingness, but due to a lack of
> time/funding which amount to the same thing.

People *want* a lot of things. But if they truly *need* it, they'll do it.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From ziade.tarek at gmail.com  Wed Nov 11 16:47:47 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 16:47:47 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
Message-ID: <94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>

On Wed, Nov 11, 2009 at 3:47 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Wed, Nov 11, 2009 at 11:13 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>> Or it is just that you want to get the "--prefix" value finalized and
>> computed by the install
>> command.
>
> Yes.

Ok. What is obvious to me now is that the "install" command does to much.
Knowing the paths is something useful for any command. So "sysconfig"
will help I think.

>
>> If it's the later, I guess you will be able to use the
>> upcoming "sysconfig" module,
>> that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.
>
> Where is the sysconfig sources ? I don't see it in bitbucket.

that's in python's svn, in a tarek_sysconfig branch. It's a revamp of
distutils/sysconfig.py
with the schemes from distutils/command/install.py
(work in progress)

[..]
>> I don't know for the first part. I have to try it out. Can you provide
>> me such an extension ?
>
> Not for make, but I can try to port numpy.distutils.command.scons to
> distutils (or distribute). The current code is damn ugly (I did most
> of it when I started digging into distutils), but you can get an idea
> here:
>
> http://github.com/cournape/numpy/blob/master/numpy/distutils/command/scons.py
>
> It calls scons, and you can thus build any C extension using scons.
> Now, both distutils and scons makes this difficult (in particular,
> there is no way to call scons from distutils, you need to launch scons
> executable).

I see. I'll take a look asap. Are you coming to Pycon btw ?


>
> For me, one of the core idea of an improved distutils would be to make
> this much easier. All compilers options form distutils would be in
> simple data files with simple API, no objects, no class with countless
> methods and complex protocol. Distutils v2 would have a default "dumb"
> build tool, but you could use whatever tool instead if desired.

The default compiler class exists for that, it's CCompiler, and is
mostly a placeholder for options.
But it's C oriented.  In my mind, implementing a new compiler for
Distutils means overriding it, and implementing, mainly:

- preprocess()
- compile()
- create_static_lib()
- link()

Now that's quite complex, and we could probably have a single method
(compile) that would
do the required work of compiling an extension the way it wants to.

So, yes, being able to register an arbitrary compiler class, with
arbitrary options passed through the Extension
could make it simpler:

setup(
..
ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
..)

where "pyd" is the name of the compiler that knows how to compile D files;
This compiler would do whatever it wants, as long as it is done in a
.compile() method:

.compile(name, files, *args, **kw)



Tarek

From wentland at cl.uni-heidelberg.de  Wed Nov 11 16:24:14 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Wed, 11 Nov 2009 16:24:14 +0100
Subject: [Distutils] Install time prefixes and data files
Message-ID: <20091111152414.GC7023@kinakuta.local>

Hi all,

I had a discussion with Tarek on IRC yesterday and would like to bring
some points to your attention and also write down some conclusions/ideas
we had yesterday.

I think that the distutils module has a serious flaw that need to be
addressed before any further work is done on getting PyPi in a better
shape. The main issue is, that there are many ways to customise the
installation of a distribution but absolutely nothing to retrieve this
information within the setup script or within your programs/libraries.

tl;dr:

save prefixes in .egg-info/PREFIX and create methods for Distribution to
retrieve these and data files

Install prefixes / Installation schemes
---------------------------------------

There are basically five installation schemes:
            
1. Default
2. User
3. Home
4. Prefix
5. Custom

All these installation schemes taken together enable the user to install
pure libraries, non-pure libraries, data files and scripts in exactly
the places s/he choose. But this also means that it is impossible to
make any assumptions about the locations in your code.

I think it is of uttermost importance to work on an implementation that
enables users to query this information:

* at build time (within setup.py)
* at run time (within scripts/libraries)

The major problem today is that the information where files of a given
distribution are placed is *only* available at build time and not saved
anywhere. And even at build time you'll have to get the finalised
install command to query that information, which is suboptimal and
counter-intuitive.

Data files
----------

Especially data files are problematic in this respect because the FHS
mandates that these are *not* placed within the packages but at exactly
defined places in the file system. The API available to retrieve data
files in the *stdlib* pkgutil.get_data() can not cope with this and is
therefore not usable.

This leads to the following problems:

* Linux package maintainers have to implement a lot of patches if they
  want to place data files in the correct places.

* Python developers have to either (i) save the various locations at
  install time in a place where they can retrieve that information later
  or (ii) accept that their library/program will break if one of the
  fancier installation schemes is used.

* Python distribution installations that violate the FHS
* ...

Proposal
--------

I propose three things:

1. Save installation prefixes for every installed distribution
2. Define standard infixes for typical data file classes
3. Implement an API that is able to query that information/retrieve data
  files

1 and 3. Discussion

The installation prefixes could for example be saved within a suitable
file in .egg-info/ which would mandate changes to PEP 376. I could think
of a PREFIX file and the following  methods/attribute for e.g. the
Distribution class

* get_install_prefixes()    -> returns a dictionary with all prefixes for
                               this distribution

* get_data_file(type, name) -> get data file of given type and name (see
                               (see below)

2. Discussion

The FHS differentiates between various classes of files and defines
proper location for them. We could define platform dependent 
standard infixes for the following data file classes for distribution
foo:

* configuration     etc
* shared data       usr/share/foo/
* readme            usr/share/foo/README        README, TODO, ... could
                                                be automatically
                                                discovered
* examples          usr/share/foo/examples
* documentation     usr/share/foo/doc
* man files         usr/share/man
* variable          var/lib/foo
* ...

There should be a set of platform dependent prefixes for all classes we
agree on *and* it should be possible to change/set their default values
with, for example, environment variables or command line options.

I call them infixes and not prefixes here, because applications like
virtualenv might want to define a different root for these files than
the system default one.

Tarek suggested to use this scheme like this:

setup(
    ...
    data_files = [
        ('$configuration', ['data/my.conf', 'data/default.conf']),
        ('$examples', ['doc/examples/do_foo.sample']),
        ('/i/want/file/here', ['custom_placed_file']),
        ...
        ]
    )

The placeholders will then be replaced with their current value at
installation time and the locations written to .egg-info/PREFIX.

This would allow/ease a couple of things, namely:

* automatic placement of typical files in the right locations
* much less work for packaging python applications on *nix distributions
  -> debhelper could take advantage of all these for example

* automatic tests/fixers for different file types
  
  I think of automatic lintian style tests on PyPi here that will reject
  a package if it does not conform to a standard or at least warns the user
  about it

* retrieval of certain types of data easily at run time

  Distribution.get_data_file('$configuration', 'my.cfg') -> file object

* ...

thanks!

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091111/4659658d/attachment.pgp>

From a.cavallo at acm.org  Wed Nov 11 17:22:41 2009
From: a.cavallo at acm.org (Antonio Cavallo)
Date: Wed, 11 Nov 2009 16:22:41 +0000
Subject: [Distutils] Fwd:  People want CPAN :-)
In-Reply-To: <a71153f0911090314m669da82crd5c6abda6798c7d9@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a71153f0911090314m669da82crd5c6abda6798c7d9@mail.gmail.com>
Message-ID: <a71153f0911110822q478d5f96s8e3b072afca60d34@mail.gmail.com>

I might be the only one that doesn't need CPAN here but ...
I'll try to explain the needs of a contractor working on some python
projects in a commercial environment:

1. I don't need a replacement for rpm/yast/yum/synaptic: they provide
file tracking and package resolution/dependency already and
? ?pepole are using them (and it has been for a very long time)

2. I use (and I'm failry happy) with bdist_wininst: is solves all my
problems in windows

3. As part of my duty I must record and track about licenses and
ownerships: I don't need a package build system that goes over
internet
? ?fetching things

4. I might be working as administrator (as normal in windows) and
doing the "CPAN" way will possibly overwrite packages upong which
other
? ?components are depending

I've used CPAN in the past and as system administrator I garantee you
is the last thing you'd like to have around (and any system like that
for that matter): there're are ALREADY platform standards, please
don't ignore them.

Regards,
Antonio Cavallo

PS. Why is that difficult a parallel install of packages under
site-packages/mypackage/X.Y.Z ?enforcing the X.Y.Z numbering?
Have I miss something? Am I too naive?


On Fri, Nov 6, 2009 at 5:53 PM, Guido van Rossum <guido at python.org> wrote:
> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?

From tepperly at llnl.gov  Wed Nov 11 18:46:25 2009
From: tepperly at llnl.gov (Tom Epperly)
Date: Wed, 11 Nov 2009 09:46:25 -0800
Subject: [Distutils] Looking for portable what to determine directory where
 extensions are installed?
Message-ID: <4AFAF871.9040004@llnl.gov>

[I am resending this because nobody responded to my first inquiry. I am 
hoping it just got overlooked in the discussion about CPAN because I am 
looking for the right solution for a bug in my tool that links Python to 
other languages.]

I work on a language interoperability tool, Babel
https://computation.llnl.gov/casc/components/components.html; and I need
a portable way to determine the directory where Python extension modules
are installed by distutils (i.e., lib or lib64) to resolve this issue:
https://www.cca-forum.org/bugs/babel/issue670

For example, I will invoke "python setup.py install
--prefix=/home/foo/_inst --exec-prefix=/home/foo/_inst", and it usually
installs the extensions in either
/home/foo/_inst/lib/python2.5/site-packages or
/home/foo/_inst/lib64/python2.5/site-packages. Initially, I did the
following to determine where the extensions actually got installed:

# assume exec_prefix=/home/foo/_inst
# assume python_version=`python -c 'import sys; print sys.version' | sed
'1s/^\(...\).*/\1/g;1q'`
RUNTIME_PYTHON="$exec_prefix/lib/python$python_verbose/site-packages"

This worked fine until I started running on 64-bit Python machines where
the correct result was

RUNTIME_PYTHON="$exec_prefix/lib64/python$python_verbose/site-packages"

The first 64-bit machine I was using seemed to have this patch:
http://bugs.python.org/file14726/Python-2.6.2-multilib.patch, so I
amended my script to the following:

pylib=`$PYTHON -c "import sys; print sys.__dict__.get('lib','lib')"`
RUNTIME_PYTHON="$exec_prefix/$pylib/python$python_version/site-packages"

However, this approach doesn't work with Fedora Core 12 prerelease or
some other 64-bit Linux distributions. They don't define "sys.lib".

I thought distutils.sysconfig.get_python_lib() might be helpful, but it
returns "/usr/lib/python2.6/site-packages" even on the system where
"/usr/lib64/python2.6/site-packages" is the right answer.

Is there a canonical, Pythonic way to determine whether an installation
of Python uses "lib" or "lib64"?  Or is there a way to determine the
full path of where distutils will install extensions relative to the
specified exec prefix?

Regards,

Tom Epperly



From ziade.tarek at gmail.com  Wed Nov 11 19:15:06 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 19:15:06 +0100
Subject: [Distutils] buildout/distribute recursion error fixed
In-Reply-To: <slrnhfisgo.1k22.reinout@isengrim.local>
References: <slrnhf53cf.tdr.reinout@isengrim.local>
	<94bdd2610911050358n4cb88665wfcf4825301182bc2@mail.gmail.com>
	<slrnhfisgo.1k22.reinout@isengrim.local>
Message-ID: <94bdd2610911111015q4f225d79j2e445b148d67b3e5@mail.gmail.com>

On Tue, Nov 10, 2009 at 3:00 PM, Reinout van Rees <reinout at vanrees.org> wrote:
[..]
>>
>> Before it's merged in trunk and eventually released, we need to :
>>
>> - add a test demonstrating this bug in easy_install.txt
>
> Buildout's test aren't setup to test this kind of things directly. ?I *did*
> add a test to easy_install.txt that tests whether the offending _get_dist()
> (which starts the recursion) gets called. ?This fails before my fix and
> succeeds afterwards.
>
> On Tarek's suggestion, I moved the test out of easy_install.txt to a separate
> file to keep the corner case test out of the user documentation.
>

Sounds enough,

>> - manually test the various scenarii (these are not fully covered with
>> zc.buildout unittests because the assmption is made that python don't
>> have setuptools installed):
>>
>> ? ? - nude python, buildout without --distribute, plone buildout cfg file
>> ? ? - python with setuptools, buildout witout --distribute, plone
>> buildout cfg file
>> ? ? - python with distribute < 0.6.8, buildout witout --distribute,
>> plone buildout cfg file
>> ? ? - these 3 tests *with* --distribute
>
> I did not test all the variants, that would cost me waaaay too much time for
> such a small fix. ?To be honest.
>
> I did reproduce it with setuptools + distribute with and without the fix with
> the development buildout itself. ?That executes the changed code point enough.

Can you merge it into trunk ?

Thanks

Tarek

From a.badger at gmail.com  Wed Nov 11 19:16:49 2009
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Wed, 11 Nov 2009 10:16:49 -0800
Subject: [Distutils] Looking for portable what to determine
	directory	where extensions are installed?
In-Reply-To: <4AFAF871.9040004@llnl.gov>
References: <4AFAF871.9040004@llnl.gov>
Message-ID: <20091111181649.GG25960@clingman.lan>

On Wed, Nov 11, 2009 at 09:46:25AM -0800, Tom Epperly wrote:
>
> I thought distutils.sysconfig.get_python_lib() might be helpful, but it
> returns "/usr/lib/python2.6/site-packages" even on the system where
> "/usr/lib64/python2.6/site-packages" is the right answer.
>
You're on the right track here.  You just need one more piece of
information:

distutils.sysconfig.get_python_lib() will return the path where pure python
extensions get installed.

distutils.sysconfig.get_python_lib(1) will return the path where compiled
extensions get installed.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091111/e1ff7891/attachment.pgp>

From wentland at cl.uni-heidelberg.de  Wed Nov 11 19:30:02 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Wed, 11 Nov 2009 19:30:02 +0100
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091111152414.GC7023@kinakuta.local>
References: <20091111152414.GC7023@kinakuta.local>
Message-ID: <20091111183002.GF7023@kinakuta.local>

On Wed, Nov 11, 2009 at 16:24 +0100, Wolodja Wentland wrote:

> The installation prefixes could for example be saved within a suitable
> file in .egg-info/ which would mandate changes to PEP 376. I could think
> of a PREFIX file and the following  methods/attribute for e.g. the
> Distribution class

I guess the pkgutil module is a much better place for these functions.
(Sorry, am not that familiar with all the PEPs yet)

> * get_data_file(type, name) -> get data file of given type and name (see

This could for example be easily implemented once
Distribution.get_installed_files(local=False) is done, if the
type of the files are noted somewhere.

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091111/de18b2db/attachment.pgp>

From tseaver at palladion.com  Wed Nov 11 19:34:50 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 11 Nov 2009 13:34:50 -0500
Subject: [Distutils] Looking for portable what to determine directory
 where extensions are installed?
In-Reply-To: <20091111181649.GG25960@clingman.lan>
References: <4AFAF871.9040004@llnl.gov> <20091111181649.GG25960@clingman.lan>
Message-ID: <hdf04a$4il$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Toshio Kuratomi wrote:
> On Wed, Nov 11, 2009 at 09:46:25AM -0800, Tom Epperly wrote:
>> I thought distutils.sysconfig.get_python_lib() might be helpful, but it
>> returns "/usr/lib/python2.6/site-packages" even on the system where
>> "/usr/lib64/python2.6/site-packages" is the right answer.
>>
> You're on the right track here.  You just need one more piece of
> information:
> 
> distutils.sysconfig.get_python_lib() will return the path where pure python
> extensions get installed.
> 
> distutils.sysconfig.get_python_lib(1) will return the path where compiled
> extensions get installed.

On my x86_64 Ubuntu box::

 $ uname -a
 Linux mred 2.6.27-15-generic #1 SMP Tue Oct 20 06:50:36 UTC 2009 \
   x86_64 GNU/Linux
 $ which python
 /usr/bin/python
 $ python -c "from distutils.sysconfig import get_python_lib; print
get_python_lib(); print get_python_lib(1); print get_python_lib(0, 1);
print get_python_lib(1, 1)"
 /usr/lib/python2.5/site-packages
 /usr/lib/python2.5/site-packages
 /usr/lib/python2.5
 /usr/lib/python2.5

which is indeed the right answer::

 $ ls -laF /usr/lib/python2.5
 total 7292
 drwxr-xr-x  22 root root  20480 2009-04-29 15:44 ./
 drwxr-xr-x 196 root root  77824 2009-10-25 21:41 ../
 -rw-r--r--   1 root root  33417 2008-10-05 16:59 aifc.py
 ...

even though it is the same answer on my 32-bit box::

 $ uname -a
 Linux scout 2.6.28-16-generic #55-Ubuntu SMP Tue Oct 20 19:48:24 UTC \
   2009 i686 GNU/Linux
 $ which python
 /usr/bin/python
 $ python -c "from distutils.sysconfig import get_python_lib; print
get_python_lib(); print get_python_lib(1); print get_python_lib(0, 1);
print get_python_lib(1, 1)"
 /usr/lib/python2.6/dist-packages
 /usr/lib/python2.6/dist-packages
 /usr/lib/python2.6
 /usr/lib/python2.6
 $ ls -laF /usr/lib/python2.6
 total 8424
 drwxr-xr-x  23 root root  20480 2009-04-20 10:00 ./
 drwxr-xr-x 230 root root  69632 2009-11-11 12:07 ../
 -rw-r--r--   1 root root  13625 2009-04-18 22:43 _abcoll.py
 ...

On a FreeBSD box:

 $ uname -a
 FreeBSD zap20 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Thu Jan  1 08:58:24 \
 UTC 2009     \
 root at driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC   amd64
 $ which python
 /usr/local/bin//python
 $ python -c "from distutils.sysconfig import get_python_lib; print
get_python_lib(); print get_python_lib(1); print get_python_lib(0, 1);
print get_python_lib(1, 1)"
 /usr/local/lib/python2.5/site-packages
 /usr/local/lib/python2.5/site-packages
 /usr/local/lib/python2.5
 /usr/local/lib/python2.5

I don't have access to a 64-bit box where the right answer would be
'/usr/lib64/python2.x'.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkr7A8UACgkQ+gerLs4ltQ5HcACdG8r2/LoOVhUFeaC15ZjY/VPq
qiEAoNNWpUR9tROkBhFSKVlFdMlEto0A
=Fu69
-----END PGP SIGNATURE-----


From sridharr at activestate.com  Wed Nov 11 20:32:56 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Wed, 11 Nov 2009 11:32:56 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
Message-ID: <op.u28wk6jj1q4jlt@whymac.activestate.com>

In line with this discussion, I found a document that details the aspects  
of CPAN that can be used for writing packaging systems in other languages.  
The author says: over the years people from at least Python, Ruby, and  
Java communities have approached me or other core CPAN people to ask  
basically "How did we do it?

   http://www.cpan.org/misc/ZCAN.html

-srid

On Fri, 06 Nov 2009 09:53:44 -0800, Guido van Rossum <guido at python.org>  
wrote:

> I just found this comment on my blog. People have told me this in
> person too, so I believe it is real pain (even if the solution may be
> elusive and the suggested solutions may not work). But I don't know
> how to improve the world. Is the work on distutils-sig going to be
> enough? Or do we need some other kind of work in addition? Do we need
> more than PyPI?
>
> --Guido
>
> ---------- Forwarded message ----------
> From: dalloliogm <noreply-comment at blogger.com>
> Date: Fri, Nov 6, 2009 at 8:01 AM
> Subject: [Neopythonic] New comment on Python in the Scientific World.
> To: gvanrossum at gmail.com
>
>
> dalloliogm has left a new comment on your post "Python in the Scientific  
> World":
>
> Python is suffering a lot in the scientific word, because it has not a
> CPAN-like repository.
>
> PyPI is fine, but it is still far from the level of CPAN, CRAN,
> Bioconductor, etc..
>
> Scientists who use programming usually have a lot of different
> interests and approaches, therefore it is really difficult to write a
> package that can be useful to everyone.
> Other programming language like Perl and R have repository-like
> structure which enable people to download packages easily, and to
> upload new ones and organize them withouth having to worry about
> having to integrate them into existing packages.
>
> This is what is happening to biopython now: it is a monolitic package
> that it is supposed to work for any bioinformatic problem; but this is
> so general that to accomplish that you would need to add a lot of
> dependencies, to numpy, networkx, suds, any kind of library.
> However, since easy_install is not as ready yet as the counterparts in
> other languages, if the biopython developers add too many
> dependencies, nobody will be able to install it properly, and nobody
> will use it.
>
>
>
> Posted by dalloliogm to Neopythonic at November 6, 2009 8:01 AM
>
>


From pav at iki.fi  Wed Nov 11 21:22:36 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Wed, 11 Nov 2009 22:22:36 +0200
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
Message-ID: <1257970956.4524.31.camel@idol>

ke, 2009-11-11 kello 16:47 +0100, Tarek Ziad? kirjoitti:
[clip]
> >> If it's the later, I guess you will be able to use the
> >> upcoming "sysconfig" module,
> >> that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.
> >
> > Where is the sysconfig sources ? I don't see it in bitbucket.
> 
> That's in python's svn, in a tarek_sysconfig branch. It's a revamp of
> distutils/sysconfig.py with the schemes from distutils/command/install.py
> (work in progress)

What if the user passes a different install prefix via

	python setup.py install --prefix=FOO

I believe David would like to know FOO here. Since sysconf is not a part
of distutils, will it know what FOO is?

[clip]
> > For me, one of the core idea of an improved distutils would be to make
> > this much easier. All compilers options form distutils would be in
> > simple data files with simple API, no objects, no class with countless
> > methods and complex protocol. Distutils v2 would have a default "dumb"
> > build tool, but you could use whatever tool instead if desired.
> 
> The default compiler class exists for that, it's CCompiler, and is
> mostly a placeholder for options.
> But it's C oriented.  In my mind, implementing a new compiler for
> Distutils means overriding it, and implementing, mainly:
> 
> - preprocess()
> - compile()
> - create_static_lib()
> - link()
> 
> Now that's quite complex, and we could probably have a single method
> (compile) that would do the required work of compiling an extension
> the way it wants to.

I think one question here is that how do the different compilers speak
to each other?

Consider the following chain needed to compile a Cython file linking to
some Fortran and C files to a Python module:

- cython foo.pyx -> foo.c
- compile foo.c -> foo.o
- compile bar.c -> bar.o
- compile quux.f90 -> quux.o
- link foo.o bar.o quux.o -> foo.so

This is a completely possible use-case, so it would be good if distutils
could handle it. Also, dependency handling would be nice here. Changing
bar.c or foo.pyx should trigger relinking foo.so etc.

> So, yes, being able to register an arbitrary compiler class, with
> arbitrary options passed through the Extension
> could make it simpler:
> 
> setup(
> ..
> ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
> ..)

Does this work easily out when there are multiple source files (in
different languages) for a single extension module?

Also, the option of determining the necessary compiler and compiler
options from file extensions does not always work. For example, Fortran
90 files come in two flavors, fixed-form and free-form, which often are
not differentiated by the file extension. However, they may require
different compiler flags to compile. (Hopefully, however, nobody uses
the fixed-form for F90 any more...)

Cheers,
-- 
Pauli Virtanen




From ziade.tarek at gmail.com  Wed Nov 11 21:42:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 21:42:32 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <1257970956.4524.31.camel@idol>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
Message-ID: <94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>

On Wed, Nov 11, 2009 at 9:22 PM, Pauli Virtanen <pav at iki.fi> wrote:
> ke, 2009-11-11 kello 16:47 +0100, Tarek Ziad? kirjoitti:
> [clip]
>> >> If it's the later, I guess you will be able to use the
>> >> upcoming "sysconfig" module,
>> >> that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.
>> >
>> > Where is the sysconfig sources ? I don't see it in bitbucket.
>>
>> That's in python's svn, in a tarek_sysconfig branch. It's a revamp of
>> distutils/sysconfig.py with the schemes from distutils/command/install.py
>> (work in progress)
>
> What if the user passes a different install prefix via
>
> ? ? ? ?python setup.py install --prefix=FOO
>
> I believe David would like to know FOO here. Since sysconf is not a part
> of distutils, will it know what FOO is?

I am not sure he wants FOO, I think he wants all installation paths,
that gets built
by the install command with the provided "FOO" root prefix.

that could be in pseudo code:

>>> get_install_paths('FOO')

And that's the API we want to add in sysconfig, roughly.


[..]
>
> I think one question here is that how do the different compilers speak
> to each other?
>
> Consider the following chain needed to compile a Cython file linking to
> some Fortran and C files to a Python module:
>
> - cython foo.pyx -> foo.c
> - compile foo.c -> foo.o
> - compile bar.c -> bar.o
> - compile quux.f90 -> quux.o
> - link foo.o bar.o quux.o -> foo.so
>
> This is a completely possible use-case, so it would be good if distutils
> could handle it. Also, dependency handling would be nice here. Changing
> bar.c or foo.pyx should trigger relinking foo.so etc.
>
>> So, yes, being able to register an arbitrary compiler class, with
>> arbitrary options passed through the Extension
>> could make it simpler:
>>
>> setup(
>> ..
>> ext_modules=[Extension('Foo', files=['foo.d'], compiler='pyd')],
>> ..)
>
> Does this work easily out when there are multiple source files (in
> different languages) for a single extension module?

Do you mean, an Extension that would require several compilers ?

I was thinking of a one-to-one relation between an Extension and a
compiler type,
even if there are are multiple source files (in different languages)
for this extension.

Meaning that *one* compiler would have to handle of those files. Does that fit
the cython use case ?

>
> Also, the option of determining the necessary compiler and compiler
> options from file extensions does not always work. For example, Fortran
> 90 files come in two flavors, fixed-form and free-form, which often are
> not differentiated by the file extension. However, they may require
> different compiler flags to compile. (Hopefully, however, nobody uses
> the fixed-form for F90 any more...)

How do they do then ? Is file extensions, inline options in setup.py,
and the environ, are enough ?

regards
Tarek

From greg.ewing at canterbury.ac.nz  Wed Nov 11 22:02:18 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Nov 2009 10:02:18 +1300
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
Message-ID: <4AFB265A.3000008@canterbury.ac.nz>

Tarek Ziad? wrote:

> If the answer to that solution is just: "Distutils sucks anyways..",
> it is not really helpfull imho..
> 
> I don't see the point to write Distutils from scratch, instead of
> making it evolve.

If you can see a way to get from the current distutils
code to something with a well-designed and well-documented
API and a clean implementation, that would be fine by
me.

When I say it's fundamentally broken, I'm really talking
about the API. My idea of what an API for a build system
should be like is more like make or scons, which slice
the functionality up in a completely orthogonal direction
to the way distutils does.

Maybe it would be possible to plug such a system in under
the existing build_ext class. I don't know. I think I
would like the same philosophy applied to other areas
of distutils, not just compiling extensions. Otherwise
it would feel like two incompatible systems bolted
together.

What we need right now, I think, is some discussion
about a new API, unconstrained by any considerations of
backwards compatibility or reuse of existing distutils
code. Once we know where we're going, then we can think
about the best way to get there.

-- 
Greg

From reinout at vanrees.org  Wed Nov 11 22:22:59 2009
From: reinout at vanrees.org (Reinout van Rees)
Date: Wed, 11 Nov 2009 21:22:59 +0000 (UTC)
Subject: [Distutils] buildout/distribute recursion error fixed
References: <slrnhf53cf.tdr.reinout@isengrim.local>
	<94bdd2610911050358n4cb88665wfcf4825301182bc2@mail.gmail.com>
	<slrnhfisgo.1k22.reinout@isengrim.local>
	<94bdd2610911111015q4f225d79j2e445b148d67b3e5@mail.gmail.com>
Message-ID: <hdf9vj$9jb$1@ger.gmane.org>

On 2009-11-11, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
> Can you merge it into trunk ?

Done!


Reinout

-- 
Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"


From greg.ewing at canterbury.ac.nz  Wed Nov 11 22:25:02 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Nov 2009 10:25:02 +1300
Subject: [Distutils] Build_ext refactoring (was: People want CPAN)
In-Reply-To: <c6b86a12ca0622d021303ba635577bfa@preisshare.net>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7788C.5090501@ar.media.kyoto-u.ac.jp> <hd85pn$4as$2@ger.gmane.org>
	<b654cd2e0911082128q1df810d9x4bdb2ed8e3107999@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<c6b86a12ca0622d021303ba635577bfa@preisshare.net>
Message-ID: <4AFB2BAE.6010004@canterbury.ac.nz>

David Lyon wrote:

> The parts that shouldn't be rewritten (because the people
> that wrote it were too clever - and the degree of difficulty
> to high) is any C interface.

Not exactly sure what you mean by that. If you mean the
knowledge of how to call the C compiler to compile an
extension module on various platforms, I agree that this
is important knowledge that should be preserved.

But if the only way to preserve it is to keep the actual
existing distutils code that implements it, we have a
problem, because that IMO is the part of distutils that
*most* needs improvement. It needs to be broken out of
the monolithic build_ext class somehow so that the
build_ext process can be extended more selectively.

> Evolution is better than revolution is better than nothing
> happening at all.

I don't agree that evolution is better in and of itself
than revolution. They're both means to an end -- getting
something better than we have now. The problem with
distutils is that evolution doesn't seem to have worked.
It has just grown randomly with no clear design and ended
up painting itself into a corner.

-- 
Greg


From reinout at vanrees.org  Wed Nov 11 22:30:26 2009
From: reinout at vanrees.org (Reinout van Rees)
Date: Wed, 11 Nov 2009 21:30:26 +0000 (UTC)
Subject: [Distutils] Fwd: installing lxml ? - todays user experience
	from python-list@python.org
References: <0d75c556f2db0cd28c5316be635d7f8f@preisshare.net>
	<hdeglj$8r5$1@ger.gmane.org>
Message-ID: <hdfadi$9jb$2@ger.gmane.org>

On 2009-11-11, Maurits van Rees <m.van.rees at zestsoftware.nl> wrote:
> David Lyon, on 2009-11-11:
>> $ sudo STATIC_DEPS=true /Library/Frameworks/Python.framework/Versions/
>> 2.6/bin/easy_install lxml
>> Password:
>> sudo: STATIC_DEPS=true: command not found
>
> On linux 'true' is a command:
> $ whatis true
> true (1)             - do nothing, successfully
>
> Apparently that is not available on Mac OS X.

The problem is that sudo expects a command. "STATIC_DEPS=true" is not a
command.

An "export STATIC_DEPS=true ought to be ok.


Building lxml is historically quite a pain on osx.  It keeps improving though.

The most sure-fire way of getting it to run is to use buildout in combination
with the staticlxml recipe at
http://pypi.python.org/pypi/z3c.recipe.staticlxml

I haven't seen a problem report about that in the last 6 months :-)



I agree with the sentiment of the original email that the initial impression
is, shall we say, sub-obtimal.  For me, this is one of the strong points of
buildout: once you buy into the general idea, it solves a whopping load of
problems for you.  Isolation, installation, setup.  And recipes like the
staticlxml one to solve those pesky weird corner cases!

reinout


-- 
Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org
Software developer at http://www.thehealthagency.com
"Military engineers build missiles. Civil engineers build targets"


From ziade.tarek at gmail.com  Wed Nov 11 22:35:43 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 22:35:43 +0100
Subject: [Distutils] Build_ext refactoring (was: People want CPAN)
In-Reply-To: <4AFB2BAE.6010004@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<c6b86a12ca0622d021303ba635577bfa@preisshare.net>
	<4AFB2BAE.6010004@canterbury.ac.nz>
Message-ID: <94bdd2610911111335r5d1d95c1n7124202575e7bce6@mail.gmail.com>

On Wed, Nov 11, 2009 at 10:25 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
[..]
> I don't agree that evolution is better in and of itself
> than revolution. They're both means to an end -- getting
> something better than we have now. The problem with
> distutils is that evolution doesn't seem to have worked.
> It has just grown randomly with no clear design and ended
> up painting itself into a corner.

We are making progress in the thread we are having with David C. in
the build_ext refactoring.
I am suggesting to keep the 'Build_ext refactoring' topic in that thread.

Tarek

From robert.kern at gmail.com  Wed Nov 11 22:36:20 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 15:36:20 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFB265A.3000008@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>	<87eio82mzh.fsf@benfinney.id.au>	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>	<4AFA5652.8080708@canterbury.ac.nz>	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
	<4AFB265A.3000008@canterbury.ac.nz>
Message-ID: <hdfaol$caj$1@ger.gmane.org>

On 2009-11-11 15:02 PM, Greg Ewing wrote:
> Tarek Ziad? wrote:
>
>> If the answer to that solution is just: "Distutils sucks anyways..",
>> it is not really helpfull imho..
>>
>> I don't see the point to write Distutils from scratch, instead of
>> making it evolve.
>
> If you can see a way to get from the current distutils
> code to something with a well-designed and well-documented
> API and a clean implementation, that would be fine by
> me.
>
> When I say it's fundamentally broken, I'm really talking
> about the API. My idea of what an API for a build system
> should be like is more like make or scons, which slice
> the functionality up in a completely orthogonal direction
> to the way distutils does.
>
> Maybe it would be possible to plug such a system in under
> the existing build_ext class. I don't know.

In fact, David has.

http://pypi.python.org/pypi/numscons/

 > I think I
 > would like the same philosophy applied to other areas
 > of distutils, not just compiling extensions. Otherwise
 > it would feel like two incompatible systems bolted
 > together.

It does feel something like that. The build system is just one of the problems 
with distutils' internals, in my experience. You can think of the rest of 
distutils as a little application framework for command line utilities. I think 
this framework simply fails to provide in very fundamental ways, like the 
"extend commands by subclassing" design pattern. That choice makes it 
fundamentally difficult to combine extensions together. I really don't see a way 
to evolve away from that (and believe me, over the last decade, I've tried). You 
just need to redesign the internals if you want to get away from that. You can't 
get from any point A to any point B by evolving in small steps that are 
functional (not to mention backwards compatible!) all the way through.

With all respect to Greg Ward and the rest of the original distutils authors, it 
was a fantastic improvement over the state of affairs ten years ago, but we've 
learned a lot about application frameworks and about building software since then.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ziade.tarek at gmail.com  Wed Nov 11 22:59:35 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 11 Nov 2009 22:59:35 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <hdfaol$caj$1@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
	<4AFB265A.3000008@canterbury.ac.nz> <hdfaol$caj$1@ger.gmane.org>
Message-ID: <94bdd2610911111359l7244e626n60a5cc86f1a8d45a@mail.gmail.com>

On Wed, Nov 11, 2009 at 10:36 PM, Robert Kern <robert.kern at gmail.com> wrote:
[..]
>
> It does feel something like that. The build system is just one of the
> problems with distutils' internals, in my experience. You can think of the
> rest of distutils as a little application framework for command line
> utilities. I think this framework simply fails to provide in very
> fundamental ways, like the "extend commands by subclassing" design pattern.
> That choice makes it fundamentally difficult to combine extensions together.
> I really don't see a way to evolve away from that (and believe me, over the
> last decade, I've tried). You just need to redesign the internals if you
> want to get away from that. You can't get from any point A to any point B by
> evolving in small steps that are functional (not to mention backwards
> compatible!) all the way through.

I am very surprised about this statement.

What did you tried for the paste decade and failed to do ? I hear some
complaints
since a week, but beside's David examples I didn't read any other
precise use cases.

We're looking through the build_ext use case, and we are making some
improvement on
the other thread. So why not doing this in other issues ?

Let's discuss your use case. And if it means adding new options to run
arbitrary commands like
post/pre hooks to a given command, to avoid subclassing an existing
command, let's do it.

And let's drop the backward compat issues in these discussions, so we
don't burn out
in details.

Tarek

From robert.kern at gmail.com  Wed Nov 11 23:22:55 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 16:22:55 -0600
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <94bdd2610911111359l7244e626n60a5cc86f1a8d45a@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>	<4AFA5652.8080708@canterbury.ac.nz>	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>	<4AFB265A.3000008@canterbury.ac.nz>
	<hdfaol$caj$1@ger.gmane.org>
	<94bdd2610911111359l7244e626n60a5cc86f1a8d45a@mail.gmail.com>
Message-ID: <hdfdfv$lbj$1@ger.gmane.org>

On 2009-11-11 15:59 PM, Tarek Ziad? wrote:
> On Wed, Nov 11, 2009 at 10:36 PM, Robert Kern<robert.kern at gmail.com>  wrote:
> [..]
>>
>> It does feel something like that. The build system is just one of the
>> problems with distutils' internals, in my experience. You can think of the
>> rest of distutils as a little application framework for command line
>> utilities. I think this framework simply fails to provide in very
>> fundamental ways, like the "extend commands by subclassing" design pattern.
>> That choice makes it fundamentally difficult to combine extensions together.
>> I really don't see a way to evolve away from that (and believe me, over the
>> last decade, I've tried). You just need to redesign the internals if you
>> want to get away from that. You can't get from any point A to any point B by
>> evolving in small steps that are functional (not to mention backwards
>> compatible!) all the way through.
>
> I am very surprised about this statement.
>
> What did you tried for the paste decade and failed to do ? I hear some
> complaints
> since a week, but beside's David examples I didn't read any other
> precise use cases.
>
> We're looking through the build_ext use case, and we are making some
> improvement on
> the other thread. So why not doing this in other issues ?
>
> Let's discuss your use case. And if it means adding new options to run
> arbitrary commands like
> post/pre hooks to a given command, to avoid subclassing an existing
> command, let's do it.

http://svn.scipy.org/svn/numpy/trunk/numpy/distutils/command/

All of it. Now consider that here we are also trying to play nicely with the 
setuptools extensions, and Pyrex, and David is working on integrating Cython 
support.

To get to one real specific problem, let's consider build_src. build_src is a 
new subcommand in numpy.distutils that builds C extension sources from other 
files. We use this to hook in f2py's wrapper generator and other more ad hoc 
forms of generating wrappers. When build_ext uses --inplace, we need build_src 
to use --inplace, too, because it will often output some "final" products in 
addition to intermediate wrapper sources. In order to integrate this with 
setuptools' develop command (which invokes build_ext --inplace but not build_src 
--inplace because setuptools knows nothing about numpy.distutils), we need to 
create a subclass of setuptool's develop command that will reinitialize 
build_src with the appropriate option. Then we need to conditionally place the 
develop command into the set of command classes so as not to introduce a 
setuptools dependency on those people who don't want to use it.

This is nuts. numpy.distutils shouldn't have to know anything about setuptools 
to accomplish this if the framework were properly designed. And this doesn't 
even get into the fact that many of the numpy.distutils command classes that are 
shared with setuptools are conditional subclasses and probably still buggily 
cobbled together.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From cournape at gmail.com  Thu Nov 12 00:03:54 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 08:03:54 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
Message-ID: <5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>

On Thu, Nov 12, 2009 at 5:42 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

>
> I am not sure he wants FOO, I think he wants all installation paths,
> that gets built
> by the install command with the provided "FOO" root prefix.
>
> that could be in pseudo code:
>
>>>> get_install_paths('FOO')
>
> And that's the API we want to add in sysconfig, roughly.

That does not solve the problem about getting FOO in the first place
when calling get_install_path.

Having a function which gives you all the build directories as well as
the install paths would already be great, though. Right now, for
numscons, I needed to reimplement all the logic, with the different
paths in different modes (inplace vs standard, Release vs Debug on
windows, etc...). Some of them are even python version dependent IIRC.

> Do you mean, an Extension that would require several compilers ?
>
> I was thinking of a one-to-one relation between an Extension and a
> compiler type,
> even if there are are multiple source files (in different languages)
> for this extension.

This cannot work - you need different tools at different stages. I
don't think any imperative method can work here. Note also that
extension alone is not enough to trigger the right tool: it should be
overridable on a per extension-basis (actually, it should be
overridable on a per source basis if needed).

I think compiler class and the likes should simply be removed of the
picture here. You need tasks to transform a source into a target, and
tasks would use compiler configuration. There should not be any
objects/classes for compilers, it it not flexible enough.  Although
the details differ in significant ways, both waf and scons use strings
for command lines (waf "compile them" for performance reason),
consisting of different parts which can be altered at will. In scons,
the task to compile C code is something like

$CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET

You need to be able to control those variable content in a very fine
grained manner: prepending and appending may lead to different
compiler behavior, for example. This is especially important when
linking, where the wrong order may be the different between a working
extension and a crashing extension.

You cannot obtain this with classes and objects (especially when you
start talking about performance: thousand of source files for one
extension is not a crazy usercase).

David

From cournape at gmail.com  Thu Nov 12 00:18:02 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 08:18:02 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people want
	CPAN)
Message-ID: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>

On Thu, Nov 12, 2009 at 6:59 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

>
> And let's drop the backward compat issues in these discussions, so we
> don't burn out
> in details.

That's the part I don't understand. If backward compatibility is not a
concern, why keeping distutils ? If you change the command and
Distribution class design, what remains of the original code  ? You
are changing the API and the implementation (which are quite tangled
with each other in distutils case), almost none of the original code
would remain.

It really feels to me like you are getting the pain of backward
compatibility without the gains. What am I missing ?

David

From david.lyon at preisshare.net  Thu Nov 12 00:17:21 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 11 Nov 2009 18:17:21 -0500
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <4AFB265A.3000008@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
	<4AFB265A.3000008@canterbury.ac.nz>
Message-ID: <aa00820b84d48ebc4aa1ea02a1b0f891@preisshare.net>


Hi Greg,

On Thu, 12 Nov 2009 10:02:18 +1300, Greg Ewing
<greg.ewing at canterbury.ac.nz>
wrote:

> What we need right now, I think, is some discussion
> about a new API, unconstrained by any considerations of
> backwards compatibility ....

A new API isn't so hard, but like anything it takes time
and effort to do.

Already this year, *a lot* of discussion went into such
a thing. For example, doing metadata based installs as
an option to traditional installs.

Backwards compatibility is going to be an issue that
*does* need to be addressed but even that I don't believe
is so hard.

There's two halves to "backwards" compatability. At
the build side and at the install side. It's confusing
I know.

Where distutils/setup is confusing is at the build
side in the way that all the files have to be specified
for being picked up.
 
On the install side, setup.py is much simpler and
hardly* does anything at all except copy files in
from the sources.txt. So compatability there is easier.

It's my opinion that a boilerplate setup.py could
completely replace 9/10 user written setup.py files.

> When I say it's fundamentally broken, I'm really talking
> about the API. My idea of what an API for a build system
> should be like is more like make or scons, which slice
> the functionality up in a completely orthogonal direction
> to the way distutils does.

yes. Broken on the build side.

> Maybe it would be possible to plug such a system in under
> the existing build_ext class. 

For even that to happen, there needs to be some work done.
As in fleshing out some code and examples and documentation.

Here's a configuration file for an alternate build system
that I'm working on at:

   http://bitbucket.org/djlyon/pypi-package-builder

Any help thrown at getting this working would be welcomed.

Regards

David

----setup.cfg--------------------
[setup]
name = artistflair
version = 1.2
description = artistflair is a package written in
    spare time to show colour variations.

[dependencies]
packages = pyopengl
    objectrelationalmapper >= 1.6

[dependencies python<=2.4]
packages = lxml >= 2.5

[dependencies windows]
packages = win32com

[dependencies linux]
packages = pyodbc

[Application]
scripts = artisticflairgui.py

[configuration]
files = artisticflair.conf

[datafiles]
files = artisticdb.db
directory = db

[Documentation]
directory = docs
url = index.html

[preinstall]
script = checksystem.py

[postinstall linux]
script = builddocumentation.py

[uninstall]
script = uninstallcleanup.py
---------------------------------





 


From robert.kern at gmail.com  Thu Nov 12 00:43:11 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 17:43:11 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
Message-ID: <hdfi6h$4d8$1@ger.gmane.org>

On 2009-11-11 17:18 PM, David Cournapeau wrote:
> On Thu, Nov 12, 2009 at 6:59 AM, Tarek Ziad?<ziade.tarek at gmail.com>  wrote:
>
>>
>> And let's drop the backward compat issues in these discussions, so we
>> don't burn out
>> in details.
>
> That's the part I don't understand. If backward compatibility is not a
> concern, why keeping distutils ? If you change the command and
> Distribution class design, what remains of the original code  ? You
> are changing the API and the implementation (which are quite tangled
> with each other in distutils case), almost none of the original code
> would remain.
>
> It really feels to me like you are getting the pain of backward
> compatibility without the gains. What am I missing ?

I think Tarek wants to avoid the Second System Effect and related problems.

   http://en.wikipedia.org/wiki/Second-system_effect
   http://www.joelonsoftware.com/articles/fog0000000069.html

While that is usually a good habit to cultivate and a good default position, 
it's not an unyielding law or anything. You have think deeply about whether the 
code is the way it is because it contains useful knowledge or if it is just 
constrained by ossified decisions from the past. I tend to think that the useful 
knowledge can be extracted from distutils and applied well in a rewrite. The 
most important useful knowledge is the extension building flags, and I think you 
have done a good job of transplanting that information into the entirely 
different build system of SCons.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ziade.tarek at gmail.com  Thu Nov 12 00:48:39 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 00:48:39 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
Message-ID: <94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>

On Thu, Nov 12, 2009 at 12:03 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Thu, Nov 12, 2009 at 5:42 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>>
>> I am not sure he wants FOO, I think he wants all installation paths,
>> that gets built
>> by the install command with the provided "FOO" root prefix.
>>
>> that could be in pseudo code:
>>
>>>>> get_install_paths('FOO')
>>
>> And that's the API we want to add in sysconfig, roughly.
>
> That does not solve the problem about getting FOO in the first place
> when calling get_install_path.

Why that ? where "FOO" comes from ? if it's an option you provide at
build time like you said,
earlier, you just pass it to the API to get the paths.

FOO is not coming from nowhere...


[..]
>
>> Do you mean, an Extension that would require several compilers ?
>>
>> I was thinking of a one-to-one relation between an Extension and a
>> compiler type,
>> even if there are are multiple source files (in different languages)
>> for this extension.
>
> This cannot work - you need different tools at different stages. I
> don't think any imperative method can work here. Note also that
> extension alone is not enough to trigger the right tool: it should be
> overridable on a per extension-basis (actually, it should be
> overridable on a per source basis if needed).
>
> I think compiler class and the likes should simply be removed of the
> picture here. You need tasks to transform a source into a target, and
> tasks would use compiler configuration. There should not be any
> objects/classes for compilers, it it not flexible enough.
> ?Although
> the details differ in significant ways, both waf and scons use strings
> for command lines (waf "compile them" for performance reason),
> consisting of different parts which can be altered at will. In scons,
> the task to compile C code is something like
>
> $CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET
>
> You need to be able to control those variable content in a very fine
> grained manner: prepending and appending may lead to different
> compiler behavior, for example. This is especially important when
> linking, where the wrong order may be the different between a working
> extension and a crashing extension.
>
> You cannot obtain this with classes and objects (especially when you
> start talking about performance: thousand of source files for one
> extension is not a crazy usercase).

Sorry, I am getting confused here. This is getting all mixed up.
What OOP has to do with performance in the first place ?

OOP is useful to describe patterns and resuability, and that's what
Distutils does.
OOP is not a bottleneck here for speed. (everything is an object in
Python anyways)

Now we are saying that the compiler pattern in Distutils doesn't fit
some requirement,
fine, let's see what other design we can have.

But that will be based on OOP with classes, and objects, and there
ought to be an object that
orchestrates the building of one given extension at the end.

That might be the existing Extension class. This class is instanciated
with a name for  the
extension, and a list of files.

Here's a proposal to restrict the scope: let's drop the concept of
"compiler class" and let's work only with
the Extension class.

This class is used by a distutils command to build a given extension.
Let's say it is standalone, and it has a unique "build()"
method. Could we take it from here and try to prototype this Extension class ?

Tarek

From ziade.tarek at gmail.com  Thu Nov 12 01:04:17 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 01:04:17 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
Message-ID: <94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>

On Thu, Nov 12, 2009 at 12:18 AM, David Cournapeau <cournape at gmail.com> wrote:
> On Thu, Nov 12, 2009 at 6:59 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>>
>> And let's drop the backward compat issues in these discussions, so we
>> don't burn out
>> in details.
>
> That's the part I don't understand. If backward compatibility is not a
> concern, why keeping distutils ? If you change the command and
> Distribution class design, what remains of the original code ?? You
> are changing the API and the implementation (which are quite tangled
> with each other in distutils case), almost none of the original code
> would remain.
>
> It really feels to me like you are getting the pain of backward
> compatibility without the gains. What am I missing ?

What you are missing is that :

- you are convinced that distutils should be written from scratch. I
am not for many reasons. Some others are not either.
  it won't happen. the only thing that could make it happen is the
replacement of distutils by another tool that
  is used by the majority of the community for several years.

- you are convinced that the design is flawed and should be changed. I
partially agree. And I also think this can happen
  in distutils, at a slow pace. but not in the way you've described it.

So, instead on jumping in *your* conclusions (==let's drop distutils)
or in *mine*,
and in order to make some progress together, I am suggesting that we
discuss the design flaws you've mentioned.

And see what we come up with, then refocus on the big picture later
(that is backward compat etc..)

Working on the build_ext and Extensions part with your use case is
where we can share some knowledge imho.

Tarek

From robert.kern at gmail.com  Thu Nov 12 01:08:20 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 18:08:20 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
Message-ID: <hdfjlk$6km$1@ger.gmane.org>

On 2009-11-11 18:04 PM, Tarek Ziad? wrote:
> On Thu, Nov 12, 2009 at 12:18 AM, David Cournapeau<cournape at gmail.com>  wrote:
>> On Thu, Nov 12, 2009 at 6:59 AM, Tarek Ziad?<ziade.tarek at gmail.com>  wrote:
>>
>>>
>>> And let's drop the backward compat issues in these discussions, so we
>>> don't burn out
>>> in details.
>>
>> That's the part I don't understand. If backward compatibility is not a
>> concern, why keeping distutils ? If you change the command and
>> Distribution class design, what remains of the original code  ? You
>> are changing the API and the implementation (which are quite tangled
>> with each other in distutils case), almost none of the original code
>> would remain.
>>
>> It really feels to me like you are getting the pain of backward
>> compatibility without the gains. What am I missing ?
>
> What you are missing is that :
>
> - you are convinced that distutils should be written from scratch. I
> am not for many reasons. Some others are not either.
>    it won't happen. the only thing that could make it happen is the
> replacement of distutils by another tool that
>    is used by the majority of the community for several years.

That's basically what David is suggesting when he says "rewrite". He doesn't 
mean to replace distutils with another package named "distutils".

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ziade.tarek at gmail.com  Thu Nov 12 01:19:52 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 01:19:52 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <hdfdfv$lbj$1@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
	<4AFB265A.3000008@canterbury.ac.nz> <hdfaol$caj$1@ger.gmane.org>
	<94bdd2610911111359l7244e626n60a5cc86f1a8d45a@mail.gmail.com>
	<hdfdfv$lbj$1@ger.gmane.org>
Message-ID: <94bdd2610911111619v147f429cj878aea4638da46e8@mail.gmail.com>

On Wed, Nov 11, 2009 at 11:22 PM, Robert Kern <robert.kern at gmail.com> wrote:
[..]
> To get to one real specific problem, let's consider build_src. build_src is
> a new subcommand in numpy.distutils that builds C extension sources from
> other files. We use this to hook in f2py's wrapper generator and other more
> ad hoc forms of generating wrappers. When build_ext uses --inplace, we need
> build_src to use --inplace, too, because it will often output some "final"
> products in addition to intermediate wrapper sources. In order to integrate
> this with setuptools' develop command (which invokes build_ext --inplace but
> not build_src --inplace because setuptools knows nothing about
> numpy.distutils), we need to create a subclass of setuptool's develop
> command that will reinitialize build_src with the appropriate option. Then
> we need to conditionally place the develop command into the set of command
> classes so as not to introduce a setuptools dependency on those people who
> don't want to use it.
>
> This is nuts.

This clearly indicates that we should be able to extend build_ext
behaviour without subclassing it.
And having the ability to drive a specific compiling from within an
Extension subclass can solve this issue.


> numpy.distutils shouldn't have to know anything about
> setuptools to accomplish this if the framework were properly designed. And
> this doesn't even get into the fact that many of the numpy.distutils command
> classes that are shared with setuptools are conditional subclasses and
> probably still buggily cobbled together.

Same goes with "install", and I've proposed in the past the ability to
run arbitrary commands as pre/post hooks for it.
So we can "configure" this command instead of replacing it.

From ziade.tarek at gmail.com  Thu Nov 12 01:48:40 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 01:48:40 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdfjlk$6km$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
Message-ID: <94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>

On Thu, Nov 12, 2009 at 1:08 AM, Robert Kern <robert.kern at gmail.com> wrote:
[...]
>>
>> - you are convinced that distutils should be written from scratch. I
>> am not for many reasons. Some others are not either.
>> ? it won't happen. the only thing that could make it happen is the
>> replacement of distutils by another tool that
>> ? is used by the majority of the community for several years.
>
> That's basically what David is suggesting when he says "rewrite". He doesn't
> mean to replace distutils with another package named "distutils".

Well, good luck with that new project then. I am here to improve the
existing Distutils.

And that doesn't forbid us to do some redesign in it.

e.g. unlike this thread title implies, improvement of an existing
system is not opposed to its partial redesign
so it answers to some use cases it was not working well with before.

Now the partial redesign of the build_ext/Extension part is something
we can work on together.

The problem of command extensibility is also a topic we have been
discussed last year at Pycon,
and if you read the wiki you'll see that I have proposed some changes
so any command
could be extended in a pluggable way.

But this has to be added as new features, on the top of something that
already works.

You have to consider the PoV of many, many developers that happily use
Distutils every day.
They don't care about the edge cases we are discussing right now. They
wouldn't mind having
some improvements that would solve some edge cases of course.

But they will not understand why Distutils is dropped in favor of a
tool we write from scratch, because
its design doesnt' work well when the FooBar compiler cannot get the
YuYu option at the right time.

And even if it happened, that would mean waiting for its maturity
outside the stdlib for some years.

I want to improve Distutils, but not for the price of a complete drop.
I don't think the edge cases we
are discussing worth it, and I still fail to see why we can't work
them out in the context of the existing
tool.

Now, maybe that's just because I didn't spent a decade on the monster
like you did. I have +6 more years to
go ;)

Tarek

From robert.kern at gmail.com  Thu Nov 12 01:59:19 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 18:59:19 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
Message-ID: <hdfml9$fbp$1@ger.gmane.org>

On 2009-11-11 18:48 PM, Tarek Ziad? wrote:

> I want to improve Distutils, but not for the price of a complete drop.
> I don't think the edge cases we
> are discussing worth it, and I still fail to see why we can't work
> them out in the context of the existing
> tool.

Mostly because I'm entirely uninterested in helping you make incremental 
improvements that are going to break all the hard work we've already done just 
to get things working as it is. I find that prospect incredibly frustrating.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From ziade.tarek at gmail.com  Thu Nov 12 02:14:17 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 02:14:17 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdfml9$fbp$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
Message-ID: <94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>

On Thu, Nov 12, 2009 at 1:59 AM, Robert Kern <robert.kern at gmail.com> wrote:
> On 2009-11-11 18:48 PM, Tarek Ziad? wrote:
>
>> I want to improve Distutils, but not for the price of a complete drop.
>> I don't think the edge cases we
>> are discussing worth it, and I still fail to see why we can't work
>> them out in the context of the existing
>> tool.
>
> Mostly because I'm entirely uninterested in helping you make incremental
> improvements that are going to break all the hard work we've already done
> just to get things working as it is. I find that prospect incredibly
> frustrating.

I can understand that. I am very frustrated too because in the last
threads, whenever we are speaking
about design, it seems that on your side and David side, dropping
Distutils seems like a post-condition
to everything. Even if I have made some proposals on some concrete
design changes.

Who's "we" by the way ? The Scons project ? or the numpy.distutils project ?

Tarek

From greg.ewing at canterbury.ac.nz  Thu Nov 12 02:17:26 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Nov 2009 14:17:26 +1300
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <hdfaol$caj$1@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<69d337f7d047877fc48710cb7d17ac14@preisshare.net>
	<87eio82mzh.fsf@benfinney.id.au>
	<b654cd2e0911082350s5b40e2e5rc65427c7fa7fa6d0@mail.gmail.com>
	<4AF7D01F.7090202@ar.media.kyoto-u.ac.jp>
	<94bdd2610911091451t35eab6d3q9366ffb9c6264ba5@mail.gmail.com>
	<4AF8C27B.20807@ar.media.kyoto-u.ac.jp>
	<94bdd2610911101017n2b2de7e9m2a19ae462963ca1d@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<94bdd2610911110109v76b3342ew7726b2923728a904@mail.gmail.com>
	<4AFB265A.3000008@canterbury.ac.nz> <hdfaol$caj$1@ger.gmane.org>
Message-ID: <4AFB6226.3070504@canterbury.ac.nz>

Robert Kern wrote:
> With all respect to Greg Ward and the rest of the original distutils 
> authors, it was a fantastic improvement over the state of affairs ten 
> years ago, but we've learned a lot about application frameworks and 
> about building software since then.

I think we knew more about it *before* then as
well. Make has been around for a lot longer, but
distutils ignores everything we've learned from
it.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Nov 12 02:48:46 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Nov 2009 14:48:46 +1300
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
Message-ID: <4AFB697E.5030803@canterbury.ac.nz>

Tarek Ziad? wrote:

> On Thu, Nov 12, 2009 at 12:03 AM, David Cournapeau <cournape at gmail.com> wrote:

>> You cannot obtain this with classes and objects (especially when you
>> start talking about performance: thousand of source files for one
>> extension is not a crazy usercase).
> 
> Sorry, I am getting confused here. This is getting all mixed up.
> What OOP has to do with performance in the first place ?

As far as I can tell David seems to be saying that
instantiating a class for every file in the system
would be too much overhead.

I'm not convinced about that -- make builds a dependency
graph with a node for every target before it goes
to work, and I think scons does something similar
(although I might be wrong, I haven't looked at
scons very closely yet).

 >> $CC $CFLAGS $CPPDEFINES $CPPPATH -c $SOURCE $TARGET
 >>
 >> You need to be able to control those variable content in a very fine
 >> grained manner: prepending and appending may lead to different
 >> compiler behavior

It's true that the current distutils compiler classes
are lacking in flexibility here -- there's an option for
"extra flags", for example, but it sticks them all at
the end of the command. Sometimes that's wrong -- I've
had trouble trying to use it for MacOSX -F arguments,
for example.

But that's not to say that a class couldn't be devised
that allowed the required flexibility without degenerating
into just a string with textual substitutions.

-- 
Greg

From robert.kern at gmail.com  Thu Nov 12 04:04:39 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 21:04:39 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
Message-ID: <hdfu07$un4$1@ger.gmane.org>

Tarek Ziad? wrote:
> On Thu, Nov 12, 2009 at 1:59 AM, Robert Kern <robert.kern at gmail.com> wrote:
>> On 2009-11-11 18:48 PM, Tarek Ziad? wrote:
>>
>>> I want to improve Distutils, but not for the price of a complete drop.
>>> I don't think the edge cases we
>>> are discussing worth it, and I still fail to see why we can't work
>>> them out in the context of the existing
>>> tool.
>> Mostly because I'm entirely uninterested in helping you make incremental
>> improvements that are going to break all the hard work we've already done
>> just to get things working as it is. I find that prospect incredibly
>> frustrating.
> 
> I can understand that. I am very frustrated too because in the last
> threads, whenever we are speaking
> about design, it seems that on your side and David side, dropping
> Distutils seems like a post-condition
> to everything. Even if I have made some proposals on some concrete
> design changes.

In our considered opinion, piecemeal changes probably aren't going to solve the 
significant problems that we face. At best, they simply aren't going to help; we 
wouldn't be able to use the new features until we can drop support for Python 
2.6. numpy and scipy still need to support Python 2.4. At worst, they would 
introduce incompatibilities that we will have to work around somehow.

> Who's "we" by the way ? The Scons project ? or the numpy.distutils project ?

numpy and scipy. While I hesitate to speak for an entire community, I must say 
that David and my opinions on distutils are shared by a good portion of our 
community that has to deal with building and packaging.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From glyph at twistedmatrix.com  Thu Nov 12 05:16:57 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Wed, 11 Nov 2009 23:16:57 -0500
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdfu07$un4$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
Message-ID: <A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>


On Nov 11, 2009, at 10:04 PM, Robert Kern wrote:

> In our considered opinion, piecemeal changes probably aren't going to solve the significant problems that we face. At best, they simply aren't going to help; we wouldn't be able to use the new features until we can drop support for Python 2.6.

While I can understand your frustration, it's important to step back and think about whether your problems are really impossible to solve.  In particular, I take issue with this idea that you can't work on stuff that's distributed with Python but depend on newer versions.  Isn't the whole point of much of setuptools' complexity supposed to be the fact that you can have side-by-side multi-version installations?  Even assuming that this functionality doesn't work at _all_, who is to say that you can't ask users to upgrade distutils?  or do a --prefix installation of distutils into a different directory?  or have a build-time option that installs the 'distutils' package as 'distutils_plus_plus' and rewrites things as necessary?  or implement an alternative to require() which *does* work? or, assuming require() works for some cases but not yours, adapt it to your needs?

There are probably a dozen other ways that you *could* work on distutils and benefit more immediately from your efforts than the next Python release.  To think otherwise is a simply a failure of imagination.  Now, if you think it's *too hard* to do that, it might be interesting to hear why you think that, and what exactly the effort would be; a nebulous assertion that it's just too hard and we should throw our hands up (while I can definitely understand the impulse to make such an assertion) serves only to discourage everyone.

The fact that a package is in the standard library is not a death sentence.  Releases can be made separately.  Heck, if you are doing good work on trunk but the release cycles are taking too long, quite frequently distributors will make packages out of your code at some revision of trunk rather than a release.  I maintained software for _years_ that required a more recent version of pysqlite bindings than were available in the standard library's 'sqlite3' module.  The 'pysqlite2' project is alive and well, and we didn't have any significant problems.

Now, as Tarek suggests, it would be more worthwhile to discuss the *specifics* of the problems that you assert require blowing up the world, as more detailed understanding of those specifics will allow both people who want rewrites *and* people who want incremental improvements to proceed with better-informed.  Any language environment's package/distribute/build/install/run pipeline is complicated enough that one can have a lot of productive discussion just nailing down exactly what is wrong with it, before even talking about solutions, and Python is no exception.


From ellisonbg.net at gmail.com  Thu Nov 12 05:29:01 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 11 Nov 2009 20:29:01 -0800
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdfu07$un4$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
Message-ID: <6ce0ac130911112029w2f36d496xb8d7f75fcdd6a55b@mail.gmail.com>

>
>  Who's "we" by the way ? The Scons project ? or the numpy.distutils project
>> ?
>>
>
> numpy and scipy. While I hesitate to speak for an entire community, I must
> say that David and my opinions on distutils are shared by a good portion of
> our community that has to deal with building and packaging.
>
>
+1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091111/4072555f/attachment.htm>

From david at ar.media.kyoto-u.ac.jp  Thu Nov 12 05:30:21 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Nov 2009 13:30:21 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <4AFB697E.5030803@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<loom.20091111T124450-168@post.gmane.org>	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>	<1257970956.4524.31.camel@idol>	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
Message-ID: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>

Greg Ewing wrote:
> As far as I can tell David seems to be saying that
> instantiating a class for every file in the system
> would be too much overhead.
>
> I'm not convinced about that -- make builds a dependency
> graph with a node for every target before it goes
> to work, and I think scons does something similar

Yes, and scons has scalability problems because of this (both from CPU
and memory POV).

Waf has also an object per node, but it is aggressively optimized. If
you are interested in the details, I can point you to the corresponding
discussions in both scons and waf, where the main designers were involved.

> But that's not to say that a class couldn't be devised
> that allowed the required flexibility without degenerating
> into just a string with textual substitutions.

I am not saying that's impossible, but both waf an scons use string
susbtitution to do it (they do it differently though). So you would have
to find a good reason not to do it IMHO.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Thu Nov 12 06:02:56 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Nov 2009 14:02:56 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
 want CPAN)
In-Reply-To: <A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
Message-ID: <4AFB9700.4030207@ar.media.kyoto-u.ac.jp>

Glyph Lefkowitz wrote:
> There are probably a dozen other ways that you *could* work on distutils and benefit more immediately from your efforts than the next Python release.  To think otherwise is a simply a failure of imagination.  Now, if you think it's *too hard* to do that, it might be interesting to hear why you think that, and what exactly the effort would be; a nebulous assertion that it's just too hard and we should throw our hands up (while I can definitely understand the impulse to make such an assertion) serves only to discourage everyone.
>   

I am trying to understand what is 'nebulous' about our claims. We have
given plenty of hard and concrete examples of things which are
problematic in distutils. The major progress in our build issues have
been achieved by dropping distutils. Up to now, the only people who have
claimed that distutils can solve our problems are the people who are not
involved at all with our projects, and the people who claim distutils
cannot solve our problems are the people involved with it. That's odd to
say the least.

Now, I am ready to accept that we are missing the big picture and the
rest  of the community knows more about it. But I certainly have not
seen strong arguments to believe it so far.

cheers,

David

From mihamina at gulfsat.mg  Thu Nov 12 06:27:27 2009
From: mihamina at gulfsat.mg (Rakotomandimby Mihamina)
Date: Thu, 12 Nov 2009 08:27:27 +0300
Subject: [Distutils] Fwd:  People want CPAN :-)
In-Reply-To: <a71153f0911110822q478d5f96s8e3b072afca60d34@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<a71153f0911090314m669da82crd5c6abda6798c7d9@mail.gmail.com>
	<a71153f0911110822q478d5f96s8e3b072afca60d34@mail.gmail.com>
Message-ID: <4AFB9CBF.8020004@gulfsat.mg>

11/11/2009 07:22 PM, Antonio Cavallo::
> 1. I don't need a replacement for rpm/yast/yum/synaptic: they provide
> file tracking and package resolution/dependency already and
>     pepole are using them (and it has been for a very long time)

Agreed.

-- 
       Architecte Informatique chez Blueline/Gulfsat:
    Administration Systeme, Recherche & Developpement
                                    +261 33 11 207 36

From regebro at gmail.com  Thu Nov 12 06:28:53 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 12 Nov 2009 06:28:53 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
Message-ID: <319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>

2009/11/12 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
> I am trying to understand what is 'nebulous' about our claims. We have
> given plenty of hard and concrete examples of things which are
> problematic in distutils. The major progress in our build issues have
> been achieved by dropping distutils. Up to now, the only people who have
> claimed that distutils can solve our problems are the people who are not
> involved at all with our projects, and the people who claim distutils
> cannot solve our problems are the people involved with it. That's odd to
> say the least.
>
> Now, I am ready to accept that we are missing the big picture and the
> rest ?of the community knows more about it. But I certainly have not
> seen strong arguments to believe it so far.

Look, there is only one way this argument can be solved, and that is
by building something better than distutils. Honestly. Now, I'm sure
as heck not going to spend time on that, and Tarek don't think it's a
good idea, so it's up to you guys.

So I again ask:

1. Do you think the new PEPs in development should be followed? In
that case, what is the benefit of rewriting, instead of fixing?

2. When are you done?

Bitching that distutils needs to be scratched and rewritten is not
going to help. You need to DO it.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From david at ar.media.kyoto-u.ac.jp  Thu Nov 12 06:08:53 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Nov 2009 14:08:53 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
 want CPAN)
In-Reply-To: <94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
Message-ID: <4AFB9865.5090400@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
> I want to improve Distutils, but not for the price of a complete drop.
> I don't think the edge cases we
> are discussing worth it, and I still fail to see why we can't work
> them out in the context of the existing
> tool.
>   

Well, the scientific python edge case is what Guido asked about in the
original thread. That's our answer: we don't think something which is
not fundamentally different than distutils will solve our problems. If
you think that what we considered as fundamental issues are corner cases
and implementation details, we can only agree to disagree, I guess.

Now, it is legitimate to dismiss our use-cases as edge cases and not
worthwhile to worry about. We are not certainly not as big as say the
web development community. But the question was directed at us, as I
understood it :)

cheers,

David

From robert.kern at gmail.com  Thu Nov 12 06:45:22 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 11 Nov 2009 23:45:22 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
Message-ID: <hdg7dj$hek$1@ger.gmane.org>

Lennart Regebro wrote:
> 2009/11/12 David Cournapeau <david at ar.media.kyoto-u.ac.jp>:
>> I am trying to understand what is 'nebulous' about our claims. We have
>> given plenty of hard and concrete examples of things which are
>> problematic in distutils. The major progress in our build issues have
>> been achieved by dropping distutils. Up to now, the only people who have
>> claimed that distutils can solve our problems are the people who are not
>> involved at all with our projects, and the people who claim distutils
>> cannot solve our problems are the people involved with it. That's odd to
>> say the least.
>>
>> Now, I am ready to accept that we are missing the big picture and the
>> rest  of the community knows more about it. But I certainly have not
>> seen strong arguments to believe it so far.
> 
> Look, there is only one way this argument can be solved, and that is
> by building something better than distutils. Honestly. Now, I'm sure
> as heck not going to spend time on that, and Tarek don't think it's a
> good idea, so it's up to you guys.

I think you're missing the point of our statements. We're not asking you to work 
or not work on anything. Guido asked, in response to a comment from our 
community: "Is the work on distutils-sig going to be enough?" And our answer is 
that no, it's probably not. It's not addressing our most significant problems. 
David's work, e.g. on numscons, is helping, but it is still constrained by the 
requirement of working within distutils' framework.

> Bitching that distutils needs to be scratched and rewritten is not
> going to help. You need to DO it.

Pardon us for sincerely answering the questions asked of us.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From cournape at gmail.com  Thu Nov 12 06:46:26 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 14:46:26 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
Message-ID: <5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>

On Thu, Nov 12, 2009 at 2:28 PM, Lennart Regebro <regebro at gmail.com> wrote:

> Bitching that distutils needs to be scratched and rewritten is not
> going to help. You need to DO it.

Nobody asked you (or anyone else) to do anything. Guido asked a
question, we answered with our rationale.

There is no need to be rude.

David

From regebro at gmail.com  Thu Nov 12 06:59:04 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 12 Nov 2009 06:59:04 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
Message-ID: <319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>

2009/11/12 David Cournapeau <cournape at gmail.com>:
> On Thu, Nov 12, 2009 at 2:28 PM, Lennart Regebro <regebro at gmail.com> wrote:
>
>> Bitching that distutils needs to be scratched and rewritten is not
>> going to help. You need to DO it.
>
> Nobody asked you (or anyone else) to do anything. Guido asked a
> question, we answered with our rationale.
>
> There is no need to be rude.

It was not my intention to be rude, and I don't believe I was. This
discussion has been ongoing for a long time, and I have just recently
asked questions (which of course has been ignored) about this. This is
a discussion that has been going on for a long time, more than a year,
at least, yet noone who says distutils needs to be scratched and
something else written does something about it.

I've pointed this out repeatedly, and been ignored. Perhaps I used to
strong language this time, but on the other hand, I wasn't ignored
this time.

The discussion is pointless. If you want something better than distutils, do it.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From robert.kern at gmail.com  Thu Nov 12 07:01:35 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Nov 2009 00:01:35 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
Message-ID: <hdg8c0$iuc$1@ger.gmane.org>

Lennart Regebro wrote:
> 2009/11/12 David Cournapeau <cournape at gmail.com>:
>> On Thu, Nov 12, 2009 at 2:28 PM, Lennart Regebro <regebro at gmail.com> wrote:
>>
>>> Bitching that distutils needs to be scratched and rewritten is not
>>> going to help. You need to DO it.
>> Nobody asked you (or anyone else) to do anything. Guido asked a
>> question, we answered with our rationale.
>>
>> There is no need to be rude.
> 
> It was not my intention to be rude, and I don't believe I was. This
> discussion has been ongoing for a long time, and I have just recently
> asked questions (which of course has been ignored) about this. This is
> a discussion that has been going on for a long time, more than a year,
> at least, yet noone who says distutils needs to be scratched and
> something else written does something about it.
> 
> I've pointed this out repeatedly, and been ignored. Perhaps I used to
> strong language this time, but on the other hand, I wasn't ignored
> this time.
> 
> The discussion is pointless. If you want something better than distutils, do it.

David's working on it:

http://pypi.python.org/pypi/numscons

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From regebro at gmail.com  Thu Nov 12 07:04:11 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 12 Nov 2009 07:04:11 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdg8c0$iuc$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
	<hdg8c0$iuc$1@ger.gmane.org>
Message-ID: <319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>

2009/11/12 Robert Kern <robert.kern at gmail.com>:
>> The discussion is pointless. If you want something better than distutils,
>> do it.
>
> David's working on it:
>
> http://pypi.python.org/pypi/numscons

"Enable to use scons within distutils to build extensions"

I'm confused now.


-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From robert.kern at gmail.com  Thu Nov 12 07:07:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Nov 2009 00:07:02 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>	<hdg8c0$iuc$1@ger.gmane.org>
	<319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
Message-ID: <hdg8m6$jpg$1@ger.gmane.org>

Lennart Regebro wrote:
> 2009/11/12 Robert Kern <robert.kern at gmail.com>:
>>> The discussion is pointless. If you want something better than distutils,
>>> do it.
>> David's working on it:
>>
>> http://pypi.python.org/pypi/numscons
> 
> "Enable to use scons within distutils to build extensions"
> 
> I'm confused now.

It's a start.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robert.kern at gmail.com  Thu Nov 12 07:09:18 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Nov 2009 00:09:18 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>	<hdg8c0$iuc$1@ger.gmane.org>
	<319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
Message-ID: <hdg8qe$jpg$2@ger.gmane.org>

Lennart Regebro wrote:
> 2009/11/12 Robert Kern <robert.kern at gmail.com>:
>>> The discussion is pointless. If you want something better than distutils,
>>> do it.
>> David's working on it:
>>
>> http://pypi.python.org/pypi/numscons
> 
> "Enable to use scons within distutils to build extensions"
> 
> I'm confused now.

In fact, David wouldn't have bothered with the distutils bits at all if I hadn't 
been so insistent that the standard "python setup.py install", etc. would 
continue to work.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From glyph at twistedmatrix.com  Thu Nov 12 07:10:29 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Thu, 12 Nov 2009 01:10:29 -0500
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
Message-ID: <B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>


On Nov 12, 2009, at 12:02 AM, David Cournapeau wrote:

> Glyph Lefkowitz wrote:
>> There are probably a dozen other ways that you *could* work on distutils and benefit more immediately from your efforts than the next Python release.  To think otherwise is a simply a failure of imagination.  Now, if you think it's *too hard* to do that, it might be interesting to hear why you think that, and what exactly the effort would be; a nebulous assertion that it's just too hard and we should throw our hands up (while I can definitely understand the impulse to make such an assertion) serves only to discourage everyone.
> 
> I am trying to understand what is 'nebulous' about our claims. We have
> given plenty of hard and concrete examples of things which are
> problematic in distutils.

I'm sorry if I gave the impression that I was contesting that particular assertion.  We all agree that distutils has deep problems.

And, I don't think that everything that has been said is overgeneral or unhelpful.  Before I dive into more criticism, let me just say that I agree 100% with Robert Kern's message where he says:

> In order to integrate this with setuptools' develop command (...) we need to create a subclass of setuptool's develop command that will reinitialize build_src with the appropriate option. Then we need to conditionally place the develop command into the set of command classes so as not to introduce a setuptools dependency on those people who don't want to use it.

> This is nuts.

This is completely correct.  I've done stuff like this, we've all probably done stuff like this.  Conditional monkeypatching and dynamic subclassing is all over the place in distutils extension code, and it is *completely* nuts.

Still, it would have been more helpful to point out how exactly this problem could be solved, and to present (for example) a description of similar objects politely interacting and delegating responsibility to one another to accomplish the same task.

I would definitely characterize these assertion from Robert as "nebulous", given that the prior messages in the thread (as far as I can tell) do not describe the kind of massive-overhaul changes which would fix things, only the problems that currently exist:

> In our considered opinion, piecemeal changes probably aren't going to solve the significant problems that we face.

Why not?  The whole of computer history is the story of piecemeal improvements of one kind or another; despite perennial claims that, for example, hierarchical filesystems or bit-mapped displays "fundamentally" cannot support one type of data or another, here we are.

Or this one, also from Robert:

> Mostly because I'm entirely uninterested in helping you make incremental improvements that are going to break all the hard work we've already done just to get things working as it is.


Why do incremental improvements have to break all the hard work that has already been done?  Surely this is what a compatibility policy is about.

Or this one, from you, which is more specific to a topic, but still doesn't really say anything useful that I can discern:

> I think the compiler class and the likes should simply be removed ... There should not be any objects / classes for compilers, it is not flexible enough ... You cannot obtain this with classes and objects (especially when you start talking about performance: ...).


It's clear to me from the responses in this thread that I'm not the only one who is only getting vague hints of what you're actually talking about from language like this.

"classes and objects" have been used in many high-performance systems.  Personally I find "classes and objects" fairly flexible as well.  In fact, if *I* were to make a nebulous claim about distutils' design structure, it would be that the parsimony with creating whole new classes and instantiating multiple objects is the problem; there should be more classes, more objects, less inheritance and fewer methods.  So why can't 

> The major progress in our build issues have
> been achieved by dropping distutils. Up to now, the only people who have
> claimed that distutils can solve our problems are the people who are not
> involved at all with our projects, and the people who claim distutils
> cannot solve our problems are the people involved with it. That's odd to
> say the least.

I'm not asserting that distutils can fix your problems; I don't know enough about your problems to say for sure.  Certainly it seems clear that present-day distutils cannot.  I just know that there are many people on this list who are committed to a particular approach to evolving distutils, and while there is a lot of value in clearly explaining problems with that approach so they can be addressed, it's unhelpful to keep asserting (as has been done many times in this thread) that incremental evolution cannot address these problems.  It's a religious belief either way: my experience suggests that rewrites are never, ever a good idea, but others' experience may differ.

However, I feel compelled to repeat that it is a matter of historical fact and, I suspect, a corollary of the Church-Turing thesis that pretty much any software system *can* be changed into just about any other software system through a series of evolutionary steps where the system does something useful at each step; it is a question of whether you believe this approach requires an unreasonable amount of effort, and how big the steps need to be.  If you believe the effort required would be unreasonable, then let's see if we can find a radical, incompatible change to distutils that we all agree would be an improvement, and see if we also agree that the effort would be impractical.  Right now I can't find any such change looking through the recent history of this discussion, just a smattering of half-expressed ideas about how certain things in distutils are crummy and should look totally different than they are.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091112/5700c518/attachment-0001.htm>

From regebro at gmail.com  Thu Nov 12 07:25:33 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 12 Nov 2009 07:25:33 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdg8m6$jpg$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
	<hdg8c0$iuc$1@ger.gmane.org>
	<319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
	<hdg8m6$jpg$1@ger.gmane.org>
Message-ID: <319e029f0911112225rea17b04id5898a35c5525777@mail.gmail.com>

2009/11/12 Robert Kern <robert.kern at gmail.com>:
> It's a start.

OK, good enough.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From robert.kern at gmail.com  Thu Nov 12 07:36:15 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Nov 2009 00:36:15 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
Message-ID: <hdgad0$neg$1@ger.gmane.org>

Glyph Lefkowitz wrote:

> Still, it would have been more helpful to point out how exactly this 
> problem could be solved, and to present (for example) a description of 
> similar objects politely interacting and delegating responsibility to 
> one another to accomplish the same task.

Sorry, I edited out the bit at the last minute where I explained that it would 
be great to have a centralized option-managing object such that any command can 
ask what options were set on any other regardless of the dependencies between 
commands.

> I would definitely characterize these assertion from Robert as 
> "nebulous", given that the prior messages in the thread (as far as I can 
> tell) do not describe the kind of massive-overhaul changes which would 
> fix things, only the problems that currently exist:
> 
>> In our considered opinion, piecemeal changes probably aren't going to 
>> solve the significant problems that we face.
> 
> Why not?  The whole of computer history is the story of piecemeal 
> improvements of one kind or another; despite perennial claims that, for 
> example, hierarchical filesystems or bit-mapped displays "fundamentally" 
> cannot support one type of data or another, here we are.

Perhaps in my head the analogy with biological evolution is unjustifiably 
strong. Species can't always get from point A to point B while making viable 
intermediates with incremental changes. Evolutionary deadends happen frequently. 
In software, design decisions early on affect how much change the software can 
tolerate (which is why we are told to "design for change"). I think that 
distutils exemplifies a design that is particularly adverse to evolution.

> Or this one, also from Robert:
> 
>> Mostly because I'm entirely uninterested in helping you make 
>> incremental improvements that are going to break all the hard work 
>> we've already done just to get things working as it is.
> 
> Why do incremental improvements have to break all the hard work that has 
> already been done?  Surely this is what a compatibility policy is about.

Since Tarek keeps asking us to make proposals without thinking about 
compatibility, I wonder what policy is being kept in mind. My comment stems from 
my worry about that attitude.

In any case, I think that keeping compatibility while making improvements to the 
code in situ is going to be quite difficult. The distutils API is not clean. 
Using distutils beyond setup() and Extension() involves too much intimate 
knowledge of the detailed implementation of distutils as-is. That's why we got a 
2.6.4 release.

This is why I think the piecemeal evolution of distutils is not going to work. 
The act of fixing distutils to make a cleaner API for modification and extension 
  *triggers* the problem that the fix is supposed to address.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Thu Nov 12 07:34:58 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Nov 2009 15:34:58 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
 want CPAN)
In-Reply-To: <B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
Message-ID: <4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>

Glyph Lefkowitz wrote:
>
> On Nov 12, 2009, at 12:02 AM, David Cournapeau wrote:
>
>> Glyph Lefkowitz wrote:
>>> There are probably a dozen other ways that you *could* work on
>>> distutils and benefit more immediately from your efforts than the
>>> next Python release.  To think otherwise is a simply a failure of
>>> imagination.  Now, if you think it's *too hard* to do that, it might
>>> be interesting to hear why you think that, and what exactly the
>>> effort would be; a nebulous assertion that it's just too hard and we
>>> should throw our hands up (while I can definitely understand the
>>> impulse to make such an assertion) serves only to discourage everyone.
>>
>> I am trying to understand what is 'nebulous' about our claims. We have
>> given plenty of hard and concrete examples of things which are
>> problematic in distutils.
>
> I'm sorry if I gave the impression that I was contesting that
> particular assertion.  We all agree that distutils has deep problems.
>
> And, I don't think that everything that has been said is overgeneral
> or unhelpful.  Before I dive into more criticism, let me just say that
> I agree 100% with Robert Kern's message where he says:
>
>> In order to integrate this with setuptools' develop command (...) we
>> need to create a subclass of setuptool's develop command that will
>> reinitialize build_src with the appropriate option. Then we need to
>> conditionally place the develop command into the set of command
>> classes so as not to introduce a setuptools dependency on those
>> people who don't want to use it.
>
>> This is nuts.
>
> This is completely correct.  I've done stuff like this, we've all
> probably done stuff like this.  Conditional monkeypatching and dynamic
> subclassing is all over the place in distutils extension code, and it
> is *completely* nuts.
>
> Still, it would have been more helpful to point out how exactly this
> problem could be solved, and to present (for example) a description of
> similar objects politely interacting and delegating responsibility to
> one another to accomplish the same task.
>
> I would definitely characterize these assertion from Robert as
> "nebulous", given that the prior messages in the thread (as far as I
> can tell) do not describe the kind of massive-overhaul changes which
> would fix things, only the problems that currently exist:
>
>> In our considered opinion, piecemeal changes probably aren't going to
>> solve the significant problems that we face.
>
> Why not?  The whole of computer history is the story of piecemeal
> improvements of one kind or another; despite perennial claims that,
> for example, hierarchical filesystems or bit-mapped displays
> "fundamentally" cannot support one type of data or another, here we are.

I think Robert meant piecemeal changes from an implementation POV, not
that we should ignore any history or existing design solutions.
Actually, I think that distutils is victim a lot of NIH, it is totally
different from any other build system I have seen.

>
> Or this one, also from Robert:
>
>> Mostly because I'm entirely uninterested in helping you make
>> incremental improvements that are going to break all the hard work
>> we've already done just to get things working as it is.
>
> Why do incremental improvements have to break all the hard work that
> has already been done?  Surely this is what a compatibility policy is
> about.

Here is what *I* mean by distutils compatibility, so that we are sure to
talk about the same thing:
    - existing setup.py should run without problem, and produce the same
software when installed/produced (bdist_wininst, sdist, etc...) under
the same conditions as distutils (actually, setuptools, since distribute
is a fork of setuptools).
    - existing usage of distutils API should remain compatible. I asked
in a previous email what is meant by distutils API, Tarek answered
anything which does not start with an underscore. But what does that
mean ? For example, in numscons, I rely on the build directory to have a
certain structure (implemented defined in distutils: you can't retrieve
them from distutils from public API). Is this part of the API ? Is using
copied commands to get some characteristics considered part of the API ?
Is the order of commands, or their attributes considered public (they
all start without an underscore, but they are not documented anywhere) ?

During a more precise discussion, I think we have more or less agreed
with Tarek that build_ext needs a significant overhaul. Although we did
not discuss concretely about other commands, I think the same kind of
arguments apply to almost any command. There is then the issue of
communicating between commands, through the Distribution class.

Let's assume for the argument's sake that we manage to convince the
community as a whole that both commands and distribution classes need to
be redesigned. At that point, what's different from a newly distribution
tool, with a totally different API ? (but which could reuse distutils
implementations parts, of course). Certainly, unless you keep the
current code and the new one, you will break almost every distutils API
user out there.

Python distutils (the one included in python) has broken our extensions
countless times already, even though no significant feature has been
added. Setuptools itself already breaks a lot of them out there. That's
why I am not convinced that you can improve distutils without causing
the exact same issues as a new distribution system. That's from my
experience from extensively writing extensions around it.

> "classes and objects" have been used in many high-performance systems.
>  Personally I find "classes and objects" fairly flexible as well.  In
> fact, if *I* were to make a nebulous claim about distutils' design
> structure, it would be that the parsimony with creating whole new
> classes and instantiating multiple objects is the problem; there
> should be more classes, more objects, less inheritance and fewer methods.

I agree this claim was vague. I was only talking about using class and
object for building, not objects in a general sense. The problem with
compilation is that you need almost total flexibility: you simply cannot
foresee how to use tools and their interaction with the launched
commands. Neither make, nor waf, nor scons does that. Instead, they
provide the fundamental abstraction source -> "action" -> target, where
action can really be anything, and should be decoupled from any tool
definition (what's common between a C compiler and a code generator, for
example ?).

Concerning classes, I don't think you can have a hierarchy for
compilers: they behave so differently depending on the platforms that
they share very little. This is true for any tool, actually. What is
common between (most) C compilers is that they produce object files from
sources, and link object files together.

About the performance claim: scons has speed issues (the issue keeps
coming up on the user and dev ML), in part because it uses too many
objects. Waf manages to be much faster (it is basically as fast as make
for reasonably sized projects, and it does automatic dependency). They
manage to do so thanks to agressive optimizations. They care about the
number of attributes of the fundamental classes, they compile string
commands into functions to avoid useless substitutions (another scons
speed issue).

>
> However, I feel compelled to repeat that it is a matter of historical
> fact and, I suspect, a corollary of the Church-Turing thesis that
> pretty much any software system *can* be changed into just about any
> other software system through a series of evolutionary steps where the
> system does something useful at each step; it is a question of whether
> you believe this approach requires an unreasonable amount of effort,
> and how big the steps need to be.

Yes, obviously you *can* go from distutils to a perfect system in
piecemeal changes. The question is how long and how much effort does it
take. So when I say "I don't think you can significantly improve
distutils", it is to be understood as "it will take less time to go to a
better system without bothering with keeping the same architecture".

>  If you believe the effort required would be unreasonable, then let's
> see if we can find a radical, incompatible change to distutils that we
> all agree would be an improvement, and see if we also agree that the
> effort would be impractical.

A few examples which are a problem for us (us being numpy/scipy/etc...
developers/users here):
    - automatic dependency handling. If I change one header, only the
files which include this header will be rebuilt; if a fortran compiler
flag is changed, only fortran source files are recompiled, etc...
    - package description so that simple packages could be built
automatically without running untrusted code
    - reliable parallel builds
    - integration with 3rd party tools

To be honest, the only one I really care about is the last one: if we
could find a solution so that I can build the C code from say make or
scons, and build/install in a distutils compatible way through simple
and stable API, it would solve all the other parts through 3rd party
code. Right now, numscons depends too much on distutils implementation
details, and cannot produce all the non build goodies from distutils
(sdist, bdist_wininst, etc...).

cheers,

David


From glyph at twistedmatrix.com  Thu Nov 12 08:04:40 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Thu, 12 Nov 2009 02:04:40 -0500
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdgad0$neg$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<hdgad0$neg$1@ger.gmane.org>
Message-ID: <E83E5CC3-2BA7-47DE-A53C-9DCE5F9B27F8@twistedmatrix.com>


On Nov 12, 2009, at 1:36 AM, Robert Kern wrote:

> Glyph Lefkowitz wrote:
> 
>> Still, it would have been more helpful to point out how exactly this problem could be solved, (...)

> Sorry, I edited out the bit at the last minute where I explained that it would be great to have a centralized option-managing object such that any command can ask what options were set on any other regardless of the dependencies between commands.

I'm familiar with the hazards of over-editing :).  I am often asked to shorten my messages, and when I try too hard to do so, I leave out important elements.

One of the things I'm asking for, actually, is fewer, longer messages, with more substantive points in them.  I know some people don't like that, but discussions about big, complex topics like this that try to address them one little conversational point at a time tend to get circular quickly.

> Perhaps in my head the analogy with biological evolution is unjustifiably strong. Species can't always get from point A to point B while making viable intermediates with incremental changes. Evolutionary deadends happen frequently.

This is explicitly _not_ biological evolution.  For example, when you wonder about this:

> Since Tarek keeps asking us to make proposals without thinking about compatibility, I wonder what policy is being kept in mind. My comment stems from my worry about that attitude.


The way I'm interpreting Tarek's comments - and he can correct me if I'm wrong - is that the strategy is to short-circuit evolution.  We should decide where we want to go - which may be an apparently discontinuous place - then path-find there.  The path-finding is rarely as hard as it seems like it's going to be.  The brute-force approach, which also happens to be an approximation of the Twisted framework compatibility baseline, is:

  release -1: import old; old.crummy_api()
  release 0: old.crummy_api() emits a PendingDeprecationWarning, new.good_api() introduced
  release 1: old.crummy_api emits DeprecationWarning pointing at new.good_api(), new.good_api() improved based on feedback from r0
  release 2: old.crummy_api raises DeprecationError pointing at new.good_api()
  release 3: old.crummy_api removed

This is almost a straw-man: you can do this for any old.crummy_api and new.good_api, regardless of whether the new thing actually satisfies the old thing's requirements at all.  It is often possible to do much better.  But the point is, if you have a clear new.good_api to get to, it's possible to do all *kinds* of crazy stuff in Python to emulate and re-direct the behavior of crummy_api, discover how it's being used, provide useful hints to the developer, etc.  It's a matter of how much effort you want to put into it.  For many distutils use-cases, it sounds to me like the path forward is to avoid using any API at all, and just ask most projects to provide static metadata.  Build-heavy projects like numpy will require new integration points to do custom heavy lifting, but it will be easier to define those integration points if they're not something that every project under the sun will potentially need to interact with.

But, during the creative design process of good_api, it's often helpful to pretend crummy_api doesn't even exist, so you can design something good that solves its problems well, and address the translation as a truly separate issue.

> In software, design decisions early on affect how much change the software can tolerate (which is why we are told to "design for change").

Who's "we", kimosabe?  *We* are told "you aren't gonna need it"; maybe some other people are told to design for change :).  In fact, I think that distutils is over-designed for change.  It has altogether too many different extension mechanisms, which often interfere with each other: subclassing, configuration files, including random bits of code in setup.py.  And then of course there's the monkey-patching for the cases that weren't covered :).


From robert.kern at gmail.com  Thu Nov 12 08:15:51 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 12 Nov 2009 01:15:51 -0600
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <E83E5CC3-2BA7-47DE-A53C-9DCE5F9B27F8@twistedmatrix.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	<hdfml9$fbp$1@ger.gmane.org>	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	<hdfu07$un4$1@ger.gmane.org>	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>	<hdgad0$neg$1@ger.gmane.org>
	<E83E5CC3-2BA7-47DE-A53C-9DCE5F9B27F8@twistedmatrix.com>
Message-ID: <hdgcn8$s8h$1@ger.gmane.org>

Glyph Lefkowitz wrote:

>> In software, design decisions early on affect how much change the software can tolerate (which is why we are told to "design for change").
> 
> Who's "we", kimosabe? 

It pops up in the Design Patterns literature. I didn't make this up. :-)

http://www.google.com/search?q=%22design+for+change%22+%22design+patterns%22

 > In fact, I think that distutils is over-designed for change.  It has
 > altogether too many different extension mechanisms, which often interfere with
 > each other: subclassing, configuration files, including random bits of code in
 > setup.py.  And then of course there's the monkey-patching for the cases that
 > weren't covered :).

That's not the kind of change I'm talking about. I'm talking about the evolution 
of distutils itself, not the configuration and extension of distutils when it is 
used. Those particular mechanisms are the antithesis of designing for change 
because their use hampers the change of distutils itself.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From cournape at gmail.com  Thu Nov 12 08:41:35 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 16:41:35 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
	<hdg8c0$iuc$1@ger.gmane.org>
	<319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
Message-ID: <5b8d13220911112341y180088adv856abd572282b0ba@mail.gmail.com>

On Thu, Nov 12, 2009 at 3:04 PM, Lennart Regebro <regebro at gmail.com> wrote:
> 2009/11/12 Robert Kern <robert.kern at gmail.com>:
>>> The discussion is pointless. If you want something better than distutils,
>>> do it.
>>
>> David's working on it:
>>
>> http://pypi.python.org/pypi/numscons
>
> "Enable to use scons within distutils to build extensions"
>
> I'm confused now.

The idea is that numpy.distutils adds a scons command, and if your
setup.py contains something like:

add_sconscript('SConscript')

It will call scons with enough options from distutils so that you can
build extensions from scons instead of distutils (it replaces all the
build_* commands except build_py, basically).

Numscons itself adds things like builders to handle python extensions,
so that you can do things like:

env.DistutilsPythonExtension('foo', source=['foo.c'])
env.DistutilsCtypesExtension('c_foo', source=['c_foo.c'])

Those CamelCase functions are called builders in scons parlance, and
any builder starting with Distutils* is such as it will put things
where distutils expects them (which is suprisingly difficult).

If you call python setup.py scons, you then get your extensions in
e.g. build/lib.win32-2.6/, and then all the bdist_wininst, etc... work
as expected. It solves our build issues beautifully:
 -  adding support for new compiler, new builder, new configuration
check, etc... is more pleasant than with distutils
 - the configuration framework is much more robust: you can check
things like type size, available header, available libraries and
functions in a cross platform way.
 - parallel builds works well
 - any change in one source file automatically trigger a new build of
the concerned files only (it goes as far as only relinking if you
change your library path).
 - changing compiler flags with CFLAGS works as expected.

But it has significant drawbacks, which means it is only useful as a
build/developer tool. This is mainly because scons and distutils
cannot interact 'bidirectionally' (both scons and distutils fault). In
particular:
 - source files added in scons are not automatically added to the
FileList object used for sdist production
 - changes in python files are not handled
 - we cannot integrate things like building a full .dmg or .msi from scons
 - we cannot build compatible eggs from scons because there is no
public API to do so

Most of the hard work in numscons is a 1000 LOC subpackage if I don't
count all the fortran stuff. Of course, it reuses scons, so that's
cheating. But note that scons has no knowledge whatsoever of python
extensions, and numscons can build extensions on windows (both 32 and
64 bits), linux, mac os x, solaris, freebsd, including debug
extensions. That's one of the reason why I find the claim of distutils
having a deep and large knowledge hard to believe.

David

From regebro at gmail.com  Thu Nov 12 09:01:15 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Thu, 12 Nov 2009 09:01:15 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <5b8d13220911112341y180088adv856abd572282b0ba@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
	<hdg8c0$iuc$1@ger.gmane.org>
	<319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
	<5b8d13220911112341y180088adv856abd572282b0ba@mail.gmail.com>
Message-ID: <319e029f0911120001s35df7f6evf5936c41e392f66f@mail.gmail.com>

2009/11/12 David Cournapeau <cournape at gmail.com>:
> On Thu, Nov 12, 2009 at 3:04 PM, Lennart Regebro <regebro at gmail.com> wrote:
>> I'm confused now.
>
> The idea is that numpy.distutils adds a scons command, and if your
> setup.py contains something like:

What I'm confused about is that you say that distutils should be
scrapped and not incrementally improved, yet your effort of doing it
"the right way" builds on distutils.

> That's one of the reason why I find the claim of distutils
> having a deep and large knowledge hard to believe.

Me too, but I think the community does. And currently that knowledge
is being focused into a bunch of PEP's. Which is the reason for my
previous (and so far ignored) question if that work should be ignored,
or if a new solution done The Right Way should build on those PEPs.
And  if it should, what the difference then is to the current effort
of fixing distutils.

If scons is the Right Way then it seems to me yet again that the whole
discussion is moot, and that what should be done is to build
extensions for scons to build/install/upload Python modules, and then
after doing that, just saying "Use scons instead". If the community
then agrees, scons will be used. Obviously you should then here on
this list say that you are building these extensions and asking for
help with it.

Less talk, more hockey, as we say in Sweden.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From cournape at gmail.com  Thu Nov 12 10:16:51 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 18:16:51 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <319e029f0911120001s35df7f6evf5936c41e392f66f@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<319e029f0911112128t3e4adc75xee0edf0044b1b879@mail.gmail.com>
	<5b8d13220911112146w4ea3ff51x5ea940817dd4c089@mail.gmail.com>
	<319e029f0911112159r5e3a7f27w19993cae52aeb211@mail.gmail.com>
	<hdg8c0$iuc$1@ger.gmane.org>
	<319e029f0911112204t3aa22f0bl43a657ace25a7de@mail.gmail.com>
	<5b8d13220911112341y180088adv856abd572282b0ba@mail.gmail.com>
	<319e029f0911120001s35df7f6evf5936c41e392f66f@mail.gmail.com>
Message-ID: <5b8d13220911120116x63e736b3sccb1ed3d5bc2eae6@mail.gmail.com>

On Thu, Nov 12, 2009 at 5:01 PM, Lennart Regebro <regebro at gmail.com> wrote:

>
> What I'm confused about is that you say that distutils should be
> scrapped and not incrementally improved, yet your effort of doing it
> "the right way" builds on distutils.

Maybe we do not mean the same by rewriting a distribution tool then.
Numscons works by totally bypassing distutils and using its own
mechanism for anything build-related. It used to use compiler options
from distutils, but I have added a mode to bypass this entirely so
that you can use new compilers by writing scons tools without having
to care about distutils (e.g. to build numpy/scipy with Intel
compilers on mac os x or windows).

Now, I am trying to see how far I can go from a purely static file for
package description, plus tools to convert existing setup.py to it.
The hope is that I can soon get a tool which does build and packaging
without using distutils at all.

>
>> That's one of the reason why I find the claim of distutils
>> having a deep and large knowledge hard to believe.
>
> Me too, but I think the community does. And currently that knowledge
> is being focused into a bunch of PEP's. Which is the reason for my
> previous (and so far ignored) question if that work should be ignored,
> or if a new solution done The Right Way should build on those PEPs.

Here are the PEP I see being discussed related to anything
distribute/setuptools-related. Feel free to indicate me the ones I am
missing.

PEP 382: namespace packages

Mostly independent from the distribution tool, I have not followed the
discussion on those ones. The current setuptools implementation causes
deployment issues because of file sharing which is why I mention it
here.

PEP 345, 390, etc...: Metadata

I think a new format ala Cabal is needed for full package description.
The current format is ok for distribute evolution, but retrofitting a
full description feels very hackish. The one I am working on in
toydist is such as many packages only require this file, no setup at
all (the format is purely straw-man, to see what kind of problems need
to be solved). Right now, I have enough code so that I can almost
convert several simple (but not trivial, e.g. I consider sphinx to be
a simple but not trivial package)  packages from pypi. This is done
through a distutils command, i.e. if you do:

python setup.py gen_toyinfo

with a setup like

"""
setup(name='foo',
 version='1.0',
 packages=['foo'],
 extensions=[Extension(_foo, sources=['bar.c', 'foo.c'])]
)
"""

you get a toysetup.info description file:

"""
Name: foo
Version: 1.0

PackageAdditionalData:
  data/*.dat
  test/*.py

Library:
  Extension: _foo
    src: bar.c, foo.c
  Packages: foo
"""

I have also code to build from this file alone using distutils and
scons at will: this file can be fed back to a trivial setup.py, so
here I get backward (and even forward if needed) compatibility. One
major problem is package data: the current way to use MANIFEST and co
is too complicated (and fragile when MANIFEST are not updated
correctly). I don't want to retrofit it because I strongly believe
distribution files should be explicitly added, but I guess it would
make sense to support it at conversion time only.

PEP 376: egg-info

I don't personally care about the egg and egg-info format details,
only that it is specified (and versioned), like ruby gems, for
interoperability. So that I can build eggs from scons, for example.

Enthought has some nice open source tools which give you reliable
install/upgrade/removal/listing of software using eggs as a format
(ala dpkg and apt-get, using eggs as the basic format). One of the
major problem with eggs is that on windows at least, some people
prefer using .exe/.msi because of its integration with add/remove. If
the egg format is rich enough, it should be possible to get tools to
convert from one format to the other.

> And ?if it should, what the difference then is to the current effort
> of fixing distutils.
>
> If scons is the Right Way then it seems to me yet again that the whole
> discussion is moot, and that what should be done is to build
> extensions for scons to build/install/upload Python modules, and then
> after doing that, just saying "Use scons instead".

Clearly, requiring distutils, distribute or whatever to reimplement
scons or waf is insane. But today, numscons only works because it uses
many implementation details of distutils, and cannot be used as it is
because all the distribution/installing/packaging part from distutils
is not reusable. If distribute could be improved so that it could make
this easy, it would be great. But I believe doing so without being
disruptive is hard - and the only way to understand why is to try
doing it.

David

From ziade.tarek at gmail.com  Thu Nov 12 10:20:16 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 10:20:16 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdgad0$neg$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<hdgad0$neg$1@ger.gmane.org>
Message-ID: <94bdd2610911120120g1b070e9fl90427162814a3e8d@mail.gmail.com>

On Thu, Nov 12, 2009 at 7:36 AM, Robert Kern <robert.kern at gmail.com> wrote:
[..]
>>
>> Why do incremental improvements have to break all the hard work that has
>> already been done? ?Surely this is what a compatibility policy is about.
>
> Since Tarek keeps asking us to make proposals without thinking about
> compatibility, I wonder what policy is being kept in mind. My comment stems
> from my worry about that attitude.

Come on, I answered on that already. This is getting nowhere now.

I keep asking you that because, after a few mails we had, I think
that's the only way we can
work on your use cases efficiently. My policy is to discuss a design
for a particular
use case you have, before we put it back in the big picture. But that
is impossible if in every answer
you guys make, you just say that "distutils must be replaced".

Since three days now, I have made concrete proposals on some changes,
but you guys, seem not to hear it. It seems that the discussion turns into some
political discussion where you just say that "distutils must be replaced".

or "you can't fix it", "your can't make small steps"

Shall we focus on the problems you have ?

Tarek

From wentland at cl.uni-heidelberg.de  Thu Nov 12 10:28:51 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Thu, 12 Nov 2009 10:28:51 +0100
Subject: [Distutils] Fwd:  People want CPAN :-)
In-Reply-To: <a71153f0911110822q478d5f96s8e3b072afca60d34@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a71153f0911090314m669da82crd5c6abda6798c7d9@mail.gmail.com>
	<a71153f0911110822q478d5f96s8e3b072afca60d34@mail.gmail.com>
Message-ID: <20091112092851.GG7023@kinakuta.local>

On Wed, Nov 11, 2009 at 16:22 +0000, Antonio Cavallo wrote:
> 1. I don't need a replacement for rpm/yast/yum/synaptic: they provide
> file tracking and package resolution/dependency already and
> ? ?pepole are using them (and it has been for a very long time)
+1

I think that the ongoing discussion about Python packaging should focus
on the needs of the users. I made a proposal that might ease basic
packaging tasks for *nix package maintainers and think it would be a
great time to contact the following groups of users and encourage them
to provide some feedback on their needs:

* Python developers
  
  - How can distributions be queried? 
  - How is access to shipped data handled? 
  - What are common patterns/workarounds they use in their setup.py that
    should be automated/made easier?

* Distribution package maintainers

  - Should a distutils packaged provide more Metadata about the files that
    are shipped? 
  - What would help in the packaging process?

* Python users

  - Should PEP 370 compliant installation become the default? (--user the
    default) 
  - What is missing from virtualenv? (like easy creation of a
    virtualenv and automatically installing distribution according to a
    pip reqirements file) 
  - Would users like PyPi to be stricter in what it
    accepts (i.e. automatic testing and rejection of distributions of bad
    quality)?

I think that some of these questions are already answered in the PEPs
that are under consideration right now. I would *strongly* suggest to
get them into shape before any discussion on CPAN and that is started
and binds a lot of resources.

I like the aproach taken by Vinay Sajip with PEP 391. He (and others?)
worked on them until they were deemed to be of sufficient quality.
Afterwards he asked python-dev for input, incorporated these changes and
then asked the whole Python community on python-list. I happily provided
input and got them incorporated into the PEP. THAT was a great user
experience and a big advantage of open (source) communities.

I would therefore suggest to:

1. Get the PEPs into shape
2. Contact the people and spread the word that *now* is the time to
   provide input that influences the way Python packaging will be done
   for some time to come.
3. Collect input on some Wiki/document/whatever and discuss the issues
   and solutions.
4. Change the PEPs to incorporate the solutions.
5. Goto 2 if necessary
6. Design the distutils/distribute/virtualenv/pip API to reflect this
7. Implement the changes.

I will ask on some distribution specific MLs (like debian-python) what
they might need, once we singled out some ideas that look promising.

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091112/ebd848e6/attachment.pgp>

From ziade.tarek at gmail.com  Thu Nov 12 10:40:57 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 10:40:57 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>

On Thu, Nov 12, 2009 at 7:34 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
[...]
>   - existing usage of distutils API should remain compatible. I asked
> in a previous email what is meant by distutils API, Tarek answered
> anything which does not start with an underscore. But what does that
> mean ? For example, in numscons, I rely on the build directory to have a
> certain structure (implemented defined in distutils: you can't retrieve
> them from distutils from public API). Is this part of the API ? Is using
> copied commands to get some characteristics considered part of the API ?
> Is the order of commands, or their attributes considered public (they
> all start without an underscore, but they are not documented anywhere) ?

The build and instalation scheme should be part of the API. If you
look in sysconfig,
you get most of the install schemes via the API. If anything is
missing, tell us.

Now for the build I am not sure what you are trying to perform here.
I'll ask you
for a precise example here.

But are you willing to do this finish the discussion of this precise example ?
I mean, if I give you an answer, *or* a proposal for a change in Distutils,
is your answer is going to be "distutils must be replaced anyway" ?
or are you willing to hear my proposal and work it out with me ?

Because, Distutils is not going to be replaced.


[..]
>
> During a more precise discussion, I think we have more or less agreed
> with Tarek that build_ext needs a significant overhaul. Although we did
> not discuss concretely about other commands, I think the same kind of
> arguments apply to almost any command. There is then the issue of
> communicating between commands, through the Distribution class.

And I made a proposal for this last one too in this thread.  But, like
the Extensions proposal I have made,
you seem not to hear them.

I don't get it. It seems to me that you just don't want to see
Distutils improved, since
everytime you mention a problem, and I start to make proposals, we
don't discuss it further and you jump
back on the 'distutils must be replaced' line.

But what is going to happen is that,  Distutils will not be replaced.
We want to make it evolve.

[..]
> Python distutils (the one included in python) has broken our extensions
> countless times already, even though no significant feature has been
> added.

Sorry, but that's pretty vague. Are you mentioning the changes in
build_ext, or something else ?


Tarek.

From ziade.tarek at gmail.com  Thu Nov 12 10:54:41 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 10:54:41 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>

On Thu, Nov 12, 2009 at 5:30 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Greg Ewing wrote:
>> As far as I can tell David seems to be saying that
>> instantiating a class for every file in the system
>> would be too much overhead.
>>
>> I'm not convinced about that -- make builds a dependency
>> graph with a node for every target before it goes
>> to work, and I think scons does something similar
>
> Yes, and scons has scalability problems because of this (both from CPU
> and memory POV).

But you were saying in the discussion that Disutils has a *design*
problem because it uses OOP...

Anyways, let's not drop the other part of this thread:

>>>>> get_install_paths('FOO')
>>
>> And that's the API we want to add in sysconfig, roughly.
>
> That does not solve the problem about getting FOO in the first place
> when calling get_install_path.

Why that ? where "FOO" comes from ? if it's an option you provide at
build time like you said,
earlier, you just pass it to the API to get the paths.

FOO is not coming from nowhere...

From david at ar.media.kyoto-u.ac.jp  Thu Nov 12 11:25:29 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 12 Nov 2009 19:25:29 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
 want CPAN)
In-Reply-To: <94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	
	<hdfjlk$6km$1@ger.gmane.org>	
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>	
	<hdfml9$fbp$1@ger.gmane.org>	
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>	
	<hdfu07$un4$1@ger.gmane.org>	
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>	
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>	
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>	
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
Message-ID: <4AFBE299.9090303@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
>
> The build and instalation scheme should be part of the API. If you
> look in sysconfig,
> you get most of the install schemes via the API. If anything is
> missing, tell us.

(I am reading the code there:
http://svn.python.org/projects/python/branches/tarek_sysconfig/Lib/distutils/sysconfig.py)

I have not tried running the code, but here is what I see missing:
    - anything related to build (where are libraries put in the build
directory, where are the extensions put - necessary if you are
interested in inter-operating with make)
    - I don't see the equivalent of
distutils.command.install.install_libbase and similar prefixes
    - How can we retrieve the full customized install path ? I have not
found how I could use the code in sysconfig to solve the use case I have
written about before (related to retrieving install prefix).

>
> Now for the build I am not sure what you are trying to perform here.

This is always the same use-case: assuming you push the build to a 3rd
party tool like make, how can you make it work within distutils: build
directories, compiler options, etc... If you are familiar with
autootols, the idea could be similar in the sense that you would have a
Makefile.in, and a distutils command build_make would generate a
makefile, filling up the variables in makefile.in. You could also
imagine to put those variables in a separate file, which could be
queried from an API (using a makefile.in-like process for scons is not
very practical for example).

So the list of variables needed:
    - everything in sysconfig (but available for any compiler, including
MS ones)
    - any distutils command option. What is needed here is a way to
query the options given to command globally, that is the possibility to
access at any point in your setup.py every option of every command,
without running them as first as we have to do currently (like running
install_cmd within build_clib).

To be even more down to earth, here is how a scons call looks like on
linux for numscons:

/usr/bin/python
"/home/david/local/lib/python2.6/site-packages/numscons/scons-local/scons.py"
-f numpy/core/SConstruct -I. scons_tool_path="" src_dir="numpy/core"
pkg_path="numpy/core" pkg_name="numpy.core" log_level=50
distutils_libdir="../../../../build/lib.linux-i686-2.6"
distutils_clibdir="../../../../build/temp.linux-i686-2.6"
distutils_install_prefix="/usr/local/lib/python2.6/dist-packages/numpy/core"
cc_opt=gcc cc_opt_path="/usr/bin" debug=0 f77_opt=gfortran
f77_opt_path="/usr/bin" cxx_opt=g++ cxx_opt_path="/usr/bin"
include_bootstrap=../../../../numpy/core/include bypass=0 import_env=0
silent=0 bootstrapping=1

To understand the weird relative paths, you need to be aware than scons
(and waf) consider all the paths relatively to a build directory. Scons
copy the sources files in the build directory (this is an
oversimplification, but should be enough for this discussion). For
example, here, we build the package numpy.core, so it looks like
(relatively to the top setup.py file):

cwd/setup.py
   /numpy/core/setup.py
   /build/lib.linux-i686-2.6/ # where distutils would put all the
compiled code if build_ext were used
   /build/scons/   # any code generated by scons is put here
               numpy/core # the build directory for the current package
numpy.core

We pass the following informations:
 * distutils_libdir: where distutils put/expects python extensions
(which is replaced by current directory + the src_dir in case of
in-place build)
 * distutils_install_prefix: install prefix  + subpackage path. Where
the extensions would be installed
 * distutils_clibdir: where disstutils put/expects pure C libraries (the
ones built through build_clib)
 * numscons also encode compilation options in .ini-like files
(regularly auto-generated from the info at sysconfig). The idea is that
if someone wants to modify the flags, there is no need to touch any
python code.

Right now, I have to reimplement the distutils code to find those, as
the logic is not available through API. Generally, the build directories
are only known through the corresponding build_* commands, but they
become available only after running the commands, which we want to avoid.

> And I made a proposal for this last one too in this thread.  But, like
> the Extensions proposal I have made,
> you seem not to hear them.

Because I genuinely don't understand why it is required not to replace
distutils but it is ok to change its behavior in a fundamental way. For
some reasons (which may well just be my own fault), I fail to see any
reason  to start from existing code, publicly and widely used, if it is
ok to break it and change its behavior, especially if it will be done
several times. I was not at Pycon, so there may have been some other
requirements I have missed.

> Sorry, but that's pretty vague. Are you mentioning the changes in
> build_ext, or something else ?

For 2.6.3, yes. For python 2.6, the mingw support was broken because of
VS manifests; our fortran tools were broken because the msvc compiler
was change (to msvc9compiler). There were also some other changes
related to install directories and the --local option in 2.6 which
caused issues. Several setuptools have also broken various parts of our
own extensions in the past. If you really need to know the details for
the last ones, I can look back in our svn log.

cheers,

David

From ziade.tarek at gmail.com  Thu Nov 12 12:14:50 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 12:14:50 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <4AFBE299.9090303@ar.media.kyoto-u.ac.jp>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
	<4AFBE299.9090303@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911120314n42d4efeap512f8d91eb6e993f@mail.gmail.com>

On Thu, Nov 12, 2009 at 11:25 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Tarek Ziad? wrote:
>>
>> The build and instalation scheme should be part of the API. If you
>> look in sysconfig,
>> you get most of the install schemes via the API. If anything is
>> missing, tell us.
>
> (I am reading the code there:
> http://svn.python.org/projects/python/branches/tarek_sysconfig/Lib/distutils/sysconfig.py)
>
> I have not tried running the code, but here is what I see missing:
> ? ?- anything related to build (where are libraries put in the build
> directory, where are the extensions put - necessary if you are
> interested in inter-operating with make)

the build part will not be in sysconfig, and wil stay in distutils,
because I don't think the rest
of the stdlib cares about it.

Now, there's no reason at all that we can''t add an API in
distutils.util for example, that let you get
the build paths, and that will be used by all build_ commands, so you
don't have to use a build_ command
to get them.

I propose that you think about its signature, and that I add it after
we agree on it.


> ? ?- I don't see the equivalent of
> distutils.command.install.install_libbase and similar prefixes

That's platlib or purelib, depending on wether you have an extension or not.

Those are the default schemes and all $VARIABLE in there can be
overriden by the end user,
that's why you have this expansion mechanism.


> ? ?- How can we retrieve the full customized install path ? I have not
> found how I could use the code in sysconfig to solve the use case I have
> written about before (related to retrieving install prefix).
>

by calling "get_paths(scheme, vars)", where scheme is the installation
scheme you want (unix, nt, etc)
and vars your variables if you want to override any $VARIABLE.


>>
>> Now for the build I am not sure what you are trying to perform here.
[...]
>
> So the list of variables needed:
> ? ?- everything in sysconfig (but available for any compiler, including
> MS ones)
> ? ?- any distutils command option. What is needed here is a way to
> query the options given to command globally, that is the possibility to
> access at any point in your setup.py every option of every command,
> without running them as first as we have to do currently (like running
> install_cmd within build_clib).

You mean, accessing other commands command line options ? or the result
of their finalizations ?

For me, the install_cmd <-> build_clib problem shows that the "install" command
do some work that is needed by other commands, and that this work has
to be taken
out of it.

A command performs something, and should not be used to set up an *environment*.

What are the other commands you run to get information on the environment ?

[..]
>
> Right now, I have to reimplement the distutils code to find those, as
> the logic is not available through API. Generally, the build directories
> are only known through the corresponding build_* commands, but they
> become available only after running the commands, which we want to avoid.

I'd like to change that then. That's one thing we can change without
any problem.
Refactoring the build commands so they use some APIs that are outside
them, make sense.

And will allow you to get the information without having to run in
some command calls madness.

>> And I made a proposal for this last one too in this thread. ?But, like
>> the Extensions proposal I have made,
>> you seem not to hear them.
>
> Because I genuinely don't understand why it is required not to replace
> distutils but it is ok to change its behavior in a fundamental way.

It seems that some of Distutils knowledge is located in some commands,
and that this knowledge
could be pushed out in simple APIs, that could be used in the build_ commands,
but also in your commands.

Do you agree that this change will help you, and is possible without
replacing Distutils ?

>
>> Sorry, but that's pretty vague. Are you mentioning the changes in
>> build_ext, or something else ?
>
> For 2.6.3, yes. For python 2.6, the mingw support was broken because of
> VS manifests; our fortran tools were broken because the msvc compiler
> was change (to msvc9compiler). There were also some other changes
> related to install directories and the --local option in 2.6 which
> caused issues. Several setuptools have also broken various parts of our
> own extensions in the past. If you really need to know the details for
> the last ones, I can look back in our svn log.

I fully agree that some commands like build_ext are leading to some of
this problems.

For that I have a partial solution until it's redesigned : buildbot.

I remember I broke Numpy once. So I've worked on a buildbot that tries
to prevents regressions.

This one: http://buildbot.ziade.org/waterfall , as a matter of fact,
is grabbing Numpy
and builds it with 2.5, 2.6, 2.7 etc to make sure the "sdist" command
still works
and to make sure that  the produced tarball is the same all the time
(on the 1.3.0 tag)

I am pretty sure it can be ehnanced to protect more things on your
side. Help is welcome.

Tarek

From ziade.tarek at gmail.com  Thu Nov 12 12:19:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 12:19:25 +0100
Subject: [Distutils] Fwd: People want CPAN :-)
In-Reply-To: <20091112092851.GG7023@kinakuta.local>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<a71153f0911090314m669da82crd5c6abda6798c7d9@mail.gmail.com>
	<a71153f0911110822q478d5f96s8e3b072afca60d34@mail.gmail.com>
	<20091112092851.GG7023@kinakuta.local>
Message-ID: <94bdd2610911120319p30815006q110b1489726adb80@mail.gmail.com>

On Thu, Nov 12, 2009 at 10:28 AM, Wolodja Wentland
<wentland at cl.uni-heidelberg.de> wrote:
[..]
>
> I would therefore suggest to:
>
> 1. Get the PEPs into shape
> 2. Contact the people and spread the word that *now* is the time to
> ? provide input that influences the way Python packaging will be done
> ? for some time to come.
> 3. Collect input on some Wiki/document/whatever and discuss the issues
> ? and solutions.
> 4. Change the PEPs to incorporate the solutions.
> 5. Goto 2 if necessary
> 6. Design the distutils/distribute/virtualenv/pip API to reflect this
> 7. Implement the changes.
>
> I will ask on some distribution specific MLs (like debian-python) what
> they might need, once we singled out some ideas that look promising.

Sounds like a good plan.


>
> --
> ?.''`. ? ? Wolodja Wentland ? ?<wentland at cl.uni-heidelberg.de>
> ?: :' ?:
> ?`. `'` ? ? 4096R/CAF14EFC
> ? `- ? ? ? 081C B7CD FF04 2BA9 94EA ?36B2 8B7F 7D30 CAF1 4EFC
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQIcBAEBCAAGBQJK+9VSAAoJEIt/fTDK8U78isIQAJyPc3xtCf7DS2RAmFWoPTZs
> lj4GKftqZPTahXHrLEw75BjbM5iOr8cBnWiaGbn5fB6Id/n+4fpW3oXxPuSAJ5US
> n8mDmpiLpknzjIlHGOJ+Z/ItzrYqCbxVwld8n/1CGxko31qMY+1Q2YEQQ+5a8ONy
> II3zAWYgd7OH/+wG3L3J8UcDH82eOS2HOv3hnbPKjK7L2/+X/XfVL5la8J09J+rx
> tTqMHvpzTMgeoiTbmSL+94AQAnaVMSMN2gNaPu9tBKX0Mx9H/FNIA2HMXAAyzVAg
> IGKYRGQPPN05t6V2dfFptF0lGXIHM3r5jcBZ+d5kPl88zHs3h3kjU8uFeJTV/DYP
> qAA6s/+XdoPynbyV0AvuanILwkhfW7XgU0os+Hab1cSE0w9cB35VWqjctq5OLzC2
> bhxb3Ci5RaYIbRtr3438AE539g/cTz/TOfj+Wd1u0dTz9polliGV9u+Oz1AbTM1S
> Tqb1TQTThlyRjSfjXGWCqcg76g5p3gKSa5DylAhCKyKghK+yhjvdHLJtP9YS298h
> SISee6jpYKmYSC0Ydd4oOwyUN4nURxg0l6whhXegqrW8q6ZJxkVjQCURoPUPH0lL
> rXxCGKfJ6IB+Zj2bMsPvMqrn9wKaH3xxBHzkHDy6JYsxjVYHCXDF02q/OZOG+ln7
> pO0AHoq9BcgsXVfemLzS
> =w+cG
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>



-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From ziade.tarek at gmail.com  Thu Nov 12 12:25:08 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 12:25:08 +0100
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091111152414.GC7023@kinakuta.local>
References: <20091111152414.GC7023@kinakuta.local>
Message-ID: <94bdd2610911120325i2b7534c6kc68fe84bf191d4b2@mail.gmail.com>

On Wed, Nov 11, 2009 at 4:24 PM, Wolodja Wentland
<wentland at cl.uni-heidelberg.de> wrote:
[..]
> Proposal
> --------
>
> I propose three things:
>
> 1. Save installation prefixes for every installed distribution
> 2. Define standard infixes for typical data file classes
> 3. Implement an API that is able to query that information/retrieve data
> ?files

+1 to this three points.


>
> 1 and 3. Discussion
>
> The installation prefixes could for example be saved within a suitable
> file in .egg-info/ which would mandate changes to PEP 376. I could think
> of a PREFIX file and the following ?methods/attribute for e.g. the
> Distribution class
>
> * get_install_prefixes() ? ?-> returns a dictionary with all prefixes for
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? this distribution
>
> * get_data_file(type, name) -> get data file of given type and name (see
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? (see below)

I recall someone proposing to have such a PREFIX file, to keep all
installation prefixes.

I think that's a good enhancement, could you work on a diff on PEP 376 ?

>
> 2. Discussion
>
> The FHS differentiates between various classes of files and defines
> proper location for them. We could define platform dependent
> standard infixes for the following data file classes for distribution
> foo:
>
> * configuration ? ? etc
> * shared data ? ? ? usr/share/foo/
> * readme ? ? ? ? ? ?usr/share/foo/README ? ? ? ?README, TODO, ... could
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?be automatically
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?discovered
> * examples ? ? ? ? ?usr/share/foo/examples
> * documentation ? ? usr/share/foo/doc
> * man files ? ? ? ? usr/share/man
> * variable ? ? ? ? ?var/lib/foo
> * ...
>
> There should be a set of platform dependent prefixes for all classes we
> agree on *and* it should be possible to change/set their default values
> with, for example, environment variables or command line options.

It sounds good for unix like system. But I am curious to see the paths
under windows.

>
> I call them infixes and not prefixes here, because applications like
> virtualenv might want to define a different root for these files than
> the system default one.
>
> Tarek suggested to use this scheme like this:
>
> setup(
> ? ?...
> ? ?data_files = [
> ? ? ? ?('$configuration', ['data/my.conf', 'data/default.conf']),
> ? ? ? ?('$examples', ['doc/examples/do_foo.sample']),
> ? ? ? ?('/i/want/file/here', ['custom_placed_file']),
> ? ? ? ?...
> ? ? ? ?]
> ? ?)
>
> The placeholders will then be replaced with their current value at
> installation time and the locations written to .egg-info/PREFIX.

Yes, and that could be done alongside the installation prefixes. Can
you provide a full example of a PREFIX  file ?


Tarek

From Jim.Vickroy at noaa.gov  Thu Nov 12 12:31:40 2009
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Thu, 12 Nov 2009 04:31:40 -0700
Subject: [Distutils] distutils.core.setup --install-script option in
	Python	2.6 ?
Message-ID: <4AFBF21C.70803@noaa.gov>


Hello,

Please excuse the repost (sent last week), but I'm hoping someone on the 
list can get me on-track with a distutils problem.

I have just upgraded from Python 2.5 to 2.6.4 (MS Windows XP service 
pack 3).

My setup.py scripts (based on distutils.core.setup) no longer work with 
the --install-script option.  Here is the error:

   * error: option --install-script not recognized

The above error occurs when passing either bdist_wininst or bdist_msi 
parameters to the setup.py scripts.

I have been unable to locate any on-line mention of this option not 
being supported in Python 2.6.

I apologize for missing something obvious here, but I am stuck.

Thanks in advance for your feedback.

-- jv

From cournape at gmail.com  Thu Nov 12 13:57:27 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 21:57:27 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <94bdd2610911120314n42d4efeap512f8d91eb6e993f@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
	<4AFBE299.9090303@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120314n42d4efeap512f8d91eb6e993f@mail.gmail.com>
Message-ID: <5b8d13220911120457i749757eboeb3c1ec6bb216d02@mail.gmail.com>

On Thu, Nov 12, 2009 at 8:14 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> Now, there's no reason at all that we can''t add an API in
> distutils.util for example, that let you get
> the build paths, and that will be used by all build_ commands, so you
> don't have to use a build_ command
> to get them.
>
> I propose that you think about its signature, and that I add it after
> we agree on it.

The signature does not really matter I think. Since those parameters
depends on user customization (through config files or command line
flags), it could be something like:

class FinalizedOptions:
    def get_build_dir_library(self):
        pass

    def get_build_dir_src(self):
        pass

    def get_prefix(self):

    ...

And a function like

def get_finalized_options():
    # compute the finalized options
    # Cache it if expensive
    ...
    return FinalizedOptions()

could be used to obtain those options.

what  really matters is:
   - it should be callable from anywhere, at any time: inside
setup.py, in any commands, etc...
   - ideally, it should work for any python >= 2.4 (not every python
version have the exact same conventions IIRC).

>
> by calling "get_paths(scheme, vars)", where scheme is the installation
> scheme you want (unix, nt, etc)
> and vars your variables if you want to override any $VARIABLE.

Assuming the above FinalizedOptions is available, I could obtain the
needed options.

>
> You mean, accessing other commands command line options ? or the result
> of their finalizations ?

The result after finalization.

>
> For me, the install_cmd <-> build_clib problem shows that the "install" command
> do some work that is needed by other commands, and that this work has
> to be taken
> out of it.
>
> A command performs something, and should not be used to set up an *environment*.

I am not sure I understand *environment* in that context. If you mean
options, then each command has its set of options, and yes, that's one
of the fundamental issue I often complain about in distutils. The
build_clib/install interaction I described is typical, and we do this
kind of stuff in pretty much every command.

> Do you agree that this change will help you, and is possible without
> replacing Distutils ?

It would definitely be helpful to get access to any finalized command
option from anywhere at anytime, especially if it is supported for all
the python versions we support (2.4 and above - I don't know what
version distribute intends to support).

Ideally, it would solve a fair shair amount of issues if this was done
at the UI level as well, that is something like

python setup.py build --compiler=mingw -> use mingw compiler for every
compiled code

But that's one of the thing which does not make much sense changing in
distutils, though: it would break almost anyone's workflow.

David

From ziade.tarek at gmail.com  Thu Nov 12 14:36:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 14:36:32 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <5b8d13220911120457i749757eboeb3c1ec6bb216d02@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
	<4AFBE299.9090303@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120314n42d4efeap512f8d91eb6e993f@mail.gmail.com>
	<5b8d13220911120457i749757eboeb3c1ec6bb216d02@mail.gmail.com>
Message-ID: <94bdd2610911120536w5f16d2abu24adf01ac0946fcd@mail.gmail.com>

On Thu, Nov 12, 2009 at 1:57 PM, David Cournapeau <cournape at gmail.com> wrote:
[...]
> The signature does not really matter I think. Since those parameters
> depends on user customization (through config files or command line
> flags), it could be something like:
>
> class FinalizedOptions:
> ? ?def get_build_dir_library(self):
> ? ? ? ?pass
>
> ? ?def get_build_dir_src(self):
> ? ? ? ?pass
>
> ? ?def get_prefix(self):
>
> ? ?...
>
> And a function like
>
> def get_finalized_options():
> ? ?# compute the finalized options
> ? ?# Cache it if expensive
> ? ?...
> ? ?return FinalizedOptions()
>
> could be used to obtain those options.

I was not referring to the finalized options of any commands, but to
the build paths you are trying to get.
e.g. like for the install paths, we can extract from the build
commands, some schemes that can be queried
the same way:

_BUILD_SCHEMES = {
  'unix' :
    {'buildlib': '$base/xxx'
     ...}
  ..
}

and then:

get_build_paths(scheme, vars)


[...]
>> A command performs something, and should not be used to set up an *environment*.
>
> I am not sure I understand *environment* in that context. If you mean
> options, then each command has its set of options, and yes, that's one
> of the fundamental issue I often complain about in distutils. The
> build_clib/install interaction I described is typical, and we do this
> kind of stuff in pretty much every command.

By environment I mean that a command should be a standalone element
that does something,
and is used by distutils with this pattern:

>>> cmd = commandA(distribution)
>>> cmd.ensure_finalized()
>>> cmd.run()

If at some point you need to look to some of its finalized options in
another commandB, it means that
- either commandA is doing more than it is supposed to do: it sets an
*environement* that is reused by others
- either commandB should be a subcommand of commandA
- either commandA doesn't let you do what you want, and you are
building a competiting command

>
>> Do you agree that this change will help you, and is possible without
>> replacing Distutils ?
>
> It would definitely be helpful to get access to any finalized command
> option from anywhere at anytime, especially if it is supported for all
> the python versions we support (2.4 and above - I don't know what
> version distribute intends to support).

The schemes currently, go as far as Python 2.2.

>
> Ideally, it would solve a fair shair amount of issues if this was done
> at the UI level as well, that is something like
>
> python setup.py build --compiler=mingw -> use mingw compiler for every
> compiled code

As a matter of fact, having a "compiler" option in build, that would be reused
by build_ext and other build_ commands, where discussed lately in an issue

> But that's one of the thing which does not make much sense changing in
> distutils, though: it would break almost anyone's workflow.

Why that ? How this would break someone's workflow, since all the changes
we have discussed so far is moving code out of commands to have new APIs
(and have those commands call it), and modifying some command options.

Now there's another pattern emerging in this discussion : paths
options like --prefix can obviously
be shared by several commands.

So, based on the change that makes it possible to get some install and
build schemes,
what about having global options that will be used to built a global
environment dictionary containing paths:

$ python setup.py --var1 --var2 --var3 cmd1 cmd2 cmd3


then, var1, and var2  etc.. are put in a "vars" dict, that is passed
to get_paths(scheme, vars).
And a global "options" dict is filled in the Distribution object
(where scheme is a key for the current platform):

Distribution.options = get_paths(scheme, vars)

A further improvement would be to be able to register a function that
is called when the options dict is flled,
allowing a third party code to work on the options. The existing
commands in distutils would be changed accordingly: they'll look into
the global options dict for finalized options, and would fallback
using the old pattern (local options + finalization)

Tarek

From cournape at gmail.com  Thu Nov 12 15:06:11 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 12 Nov 2009 23:06:11 +0900
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <94bdd2610911120536w5f16d2abu24adf01ac0946fcd@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
	<4AFBE299.9090303@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120314n42d4efeap512f8d91eb6e993f@mail.gmail.com>
	<5b8d13220911120457i749757eboeb3c1ec6bb216d02@mail.gmail.com>
	<94bdd2610911120536w5f16d2abu24adf01ac0946fcd@mail.gmail.com>
Message-ID: <5b8d13220911120606s6a2a43b0h9d6f0c03405e1825@mail.gmail.com>

On Thu, Nov 12, 2009 at 10:36 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Nov 12, 2009 at 1:57 PM, David Cournapeau <cournape at gmail.com> wrote:
> [...]
>> The signature does not really matter I think. Since those parameters
>> depends on user customization (through config files or command line
>> flags), it could be something like:
>>
>> class FinalizedOptions:
>> ? ?def get_build_dir_library(self):
>> ? ? ? ?pass
>>
>> ? ?def get_build_dir_src(self):
>> ? ? ? ?pass
>>
>> ? ?def get_prefix(self):
>>
>> ? ?...
>>
>> And a function like
>>
>> def get_finalized_options():
>> ? ?# compute the finalized options
>> ? ?# Cache it if expensive
>> ? ?...
>> ? ?return FinalizedOptions()
>>
>> could be used to obtain those options.
>
> I was not referring to the finalized options of any commands, but to
> the build paths you are trying to get.

But those are linked. Build directories are customizable through
pretty much any build_command (I see at least build_clib, build_ext
and build on python 2.6).

So :

>
> get_build_paths(scheme, vars)
>

This should return different value depending on whether the in place
build is set up, or the develop command is called, etc...

> If at some point you need to look to some of its finalized options in
> another commandB, it means that
> - either commandA is doing more than it is supposed to do: it sets an
> *environement* that is reused by others
> - either commandB should be a subcommand of commandA
> - either commandA doesn't let you do what you want, and you are
> building a competiting command

Those differences do not make much sense. If distutils wants command
foo to be a subcommand of bar, and we need a different ordering, how
can we do it ?

Concerning different options from different commands, I quickly looked
at build, install and build_ext commands options (the ones from
straight distutils). Very few options concerned only the command it
belongs to. For example, the install command list is:

  --prefix            installation prefix
  --exec-prefix       (Unix only) prefix for platform-specific files
  --home              (Unix only) home directory to install under
  --user              install in user site-package
                      '/Users/david/.local/lib/python2.6/site-packages'
  --install-base      base installation directory (instead of --prefix or --
                      home)
  --install-platbase  base installation directory for platform-specific files
                      (instead of --exec-prefix or --home)
  --root              install everything relative to this alternate root
                      directory
  --install-purelib   installation directory for pure Python module
                      distributions
  --install-platlib   installation directory for non-pure module distributions
  --install-lib       installation directory for all module distributions
                      (overrides --install-purelib and --install-platlib)
  --install-headers   installation directory for C/C++ headers
  --install-scripts   installation directory for Python scripts
  --install-data      installation directory for data files
  --compile (-c)      compile .py to .pyc [default]
  --no-compile        don't compile .py files
  --optimize (-O)     also compile with optimization: -O1 for "python -O", -O2
                      for "python -OO", and -O0 to disable [default: -O0]
  --force (-f)        force installation (overwrite any existing files)
  --skip-build        skip rebuilding everything (for testing/debugging)
  --record            filename in which to record list of installed files

Maybe --record is not needed, but I can see a usage for all the other
ones. For example, if I have a scons builder to compile/optimize
python files, I need to know it in my scons command.

>
> The schemes currently, go as far as Python 2.2.

cool

>
>>
>> Ideally, it would solve a fair shair amount of issues if this was done
>> at the UI level as well, that is something like
>>
>> python setup.py build --compiler=mingw -> use mingw compiler for every
>> compiled code
>
> As a matter of fact, having a "compiler" option in build, that would be reused
> by build_ext and other build_ commands, where discussed lately in an issue

When I was considering this example, I had in mind that any option
which can be customized in any build_ command should be customizable
in build. Compiler, compiler options, in-place vs non in-place, etc...
Pretty much any option.

> Why that ? How this would break someone's workflow, since all the changes
> we have discussed so far is moving code out of commands to have new APIs
> (and have those commands call it), and modifying some command options.

One of the problem is dealing with inconsistencies between
subcommands. Modifying their behavior has many corner cases. One of
the big win with numscons was that build was handled by only one
command, so the inconsistencies disappear.

>
> Now there's another pattern emerging in this discussion : paths
> options like --prefix can obviously
> be shared by several commands.
>
> So, based on the change that makes it possible to get some install and
> build schemes,
> what about having global options that will be used to built a global
> environment dictionary containing paths:
>
> $ python setup.py --var1 --var2 --var3 cmd1 cmd2 cmd3
>
>
> then, var1, and var2  etc.. are put in a "vars" dict, that is passed
> to get_paths(scheme, vars).

As mentioned above, please keep in mind that many of those options see
their behavior modified (directly or indirectly) through commands
options. Think about inplace option, develop command, etc...

David

From ziade.tarek at gmail.com  Thu Nov 12 15:55:21 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 15:55:21 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <5b8d13220911120606s6a2a43b0h9d6f0c03405e1825@mail.gmail.com>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<4AFBAC92.9050304@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120140v1ae9807ewf7d10eb06168b565@mail.gmail.com>
	<4AFBE299.9090303@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120314n42d4efeap512f8d91eb6e993f@mail.gmail.com>
	<5b8d13220911120457i749757eboeb3c1ec6bb216d02@mail.gmail.com>
	<94bdd2610911120536w5f16d2abu24adf01ac0946fcd@mail.gmail.com>
	<5b8d13220911120606s6a2a43b0h9d6f0c03405e1825@mail.gmail.com>
Message-ID: <94bdd2610911120655p21c9521ct9c027b670e2579c7@mail.gmail.com>

On Thu, Nov 12, 2009 at 3:06 PM, David Cournapeau <cournape at gmail.com> wrote:
[..]
>>
>> I was not referring to the finalized options of any commands, but to
>> the build paths you are trying to get.
>
> But those are linked. Build directories are customizable through
> pretty much any build_command (I see at least build_clib, build_ext
> and build on python 2.6).

yes, like install can customize the install paths with --prefix. but
at the end, they
you just give them root paths, or things like that, and they return build paths.

So, as we said for the install command, this mechanism can be
extracted and put in an API,
that takes variables to generate build paths, and that can be used by
those commands.

Meaning that you won't need them anymore to get your build paths.

>
> So :
>
>>
>> get_build_paths(scheme, vars)
>>
>
> This should return different value depending on whether the in place
> build is set up, or the develop command is called, etc...

yes, like platlib and purelib. Build schemes will have several paths indeed.
we need to list them.

>
>> If at some point you need to look to some of its finalized options in
>> another commandB, it means that
>> - either commandA is doing more than it is supposed to do: it sets an
>> *environement* that is reused by others
>> - either commandB should be a subcommand of commandA
>> - either commandA doesn't let you do what you want, and you are
>> building a competiting command
>
> Those differences do not make much sense. If distutils wants command
> foo to be a subcommand of bar, and we need a different ordering, how
> can we do it ?

I don't understand this question.

What is the need for you to order two commands ?

This pattern is useful only if a command uses what another command has produced.

For example:

$ python setup.py sdist upload

-> upload requires sdist to be run so it can upload an archive

If ordering means that you need a command to be run, so you can get its
finalized options for your own command, something is wrong. It means that
you are unable to build a command that is standalone, and that instead
of using an
API or the Distribution class on its own, you need to work with another command.

That is what wee need to change, by externalizing for the commands the code
so you can reuse that.


[..]
>>
>> $ python setup.py --var1 --var2 --var3 cmd1 cmd2 cmd3
>>
>>
>> then, var1, and var2 ?etc.. are put in a "vars" dict, that is passed
>> to get_paths(scheme, vars).
>
> As mentioned above, please keep in mind that many of those options see
> their behavior modified (directly or indirectly) through commands
> options. Think about inplace option, develop command, etc...

What's an option behavior ? You mean the value is changed ?

The develop command is not part of Distutils and it's doing a big hack
on calling the build_ext
command and changing the inplace option on the fly like this.

It should instead, like what the upload command does, leave it up to any build
command to build the extensions in place. And just work on adding a
.pth file for example,
that adds the source to the path. That would prevent troubles.

From tseaver at palladion.com  Thu Nov 12 18:55:37 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 12 Nov 2009 12:55:37 -0500
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <4AFB9865.5090400@ar.media.kyoto-u.ac.jp>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>	<hdfjlk$6km$1@ger.gmane.org>	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<4AFB9865.5090400@ar.media.kyoto-u.ac.jp>
Message-ID: <hdhi6q$p0n$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Cournapeau wrote:
> Tarek Ziad? wrote:
>> I want to improve Distutils, but not for the price of a complete drop.
>> I don't think the edge cases we
>> are discussing worth it, and I still fail to see why we can't work
>> them out in the context of the existing
>> tool.
>>   
> 
> Well, the scientific python edge case is what Guido asked about in the
> original thread. That's our answer: we don't think something which is
> not fundamentally different than distutils will solve our problems. If
> you think that what we considered as fundamental issues are corner cases
> and implementation details, we can only agree to disagree, I guess.
> 
> Now, it is legitimate to dismiss our use-cases as edge cases and not
> worthwhile to worry about. We are not certainly not as big as say the
> web development community. But the question was directed at us, as I
> understood it :)

FWIW, I am a web developer who agrees wholeheartedly that trying to fix
distutils incrementally is going to be way harder / more painful, and
take *much* longer, than writing something which abandons the "100%
backward compatibility" goal completely.  The phantom "distutils API"
doesn't exist, which means that almost any improvement (and I don't deny
that such improvements are possible) potentially breaks substantial
installed codebases.  In this sense, fixing distutils is like working
without tests, because you don't know until too late that you've broken
something.

If we just don't call the new thing "distutils", we get to keep "100%
backward compatibility" for free:  people can begin switching to the new
thing incrementally / opportunistically, but their existing code doesn't
blow up.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkr8TBQACgkQ+gerLs4ltQ7wNACeNWDIdwAn1xasjs/zbJkiSU/0
k7UAniByqvgshfNsiXy5KPxQxvwd9OHE
=ZUZt
-----END PGP SIGNATURE-----


From ziade.tarek at gmail.com  Thu Nov 12 19:55:59 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 19:55:59 +0100
Subject: [Distutils] Improving distutils vs redesigning it (was people
	want CPAN)
In-Reply-To: <hdhi6q$p0n$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<4AFB9865.5090400@ar.media.kyoto-u.ac.jp> <hdhi6q$p0n$1@ger.gmane.org>
Message-ID: <94bdd2610911121055v522e497id76bed7c5e158f4b@mail.gmail.com>

On Thu, Nov 12, 2009 at 6:55 PM, Tres Seaver <tseaver at palladion.com> wrote:
[..]
> The phantom "distutils API"
> doesn't exist, which means that almost any improvement (and I don't deny
> that such improvements are possible) potentially breaks substantial
> installed codebases. ?In this sense, fixing distutils is like working
> without tests, because you don't know until too late that you've broken
> something.

First of all, I have not said that Distutils will stay backward
compatible. If the compatibility
is broken at some point, it'll have a deprecation process, that's all.

And a backport will be provided for the previous Python versions.

But we can't talk about breaking compatibility until we do have a new
design in our
hands that fixes some problems, and require to break the comaptibility.

Next, the distutils code base is test-covered around 80% now
(depending on your platform)
Maybe you ignore about it, but I have been working to raise the test coverage
during the last year, (it went from 20% to 80%)

Next, there are buildbots now that builds projects like NumPy to make
sure the trunk still works.

Moving some code from the commands to some new APIs will not blow those
commands because they are tested.

Meaning that the old tests will have to still pass once some code has
been moved.

That's the whole point of test coverage, and a basic rule in refactoring code.

So I don't really understand why you are saying that we will go blind
in refactoring some stuff.

Last, I am now not listening anymore to this "distutils must be fully
replaced" answers, they are going
no where, because Distutils will not be dropped for the reasons I've
already mentioned earlier.

I'll focus on the design work we are doing with David now to enhance
Distutils on his use cases.
As a matter of fact, we have made progress in sharing knowledge and
trying to find designs that fits
our brains, and that the only stuff that matters to me at this point.

And if it doesn't make it to Distutils 2.7/3.2, I expect Distribute to
be the incubator of those particular
changes.

If some some people want to build a brand new shiny tool from scratch,
good luck then. If some others
want to help in Distutils, welcome then.

Tarek

From pav+sp at iki.fi  Thu Nov 12 12:31:50 2009
From: pav+sp at iki.fi (Pauli Virtanen)
Date: Thu, 12 Nov 2009 11:31:50 +0000 (UTC)
Subject: [Distutils] People want CPAN
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
Message-ID: <hdgrn6$4l9$1@ger.gmane.org>

Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziad? wrote:
[clip]
>>>>>> get_install_paths('FOO')
>>>
>>> And that's the API we want to add in sysconfig, roughly.
>>
>> That does not solve the problem about getting FOO in the first place
>> when calling get_install_path.
> 
> Why that ? where "FOO" comes from ? if it's an option you provide at
> build time like you said, earlier, you just pass it to the API to get
> the paths.
> 
> FOO is not coming from nowhere...

If this is not painfully clear already, the user passes FOO to the command

	python setup.py install --prefix=FOO

Now, clearly the distutils install subcommand knows what FOO is.
But does build_ext? Does sysconfig?


From jkleint at gmail.com  Thu Nov 12 21:42:08 2009
From: jkleint at gmail.com (John Kleint)
Date: Thu, 12 Nov 2009 15:42:08 -0500
Subject: [Distutils] People want CPAN :-)
Message-ID: <6496ee60911121242t730f5468sc22fbf4e3cdd944f@mail.gmail.com>

>Tarek Ziad? schrieb:
>
>> For the documentation part I am afraid it will be messy for the end
>> users trying to package apps in Python *until* all PEPs
>> have made it into Python.
>>
>> Although, as Ian Bicking says: we could write today some kind of
>> all-in-one tutorial so end-users can work out without having to run
>> after the documenation in several places.
>
>Big +1, and I would be more than willing to include it in the standard
>Python documentation, *even* if it mostly describes setuptools/distribute/pip.
>
>When people want to package a library, they *will* look for docs in Python,
>and at the moment they only find the distutils reference.  While the latter
>is necessary, a more howto-like standard document is much needed.
>
>Georg


I have the start of such a document, derived from my own recent
experience pulling together all the information needed to release a
new Python project.  It's aimed at people new to packaging and
possibly open source in general, so it's broader than just packaging.
It includes things like basic file layout, unit testing, Pylint,
getting started with Sphinx, a sample (distutils) setup.py, and how to
register with PyPI.  I try to pull together all the info a novice
would need to have a reasonable shot at sharing their code.  It's by
no means all-encompassing (e.g. no mention of native extensions), but
perhaps someone would be willing to write a complimentary advanced
packaging howto.  You can find it at:

http://infinitemonkeycorps.net/docs/pph/

Let me know what you think.

-John

From setuptools at bugs.python.org  Thu Nov 12 22:48:09 2009
From: setuptools at bugs.python.org (Zooko O'Whielacronx)
Date: Thu, 12 Nov 2009 21:48:09 +0000
Subject: [Distutils] [issue95] [PATCH] ez_setup.py fails when pkg_resources
	is installed	but setuptools is not
In-Reply-To: <1258062489.37.0.55142991522.issue95@psf.upfronthosting.co.za>
Message-ID: <1258062489.37.0.55142991522.issue95@psf.upfronthosting.co.za>


New submission from Zooko O'Whielacronx <zooko at zooko.com>:

This just happened on Ubuntu Karmic:

HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ /usr/bin/python
setup.py test
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    use_setuptools()
  File "/sdb/zooko/ecometry2rabbit/ecometry2rabbit/ez_setup.py", line 116, in
use_setuptools
    if not setuptools_is_new_enough(min_version):
  File "/sdb/zooko/ecometry2rabbit/ecometry2rabbit/ez_setup.py", line 88, in
setuptools_is_new_enough
    pkg_resources.require('setuptools >= %s' % (required_version,))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: setuptools>=0.6c7

As you can see from the following, the pkg_resources module is importable even
though setuptools is not installed:

HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ /usr/bin/python
-c 'import pkg_resources;print pkg_resources;print
pkg_resources.require("setuptools")'
<module 'pkg_resources' from '/usr/lib/python2.6/dist-packages/pkg_resources.pyc'>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 626, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 524, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: setuptools

This is because Debian and Ubuntu package pkg_resources separately:

HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ dpkg --search
pkg_resources.py
python-pkg-resources: /usr/share/pyshared/pkg_resources.py
python-pkg-resources: /usr/lib/python2.5/site-packages/pkg_resources.py
python-pkg-resources: /usr/lib/python2.6/dist-packages/pkg_resources.py
HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ dpkg --list
python-pkg-resources
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version              
                                Description
+++-=====================================================-=====================================================-==========================================================================================================================
ii  python-pkg-resources                                  0.6c9-0ubuntu5       
                                Package Discovery and Resource Access using
pkg_resources
HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ dpkg --list
python-setuptools
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version              
                                Description
+++-=====================================================-=====================================================-==========================================================================================================================
un  python-setuptools                                     <none>               
                                (no description available)

The following patch makes ez_setup.py treat this as a case where it needs to
bootstrap setuptools itself:

diff --git a/ez_setup.py b/ez_setup.py
index 6dd60f9..8b2ad2a 100644
--- a/ez_setup.py
+++ b/ez_setup.py
@@ -86,6 +86,11 @@ def setuptools_is_new_enough(required_version):
         else:
             try:
                 pkg_resources.require('setuptools >= %s' % (required_version,))
+            except pkg_resources.DistributionNotFound:
+                # No setuptools installed (even though pkg_resources is
installed)...
+                pkg_resources.__dict__.clear() # "If you want to be absolutely
sure... before deleting it." --said PJE on IRC
+                del sys.modules['pkg_resources']
+                return False
             except pkg_resources.VersionConflict:
                 # An insufficiently new version is installed.
                 pkg_resources.__dict__.clear() # "If you want to be absolutely
sure... before deleting it." --said PJE on IRC

----------
messages: 462
nosy: zooko
priority: bug
status: unread
title: [PATCH] ez_setup.py fails when pkg_resources is installed but setuptools is not

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue95>
_______________________________________________

From ziade.tarek at gmail.com  Thu Nov 12 23:12:28 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 23:12:28 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <hdgrn6$4l9$1@ger.gmane.org>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
Message-ID: <94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>

On Thu, Nov 12, 2009 at 12:31 PM, Pauli Virtanen <pav+sp at iki.fi> wrote:
> Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziad? wrote:
> [clip]
>>>>>>> get_install_paths('FOO')
>>>>
>>>> And that's the API we want to add in sysconfig, roughly.
>>>
>>> That does not solve the problem about getting FOO in the first place
>>> when calling get_install_path.
>>
>> Why that ? where "FOO" comes from ? if it's an option you provide at
>> build time like you said, earlier, you just pass it to the API to get
>> the paths.
>>
>> FOO is not coming from nowhere...
>
> If this is not painfully clear already, the user passes FOO to the command
>
> ? ? ? ?python setup.py install --prefix=FOO
>
> Now, clearly the distutils install subcommand knows what FOO is.
> But does build_ext? Does sysconfig?

The install command takes FOO, and build several paths with it:

/FOO/lib/site-package/
/FOO/xxx

The other commands that needs to get the same path can rebuild it using an API,
that does:

get_paths(scheme, vars={'prefix': 'FOO')

Instead of doing what David did:

$ python setup.py build_ext install --prefix=FOO

They can do:

$ python setup.py build_ext --prefix=FOO

and don't require to use the install command anymore to get these paths cooked.

Remember that the build step has nothing to do with the install step,
and David in his example was doing a build that was not relocatable,
e.g. receiving the install paths and applying them in the binaries.

So instead of having this paths created by code in the install command,
they are moved in an API than can be used by install or by any other command.

Is that clearer ?

Tarek

From ziade.tarek at gmail.com  Thu Nov 12 23:19:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 23:19:37 +0100
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <6496ee60911121242t730f5468sc22fbf4e3cdd944f@mail.gmail.com>
References: <6496ee60911121242t730f5468sc22fbf4e3cdd944f@mail.gmail.com>
Message-ID: <94bdd2610911121419t402a934dufd35c6ddc683a2c3@mail.gmail.com>

On Thu, Nov 12, 2009 at 9:42 PM, John Kleint <jkleint at gmail.com> wrote:
>>Tarek Ziad? schrieb:
>>
>>> For the documentation part I am afraid it will be messy for the end
>>> users trying to package apps in Python *until* all PEPs
>>> have made it into Python.
>>>
>>> Although, as Ian Bicking says: we could write today some kind of
>>> all-in-one tutorial so end-users can work out without having to run
>>> after the documenation in several places.
>>
>>Big +1, and I would be more than willing to include it in the standard
>>Python documentation, *even* if it mostly describes setuptools/distribute/pip.
>>
>>When people want to package a library, they *will* look for docs in Python,
>>and at the moment they only find the distutils reference. ?While the latter
>>is necessary, a more howto-like standard document is much needed.
>>
>>Georg
>
>
> I have the start of such a document, derived from my own recent
> experience pulling together all the information needed to release a
> new Python project. ?It's aimed at people new to packaging and
> possibly open source in general, so it's broader than just packaging.
> It includes things like basic file layout, unit testing, Pylint,
> getting started with Sphinx, a sample (distutils) setup.py, and how to
> register with PyPI. ?I try to pull together all the info a novice
> would need to have a reasonable shot at sharing their code. ?It's by
> no means all-encompassing (e.g. no mention of native extensions), but
> perhaps someone would be willing to write a complimentary advanced
> packaging howto. ?You can find it at:
>
> http://infinitemonkeycorps.net/docs/pph/
>
> Let me know what you think.

That's a great start indeed. If we can expand on the packaging thing we could
take it from here and see who wants to help on this. I could try to convince
some developers from pip and virtualenv to help on this.

Are you willing to share it in a DVCS or something for that work ? How
would you like
to manage contributions in it ?

Tarek

From ziade.tarek at gmail.com  Thu Nov 12 23:28:21 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 12 Nov 2009 23:28:21 +0100
Subject: [Distutils] distutils.core.setup --install-script option in
	Python 2.6 ?
In-Reply-To: <4AFBF21C.70803@noaa.gov>
References: <4AFBF21C.70803@noaa.gov>
Message-ID: <94bdd2610911121428h638d926bt771aafb3c146d622@mail.gmail.com>

On Thu, Nov 12, 2009 at 12:31 PM, Jim Vickroy <Jim.Vickroy at noaa.gov> wrote:
>
> Hello,
>
> Please excuse the repost (sent last week), but I'm hoping someone on the
> list can get me on-track with a distutils problem.
>
> I have just upgraded from Python 2.5 to 2.6.4 (MS Windows XP service pack
> 3).
>
> My setup.py scripts (based on distutils.core.setup) no longer work with the
> --install-script option. ?Here is the error:
>
> ?* error: option --install-script not recognized

Hi Jim,

the option (in 2.5 and 2.6) is "--install-scripts" (notice the final 's')

Regards
Tarek

From cournape at gmail.com  Thu Nov 12 23:45:35 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Nov 2009 07:45:35 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
Message-ID: <5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>

On Fri, Nov 13, 2009 at 7:12 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Nov 12, 2009 at 12:31 PM, Pauli Virtanen <pav+sp at iki.fi> wrote:
>> Thu, 12 Nov 2009 10:54:41 +0100, Tarek Ziad? wrote:
>> [clip]
>>>>>>>> get_install_paths('FOO')
>>>>>
>>>>> And that's the API we want to add in sysconfig, roughly.
>>>>
>>>> That does not solve the problem about getting FOO in the first place
>>>> when calling get_install_path.
>>>
>>> Why that ? where "FOO" comes from ? if it's an option you provide at
>>> build time like you said, earlier, you just pass it to the API to get
>>> the paths.
>>>
>>> FOO is not coming from nowhere...
>>
>> If this is not painfully clear already, the user passes FOO to the command
>>
>> ? ? ? ?python setup.py install --prefix=FOO
>>
>> Now, clearly the distutils install subcommand knows what FOO is.
>> But does build_ext? Does sysconfig?
>
> The install command takes FOO, and build several paths with it:
>
> /FOO/lib/site-package/
> /FOO/xxx
>
> The other commands that needs to get the same path can rebuild it using an API,
> that does:
>
> get_paths(scheme, vars={'prefix': 'FOO')
>
> Instead of doing what David did:
>
> $ python setup.py build_ext install --prefix=FOO
>
> They can do:
>
> $ python setup.py build_ext --prefix=FOO
>
> and don't require to use the install command anymore to get these paths cooked.

I think I was confusing with my rpath example, which may be the source
of our misunderstanding. I don't want to add a --prefix option to
build_ext. I want to support the following user cases:

python setup.py build # here, prefix would be whatever default, as
available from sysconfig
python setup.py install --prefix=foo # here, prefix would be the one
as computer by install command
python setup.py build_ext -i # here prefix is the current directory

Requiring users to handle new options of commands is impractical IMHO,
and a prefix option to build has a strange feeling to it.

> Remember that the build step has nothing to do with the install step

Ideally, what we want here is like autoconf does it. You have a
configure step, and then at build/install stages, you have access to
*all* options. Those options can be customized at build through usual
make mechanism, but this is taken into account as well without the
makefile writer having to care.

IOW, we need to know *all* the finalized options *before* the first
build command is even run.

David

From ziade.tarek at gmail.com  Fri Nov 13 00:47:35 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 13 Nov 2009 00:47:35 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
Message-ID: <94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>

On Thu, Nov 12, 2009 at 11:45 PM, David Cournapeau <cournape at gmail.com> wrote:
[..]
> I think I was confusing with my rpath example, which may be the source
> of our misunderstanding. I don't want to add a --prefix option to
> build_ext. I want to support the following user cases:
>
> python setup.py build # here, prefix would be whatever default, as
> available from sysconfig
> python setup.py install --prefix=foo # here, prefix would be the one
> as computer by install command
> python setup.py build_ext -i # here prefix is the current directory
>
> Requiring users to handle new options of commands is impractical IMHO,
> and a prefix option to build has a strange feeling to it.

I am not sure to follow here: let's forget about your example where
you call build_src and install together.

-> in the real world, how a --prefix passed to install is going to
impact a build command and vice-versa ?

install just copy files, where it's told to copy them, and build does
some work with some options,
and as a matter of fact you seem to use installation prefix at this stage.

So, if you are not *installing*, it doesn't make sense to call the
*install* command, and build could
have its --prefix option in your case.


>
>> Remember that the build step has nothing to do with the install step
>
> Ideally, what we want here is like autoconf does it. You have a
> configure step, and then at build/install stages, you have access to
> *all* options. Those options can be customized at build through usual
> make mechanism, but this is taken into account as well without the
> makefile writer having to care.
>
> IOW, we need to know *all* the finalized options *before* the first
> build command is even run.

So, IOW, these options cannot be finalized by *any* command. So the
proposal I've made
to have global options that are common to all commands, and that can
be used to create
an execution environment through public APIs would help in there.

So if the install command transforms "prefix" into "path1" "path2" and
"path3", we need this transformation
to occur outside install, so it can be done before the commands are run.

that is:

$ python setup.py --prefix=foo cmd1 cmd1 etc

and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}


Tarek

From cournape at gmail.com  Fri Nov 13 01:08:36 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Nov 2009 09:08:36 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<5b8d13220911111503u4b25ce8bv7d58b1e71b932895@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
Message-ID: <5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>

On Fri, Nov 13, 2009 at 8:47 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

>
> I am not sure to follow here: let's forget about your example where
> you call build_src and install together.
>
> -> in the real world, how a --prefix passed to install is going to
> impact a build command and vice-versa ?

In many cases. I gave the rpath example, but there is also the
pkg-config-like files generation example, etc... It is not possible to
foresee usage of this in a build tool, any sufficiently complex
project will need to share those options, and different ones depending
on the tool. I think it is a uncontroversial design decision followed
by almost every build tool.


> So, if you are not *installing*, it doesn't make sense to call the
> *install* command, and build could
> have its --prefix option in your case.

I don't *want* to call the install command, but I want to know the
prefix option of install. I do not want a build specific prefix
option, I want to know the global install option, whatever the user
command lines are.

prefix is only an example - as I mentioned in my previous email, I
potentially need every option of install in build command.

> $ python setup.py --prefix=foo cmd1 cmd1 etc
>
> and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}

This is a major change in distutils behavior, so we need to solve the
following issues:
 - every user will have to change how to call distutils
 - what happens if people still use python setup.py install
--option1=foo instead of python setup.py --prefix=option1 install

cheers,

David

From ziade.tarek at gmail.com  Fri Nov 13 01:16:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 13 Nov 2009 01:16:37 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>
	<4AFB697E.5030803@canterbury.ac.nz>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
Message-ID: <94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>

On Fri, Nov 13, 2009 at 1:08 AM, David Cournapeau <cournape at gmail.com> wrote:
[..]
>
>> $ python setup.py --prefix=foo cmd1 cmd1 etc
>>
>> and the result would be in Distribution.options = {'path1': xxx, 'path2': xx}
>
> This is a major change in distutils behavior, so we need to solve the
> following issues:
> ?- every user will have to change how to call distutils
> ?- what happens if people still use python setup.py install
> --option1=foo instead of python setup.py --prefix=option1 install

A deprecation warning would be added in install, if it finds a local
option, rather
than a global. Meaning both would work in 2.7/3.2.

That would give them 18 to 24 months * 2 to go with the new style.

Tarek

From kevin at bud.ca  Fri Nov 13 05:48:45 2009
From: kevin at bud.ca (Kevin Teague)
Date: Thu, 12 Nov 2009 20:48:45 -0800
Subject: [Distutils] People want CPAN :-)
In-Reply-To: <6496ee60911121242t730f5468sc22fbf4e3cdd944f@mail.gmail.com>
References: <6496ee60911121242t730f5468sc22fbf4e3cdd944f@mail.gmail.com>
Message-ID: <923D0AB4-CBC4-4E69-A5C8-670F31C8B079@bud.ca>


On Nov 12, 2009, at 12:42 PM, John Kleint wrote:
>
> I have the start of such a document, derived from my own recent
> experience pulling together all the information needed to release a
> new Python project.  It's aimed at people new to packaging and
> possibly open source in general, so it's broader than just packaging.
> It includes things like basic file layout, unit testing, Pylint,
> getting started with Sphinx, a sample (distutils) setup.py, and how to
> register with PyPI.  I try to pull together all the info a novice
> would need to have a reasonable shot at sharing their code.  It's by
> no means all-encompassing (e.g. no mention of native extensions), but
> perhaps someone would be willing to write a complimentary advanced
> packaging howto.  You can find it at:
>
> http://infinitemonkeycorps.net/docs/pph/
>
> Let me know what you think.
>

It's a good start, but a few critques on the packaging section:

  * Avoid importing your package before you've installed it.

   Importing a package works fine for packages which have no  
dependencies, but once your package depends upon other packages you  
can't use this technique. It's better to teach people how to write  
setup.py scripts which don't import the package so that their  
setup.py's can be consistent once they do write packages with  
dependencies. There are lots of cases where the assumptions of the  
configuration of a developer's environment where dependencies are  
installed that won't work for other people trying to install such a  
package, which leads to setup.py files which don't work.

   One example for handling the long_description field can be done  
with (there are lots of variants on this, and you can just call open()  
for a single text file:

def read(*rnames):
     return open(os.path.join(os.path.dirname(__file__),  
*rnames)).read()

long_description=read(
     read('README.txt')
     + '\n' +
     'Detailed Documentation\n'
     '**********************\n'
     + '\n' +
     'src/some-other-file.txt',
)


  * download_url field is not necessary.

    Libraries are often installed automatically by tools (buildout,  
pip, easy_install). This work best if the downloads are in a  
consistent structure (e.g. on PyPI). If releases are made to PyPI,  
then you don't need a download_url to get to another place where the  
file can be downloaded. The field is also prone to mistakes, since  
easy_install will follow the download URL and screen-scrape that page  
looking for a download link. Which can lead to older or wrong versions  
of the package being downloaded. Basically it just causes room for  
errors, and is just one extra step that you need to do to create a  
package, so why do it?

  * package_dir location.

     This is a very minor point, but I find putting the source in a  
directory named after the module or package which isn't the module or  
package confusing. A common alternative is to use 'src' as the name  
for the source directory. Then that location can be consistent from  
package to package.

  * provides (and requires) are scheduled for deprecation

    These fields don't make any sense. If you are listing the imports  
that your package needs, you need to list the name of the distribution  
which provides those packages, not the names of the imports that your  
code is using. setuptools (or now distribute) lets you use an  
'install_requires' field, and this fields is also likely to be in  
Distutils in the next Python release.


  * Use a tool to help automate releases

    Creating each release requires a lot of manual changes here and  
there (bump the version, update the changelog, run sdist, upload to  
pypi). In the spirit of "release early, release often", I like to use  
a tool to automate releases, and also helps establish a few more  
common conventions between Python projects. I'm using zest.releaser  
for this ATM (shameless plug, since I'm also a contributor to the  
project):

    http://pypi.python.org/pypi/zest.releaser




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091112/bc683f31/attachment.htm>

From david at ar.media.kyoto-u.ac.jp  Fri Nov 13 06:22:23 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 13 Nov 2009 14:22:23 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>	<94bdd2610911111548s2b98e55fnb43b80b5573496b1@mail.gmail.com>	<4AFB697E.5030803@canterbury.ac.nz>	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>	<hdgrn6$4l9$1@ger.gmane.org>	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
	<94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
Message-ID: <4AFCED0F.40703@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
> A deprecation warning would be added in install, if it finds a local
> option, rather
> than a global. Meaning both would work in 2.7/3.2.
>   

If changing the command line in incompatible ways is acceptable, what do
you think of scrapping the commands (at the UI level only) altogether ?
This would be more consistent and easier to deal for the user, and
easier to implement as well:

python setup.py configure --option1 --option2=value2 ....
python setup.py build
python setup.py install

We could then make this work:

python setup.py install (would run both build and configure implicitly).
Making all finalized options available at the build stage would then be
easier.

David

From cournape at gmail.com  Fri Nov 13 10:22:06 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Nov 2009 18:22:06 +0900
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091111152414.GC7023@kinakuta.local>
References: <20091111152414.GC7023@kinakuta.local>
Message-ID: <5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>

On Thu, Nov 12, 2009 at 12:24 AM, Wolodja Wentland
<wentland at cl.uni-heidelberg.de> wrote:

>
> The FHS differentiates between various classes of files and defines
> proper location for them. We could define platform dependent
> standard infixes for the following data file classes for distribution
> foo:
>
> * configuration ? ? etc
> * shared data ? ? ? usr/share/foo/
> * readme ? ? ? ? ? ?usr/share/foo/README ? ? ? ?README, TODO, ... could
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?be automatically
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?discovered
> * examples ? ? ? ? ?usr/share/foo/examples
> * documentation ? ? usr/share/foo/doc
> * man files ? ? ? ? usr/share/man
> * variable ? ? ? ? ?var/lib/foo
> * ...
>

One could start from all the autoconf-defined variables such as
prefix, eprefix, sbindir, bindir, etc... One would need in addition
some variables for python files (.py, pyc and pyo). Then, each scheme
would just be different mapping for those variables. For example,
--user option just means changing prefix to $HOME/.local on linux,
gobolinux (which as a similar file organization as mac os x) could
easily package as they do for autoconf packages.

For data files, the major problems currently in distutils are:
 -  that you cannot 'tag' the different kind of data files. Tarek
suggestion of using ('$variable_name', [list of files]) is one way of
solving this. One could also imagine having an argument to setup per
type (man_files=(list of files), etc...).
  - One should think about defining its own data 'type' as well. For
example, let's say application MegaBar can be extended through python
plugins. You may want to add your plugin to MegaBar plugin directory:
($megabar_dir, [list of files for megabar plugins]). megabar_dir value
should be customizable from the build process.

cheers,

David

From ziade.tarek at gmail.com  Fri Nov 13 11:26:23 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 13 Nov 2009 11:26:23 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <4AFCED0F.40703@ar.media.kyoto-u.ac.jp>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
	<94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
	<4AFCED0F.40703@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911130226u7eedaa4dg21f7c1d3598449b4@mail.gmail.com>

On Fri, Nov 13, 2009 at 6:22 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Tarek Ziad? wrote:
>> A deprecation warning would be added in install, if it finds a local
>> option, rather
>> than a global. Meaning both would work in 2.7/3.2.
>>
>
> If changing the command line in incompatible ways is acceptable, what do
> you think of scrapping the commands (at the UI level only) altogether ?
> This would be more consistent and easier to deal for the user, and
> easier to implement as well:
>
> python setup.py configure --option1 --option2=value2 ....
> python setup.py build
> python setup.py install
>
> We could then make this work:
>
> python setup.py install (would run both build and configure implicitly).
> Making all finalized options available at the build stage would then be
> easier.

Is that scraping, or just preparing finalized options using "configure" ?
Meaning other command would just have to get them when they run, if present ?

How would that work ? configure would create a file ?

It seems that you are pushing all the work in the "configure" option,
wich is fine to me,
but it also looks like you can already achieve this with the existing
system, by
changing the subcommands that are in the install command and their
order. That is:

install
     configure
     build
     all the install_*

But if we want to see this working with "build" alone, configure has
to be a subcommand of build:

install
     build
          configure
     all the install_*

IOW this would just require:

1/ adding a "configure" command
2/ inserting it as the first sub command of "build"
3/ make it possible, to share in the whole chain of commands, the
passed arguments

Tarek

From wentland at cl.uni-heidelberg.de  Fri Nov 13 11:38:20 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Fri, 13 Nov 2009 11:38:20 +0100
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
Message-ID: <20091113103820.GB3246@kinakuta.local>

On Fri, Nov 13, 2009 at 18:22 +0900, David Cournapeau wrote:
> On Thu, Nov 12, 2009 at 12:24 AM, Wolodja Wentland
> <wentland at cl.uni-heidelberg.de> wrote:

> > The FHS differentiates between various classes of files and defines
> > proper location for them. We could define platform dependent
> > standard infixes for the following data file classes for distribution
> > foo:
> >
> > * configuration ? ? etc
> > * shared data ? ? ? usr/share/foo/
> > * readme ? ? ? ? ? ?usr/share/foo/README ? ? ? ?README, TODO, ... could
> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?be automatically
> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?discovered
> > * examples ? ? ? ? ?usr/share/foo/examples
> > * documentation ? ? usr/share/foo/doc
> > * man files ? ? ? ? usr/share/man
> > * variable ? ? ? ? ?var/lib/foo
> > * ...

> One could start from all the autoconf-defined variables such as
> prefix, eprefix, sbindir, bindir, etc... One would need in addition
> some variables for python files (.py, pyc and pyo). Then, each scheme
> would just be different mapping for those variables. For example,
> --user option just means changing prefix to $HOME/.local on linux,
> gobolinux (which as a similar file organization as mac os x) could
> easily package as they do for autoconf packages.
+1


> For data files, the major problems currently in distutils are:
>  -  that you cannot 'tag' the different kind of data files. Tarek
> suggestion of using ('$variable_name', [list of files]) is one way of
> solving this. One could also imagine having an argument to setup per
> type (man_files=(list of files), etc...).
        ^^^^^^^^^
I thought of this as well yesterday and actually like it a lot better.

>   - One should think about defining its own data 'type' as well. For
> example, let's say application MegaBar can be extended through python
> plugins. You may want to add your plugin to MegaBar plugin directory:
> ($megabar_dir, [list of files for megabar plugins]). megabar_dir value
> should be customizable from the build process.

Could you elaborate on this idea please? Do you mean that megabars
plugins are shipped in a different distribution than megabar itself? 

I would like this, because it enables developers to split distributions
into different small distributions that might have different release
cycles, only enhance the program/library in question, ...

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091113/c1ba4993/attachment.pgp>

From floris.bruynooghe at gmail.com  Fri Nov 13 13:18:32 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 13 Nov 2009 12:18:32 +0000
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911130226u7eedaa4dg21f7c1d3598449b4@mail.gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911120154u308bf156p301369473250fc5a@mail.gmail.com>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
	<94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
	<4AFCED0F.40703@ar.media.kyoto-u.ac.jp>
	<94bdd2610911130226u7eedaa4dg21f7c1d3598449b4@mail.gmail.com>
Message-ID: <20091113121832.GB32483@laurie.devork>

On Fri, Nov 13, 2009 at 11:26:23AM +0100, Tarek Ziad? wrote:
> On Fri, Nov 13, 2009 at 6:22 AM, David Cournapeau
> <david at ar.media.kyoto-u.ac.jp> wrote:
> > Tarek Ziad? wrote:
> >> A deprecation warning would be added in install, if it finds a local
> >> option, rather
> >> than a global. Meaning both would work in 2.7/3.2.
> >>
> >
> > If changing the command line in incompatible ways is acceptable, what do
> > you think of scrapping the commands (at the UI level only) altogether ?
> > This would be more consistent and easier to deal for the user, and
> > easier to implement as well:
> >
> > python setup.py configure --option1 --option2=value2 ....
> > python setup.py build
> > python setup.py install
> >
> > We could then make this work:
> >
> > python setup.py install (would run both build and configure implicitly).
> > Making all finalized options available at the build stage would then be
> > easier.
> 
> Is that scraping, or just preparing finalized options using "configure" ?
> Meaning other command would just have to get them when they run, if present ?
> 
> How would that work ? configure would create a file ?

That would be the obvious solution.  Both autoconf and CPAN do this by
my understanding so it's a pretty logical thing to do.

> It seems that you are pushing all the work in the "configure" option,
> wich is fine to me,
> but it also looks like you can already achieve this with the existing
> system, by
> changing the subcommands that are in the install command and their
> order. That is:
> 
> install
>      configure
>      build
>      all the install_*
> 
> But if we want to see this working with "build" alone, configure has
> to be a subcommand of build:
> 
> install
>      build
>           configure
>      all the install_*

Would it not be harder to add new command (or "build tasks") that
require information from the configure step in you structure it like
this?

Regards
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From cournape at gmail.com  Fri Nov 13 13:36:41 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Nov 2009 21:36:41 +0900
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091113103820.GB3246@kinakuta.local>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113103820.GB3246@kinakuta.local>
Message-ID: <5b8d13220911130436k691fa3a6k83d0f2dda160b0cc@mail.gmail.com>

On Fri, Nov 13, 2009 at 7:38 PM, Wolodja Wentland
<wentland at cl.uni-heidelberg.de> wrote:

>
> Could you elaborate on this idea please? Do you mean that megabars
> plugins are shipped in a different distribution than megabar itself?

Yes. Imagine that I create a library, which includes a firefox plugin
in python. I need to differentiate between what will end up in firefox
and the rest.

Automake has this ability, for example. You say something like:

megabardir = $(exec_prefix)/share/megabar/site
magabar_DATA = megabar_plugin.py

And automake has this convention that *dir is a dir and *_DATA is the
data which go into this directory. Given that we have a real language
here, there is no need for those conventions.

David

From ziade.tarek at gmail.com  Fri Nov 13 13:46:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 13 Nov 2009 13:46:37 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <20091113121832.GB32483@laurie.devork>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<hdgrn6$4l9$1@ger.gmane.org>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
	<94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
	<4AFCED0F.40703@ar.media.kyoto-u.ac.jp>
	<94bdd2610911130226u7eedaa4dg21f7c1d3598449b4@mail.gmail.com>
	<20091113121832.GB32483@laurie.devork>
Message-ID: <94bdd2610911130446k6773e52du1e6edec6417de103@mail.gmail.com>

On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
[..]
>> Is that scraping, or just preparing finalized options using "configure" ?
>> Meaning other command would just have to get them when they run, if present ?
>>
>> How would that work ? configure would create a file ?
>
> That would be the obvious solution. ?Both autoconf and CPAN do this by
> my understanding so it's a pretty logical thing to do.

I find it logical too now.

[..]
> Would it not be harder to add new command (or "build tasks") that
> require information from the configure step in you structure it like
> this?
>

I was thinking about an API that would allow any command to read/write
the configuration data,
and using it from the "configure" command to write it, and from the
others to read it.

That would allow including this new behaviour in existing commands
with a deprecation step
(in today's distutils, a build is triggered when you call install in
any case, when there are some extensions)

Tarek

From cournape at gmail.com  Fri Nov 13 13:52:10 2009
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 13 Nov 2009 21:52:10 +0900
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911130446k6773e52du1e6edec6417de103@mail.gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<94bdd2610911121412k59f125f2qcf77287029500618@mail.gmail.com>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
	<94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
	<4AFCED0F.40703@ar.media.kyoto-u.ac.jp>
	<94bdd2610911130226u7eedaa4dg21f7c1d3598449b4@mail.gmail.com>
	<20091113121832.GB32483@laurie.devork>
	<94bdd2610911130446k6773e52du1e6edec6417de103@mail.gmail.com>
Message-ID: <5b8d13220911130452i549e22a0obf51ec3d84735416@mail.gmail.com>

On Fri, Nov 13, 2009 at 9:46 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe
> <floris.bruynooghe at gmail.com> wrote:
> [..]
>>> Is that scraping, or just preparing finalized options using "configure" ?
>>> Meaning other command would just have to get them when they run, if present ?
>>>
>>> How would that work ? configure would create a file ?
>>
>> That would be the obvious solution. ?Both autoconf and CPAN do this by
>> my understanding so it's a pretty logical thing to do.
>
> I find it logical too now.

Scons and waf do it through a db kind of file, but that's because they
are much more fancy (scons for example keeps so called signature of
every command and nodes to know what has already been built or not).

A first step as a plain file, which should clearly be marked as
implementation-defined and only guaranteed to work through an API is
the way to go I think.

David

From floris.bruynooghe at gmail.com  Fri Nov 13 14:04:47 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Fri, 13 Nov 2009 13:04:47 +0000
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
Message-ID: <20091113130447.GC32483@laurie.devork>

On Fri, Nov 13, 2009 at 06:22:06PM +0900, David Cournapeau wrote:
> On Thu, Nov 12, 2009 at 12:24 AM, Wolodja Wentland
> <wentland at cl.uni-heidelberg.de> wrote:
> 
> >
> > The FHS differentiates between various classes of files and defines
> > proper location for them. We could define platform dependent
> > standard infixes for the following data file classes for distribution
> > foo:
> >
> > * configuration ? ? etc
> > * shared data ? ? ? usr/share/foo/
> > * readme ? ? ? ? ? ?usr/share/foo/README ? ? ? ?README, TODO, ... could
> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?be automatically
> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?discovered
> > * examples ? ? ? ? ?usr/share/foo/examples
> > * documentation ? ? usr/share/foo/doc
> > * man files ? ? ? ? usr/share/man
> > * variable ? ? ? ? ?var/lib/foo
> > * ...
> >
> 
> One could start from all the autoconf-defined variables such as
> prefix, eprefix, sbindir, bindir, etc... One would need in addition
> some variables for python files (.py, pyc and pyo). Then, each scheme
> would just be different mapping for those variables. For example,
> --user option just means changing prefix to $HOME/.local on linux,
> gobolinux (which as a similar file organization as mac os x) could
> easily package as they do for autoconf packages.

+1

But don't expect all package maintainers to suddenly classify their
datafiles properly.  Last time this got discussed there was
significant opposition to this, maintainers did not see the point and
didn't want *any* change to their code at all unless it made *their*
lives easier.

Personally I think the API you proposed is reasonable and would love
to see something like that in distutils so at least distros could
patch upstream (and hopefully get it accepted).

The only issue I can see it that I'd like the API to retrieve the
data-files to work both when running from a development repository as
from an installed distribution.  Making the API methods of the
Distribution class would force you to have an installed Distribution
on you PYTHONPATH I think (as would relying on a file in the egg-info
directory).


Regards
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ziade.tarek at gmail.com  Fri Nov 13 14:44:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 13 Nov 2009 14:44:37 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <5b8d13220911130452i549e22a0obf51ec3d84735416@mail.gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<5b8d13220911121445t7da82de2oc46a7b418f6fe344@mail.gmail.com>
	<94bdd2610911121547w17d2f58cie3fb62b7f2256641@mail.gmail.com>
	<5b8d13220911121608s412f52cahb55b283993000c65@mail.gmail.com>
	<94bdd2610911121616v12f6effao7784111af0fd27c1@mail.gmail.com>
	<4AFCED0F.40703@ar.media.kyoto-u.ac.jp>
	<94bdd2610911130226u7eedaa4dg21f7c1d3598449b4@mail.gmail.com>
	<20091113121832.GB32483@laurie.devork>
	<94bdd2610911130446k6773e52du1e6edec6417de103@mail.gmail.com>
	<5b8d13220911130452i549e22a0obf51ec3d84735416@mail.gmail.com>
Message-ID: <94bdd2610911130544r49aab04fne844a7cb468c90c3@mail.gmail.com>

On Fri, Nov 13, 2009 at 1:52 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Nov 13, 2009 at 9:46 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> On Fri, Nov 13, 2009 at 1:18 PM, Floris Bruynooghe
>> <floris.bruynooghe at gmail.com> wrote:
>> [..]
>>>> Is that scraping, or just preparing finalized options using "configure" ?
>>>> Meaning other command would just have to get them when they run, if present ?
>>>>
>>>> How would that work ? configure would create a file ?
>>>
>>> That would be the obvious solution. ?Both autoconf and CPAN do this by
>>> my understanding so it's a pretty logical thing to do.
>>
>> I find it logical too now.
>
> Scons and waf do it through a db kind of file, but that's because they
> are much more fancy (scons for example keeps so called signature of
> every command and nodes to know what has already been built or not).
>
> A first step as a plain file, which should clearly be marked as
> implementation-defined and only guaranteed to work through an API is
> the way to go I think.
>

Here's my proposal for this to happen, if you (and others) want to contribute:

Let's build this new "configure" command in Distribute 0.7, together
with the APIs to  read/write the data.

Then let's change the other commands consequently (without thinking
about backward compat first)
so we can try out this new behaviour.

Once it's proven to work good, we could publish Distribute 0.7 with
it, and depending on the community
feedback, we could integrate it to Distutils and work on the backward
compat part.

this two phase step wouldn't be a problem imho, for early adopters
that would use and test it.

Tarek

-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From wentland at cl.uni-heidelberg.de  Fri Nov 13 14:48:24 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Fri, 13 Nov 2009 14:48:24 +0100
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <5b8d13220911130436k691fa3a6k83d0f2dda160b0cc@mail.gmail.com>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113103820.GB3246@kinakuta.local>
	<5b8d13220911130436k691fa3a6k83d0f2dda160b0cc@mail.gmail.com>
Message-ID: <20091113134824.GD3246@kinakuta.local>

On Fri, Nov 13, 2009 at 21:36 +0900, David Cournapeau wrote:
> On Fri, Nov 13, 2009 at 7:38 PM, Wolodja Wentland
> <wentland at cl.uni-heidelberg.de> wrote:

> > Could you elaborate on this idea please? Do you mean that megabars
> > plugins are shipped in a different distribution than megabar itself?
> 
> Yes. Imagine that I create a library, which includes a firefox plugin
> in python. I need to differentiate between what will end up in firefox
> and the rest.

I can certainly see the need for this and would be +1 on this idea.

One way to do this could be:

In megabar:

setup(
    ...
    custom_prefixes=[
        ('$plugins', '$purelib/plugins'),
        ('$foo', '$base_prefix/foo'),
        ...
        ],
    ...
    data_files=[
        ('$plugins', 'plugins/standard_plugin.py'),
        ...
        ]
    )

And in megabar-plugins:

setup(
    ...
    data_files=[
        (pkgutil.get_prefix('megabar', '$plugin'),
        'plugins/additional_plugin.py'),
        ...
        ]
    )

You could use a standard prefix if megabar does not define a $plugin
custom prefix within megabar-plugins, which might even be the default if
introduction of custom_prefixes is not accepted/ok.

I think we need to:

* compile a list of standard prefixes and their default values on all
  platforms. I still like the idea to be able to modify these *only* by
  setting environment variables, so it would look something like this:

  stdprefix1 = os.environ.get('PYDIST_STDPREFIX1', 
                              distutils.sysconfig.get_std_prefix('stdprefix1'))

  Create this list on a wiki? 

* Define an API that clarifies how to query this information:
    
    - within setup.py
    - at built time
    - at runtime
    - ... ??

Suggestions? Corrections? Better ideas? I am just brainstorming...

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091113/e3c7eb45/attachment.pgp>

From wentland at cl.uni-heidelberg.de  Fri Nov 13 14:51:56 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Fri, 13 Nov 2009 14:51:56 +0100
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091113130447.GC32483@laurie.devork>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113130447.GC32483@laurie.devork>
Message-ID: <20091113135156.GE3246@kinakuta.local>

On Fri, Nov 13, 2009 at 13:04 +0000, Floris Bruynooghe wrote:

> +1
:-)

> But don't expect all package maintainers to suddenly classify their
> datafiles properly.  Last time this got discussed there was
> significant opposition to this, maintainers did not see the point and
> didn't want *any* change to their code at all unless it made *their*
> lives easier.

True, but we will see about that.

> Personally I think the API you proposed is reasonable and would love
> to see something like that in distutils so at least distros could
> patch upstream (and hopefully get it accepted).

Simplifying the work of distro maintainers easier is one of the main
incentives I had in mind with this proposal.

> The only issue I can see it that I'd like the API to retrieve the
> data-files to work both when running from a development repository as
> from an installed distribution.  Making the API methods of the
> Distribution class would force you to have an installed Distribution
> on you PYTHONPATH I think (as would relying on a file in the egg-info
> directory).

Any ideas on how to implement this?

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091113/3fed714e/attachment.pgp>

From wentland at cl.uni-heidelberg.de  Fri Nov 13 15:01:08 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Fri, 13 Nov 2009 15:01:08 +0100
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <5b8d13220911130436k691fa3a6k83d0f2dda160b0cc@mail.gmail.com>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113103820.GB3246@kinakuta.local>
	<5b8d13220911130436k691fa3a6k83d0f2dda160b0cc@mail.gmail.com>
Message-ID: <20091113140108.GF3246@kinakuta.local>

On Fri, Nov 13, 2009 at 21:36 +0900, David Cournapeau wrote:

> Yes. Imagine that I create a library, which includes a firefox plugin
> in python. I need to differentiate between what will end up in firefox
> and the rest.

One thing I find noteworthy/horrifying about your *Firefox* plugin
example is that this would mean that pip would have to install files in
a directory completely unrelated to Python and one that is under the
control of the package manager of the system.

I guess a lot of distribution maintainers would curse the distutils
developers if this becomes common practice for a lot of python packages.

But this is more related to the fact that Python provides a second
package manager that is *distinct* from the system one. It might be a
good idea to display a big WARNING whenever somebody tries to install
distributions (-> python, who came up with this terminology) within a
path that is usually handled by the systems package manager.

This would mean that the *default* install should be PEP 370 compliant
or inside a virtualenv and that installing into system paths should be
enabled explicitly.

BTW - How are uninstalls of distributions (python) handled on which
other distributions depend. Will all rdepends also be uninstalled?

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091113/1657e7b0/attachment.pgp>

From jeremy.kloth at gmail.com  Fri Nov 13 15:46:19 2009
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Fri, 13 Nov 2009 07:46:19 -0700
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911130544r49aab04fne844a7cb468c90c3@mail.gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<5b8d13220911130452i549e22a0obf51ec3d84735416@mail.gmail.com>
	<94bdd2610911130544r49aab04fne844a7cb468c90c3@mail.gmail.com>
Message-ID: <200911130746.19487.jeremy.kloth@gmail.com>

On Friday 13 November 2009 06:44:37 am Tarek Ziad? wrote:
> Here's my proposal for this to happen, if you (and others) want to
>  contribute:
> 
> Let's build this new "configure" command in Distribute 0.7, together
> with the APIs to  read/write the data.
> 
> Then let's change the other commands consequently (without thinking
> about backward compat first)
> so we can try out this new behaviour.

Fourthought's 4Suite distutils extensions already provide this behavior. You 
may consider looking at how the 'config' command is handled there as a guide 
in implementing this. Or just copy it wholesale, for that matter, as it is my 
doing.

> Once it's proven to work good, we could publish Distribute 0.7 with
> it, and depending on the community
> feedback, we could integrate it to Distutils and work on the backward
> compat part.
> 
> this two phase step wouldn't be a problem imho, for early adopters
> that would use and test it.

The exact thing being described has been done in 4Suite for 6 years (along 
with many other distutils improvements). Feel free to take or discuss or 
request help with any of the features/additions (like FHS layout of files) in 
the 4Suite distutils extensions. The code is available for browsing:
http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/DistExt/

The way we've impl'd 'config' was as a prerequisite for 'build', just as 
'build' is for 'install'.  If any of the options stored by the 'config' 
command are overridden via 'build' or 'install' options, the 'config' command 
would be re-run to store the "new" choices.

Any questions or just simple help with integrating a similar system, just let 
me know. Been there, done that.

Jeremy

-- 
Jeremy Kloth
http://4suite.org/

From Jim.Vickroy at noaa.gov  Fri Nov 13 16:36:43 2009
From: Jim.Vickroy at noaa.gov (Jim Vickroy)
Date: Fri, 13 Nov 2009 08:36:43 -0700
Subject: [Distutils] distutils.core.setup --install-script option
 in	Python 2.6 ?
In-Reply-To: <94bdd2610911121428h638d926bt771aafb3c146d622@mail.gmail.com>
References: <4AFBF21C.70803@noaa.gov>
	<94bdd2610911121428h638d926bt771aafb3c146d622@mail.gmail.com>
Message-ID: <4AFD7D0B.5090701@noaa.gov>

Tarek Ziad? wrote:
> On Thu, Nov 12, 2009 at 12:31 PM, Jim Vickroy <Jim.Vickroy at noaa.gov> wrote:
>   
>> Hello,
>>
>> Please excuse the repost (sent last week), but I'm hoping someone on the
>> list can get me on-track with a distutils problem.
>>
>> I have just upgraded from Python 2.5 to 2.6.4 (MS Windows XP service pack
>> 3).
>>
>> My setup.py scripts (based on distutils.core.setup) no longer work with the
>> --install-script option.  Here is the error:
>>
>>  * error: option --install-script not recognized
>>     
>
> Hi Jim,
>
> the option (in 2.5 and 2.6) is "--install-scripts" (notice the final 's')
>
> Regards
> Tarek
>   
Hello Tarek,

*--*install-script_*s*_ is not recognized (see below) -- at least with 
Python 2.6.4 on MS Windows.

C:\Documents and Settings\jim.vickroy\My Documents\Projects>"D-RAP 
INSTALLER BUILDER.py" bdist_wininst 
--install-scripts=GOES-12-SXI-processor-setup-t
eardown.py

usage: D-RAP INSTALLER BUILDER.py [global_opts] cmd1 [cmd1_opts] [cmd2 
[cmd2_opts] ...]
   or: D-RAP INSTALLER BUILDER.py --help [cmd1 cmd2 ...]
   or: D-RAP INSTALLER BUILDER.py --help-commands
   or: D-RAP INSTALLER BUILDER.py cmd --help

error: option --install-scripts not recognized

C:\Documents and Settings\jim.vickroy\My Documents\Projects>

-- jv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091113/bbc89036/attachment-0001.htm>

From ziade.tarek at gmail.com  Fri Nov 13 16:42:43 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 13 Nov 2009 16:42:43 +0100
Subject: [Distutils] distutils.core.setup --install-script option in
	Python 2.6 ?
In-Reply-To: <4AFD7D0B.5090701@noaa.gov>
References: <4AFBF21C.70803@noaa.gov>
	<94bdd2610911121428h638d926bt771aafb3c146d622@mail.gmail.com>
	<4AFD7D0B.5090701@noaa.gov>
Message-ID: <94bdd2610911130742p2fd75b5ie2ff11e9d252bc71@mail.gmail.com>

On Fri, Nov 13, 2009 at 4:36 PM, Jim Vickroy <Jim.Vickroy at noaa.gov> wrote:
[..]
> Hello Tarek,
>
> --install-scripts is not recognized (see below) -- at least with Python
> 2.6.4 on MS Windows.

In Python 2.5 *and* 2.6:

--install-scripts is an option for "install"
--install-script is an option for "bdist_wininst"

While this name coincidence is a bit annoying, bdist_wininst option is
something else:

  --install-script           basename of installation script to be run
afterinstallation or before deinstallation

And in install:

  --install-scripts                    installation directory for Python scripts

Tarek

From glyph at twistedmatrix.com  Fri Nov 13 17:49:43 2009
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Fri, 13 Nov 2009 11:49:43 -0500
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091113140108.GF3246@kinakuta.local>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113103820.GB3246@kinakuta.local>
	<5b8d13220911130436k691fa3a6k83d0f2dda160b0cc@mail.gmail.com>
	<20091113140108.GF3246@kinakuta.local>
Message-ID: <BA9A25A5-0D69-44D3-AEF1-516F5AD44490@twistedmatrix.com>


On Nov 13, 2009, at 9:01 AM, Wolodja Wentland wrote:

> This would mean that the *default* install should be PEP 370 compliant
> or inside a virtualenv and that installing into system paths should be
> enabled explicitly.

+1.



From tseaver at palladion.com  Fri Nov 13 21:25:38 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Fri, 13 Nov 2009 15:25:38 -0500
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091113130447.GC32483@laurie.devork>
References: <20091111152414.GC7023@kinakuta.local>	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113130447.GC32483@laurie.devork>
Message-ID: <hdkfc2$86n$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Floris Bruynooghe wrote:
> On Fri, Nov 13, 2009 at 06:22:06PM +0900, David Cournapeau wrote:
>> On Thu, Nov 12, 2009 at 12:24 AM, Wolodja Wentland
>> <wentland at cl.uni-heidelberg.de> wrote:
>>
>>> The FHS differentiates between various classes of files and defines
>>> proper location for them. We could define platform dependent
>>> standard infixes for the following data file classes for distribution
>>> foo:
>>>
>>> * configuration     etc
>>> * shared data       usr/share/foo/
>>> * readme            usr/share/foo/README        README, TODO, ... could
>>>                                                be automatically
>>>                                                discovered
>>> * examples          usr/share/foo/examples
>>> * documentation     usr/share/foo/doc
>>> * man files         usr/share/man
>>> * variable          var/lib/foo
>>> * ...
>>>
>> One could start from all the autoconf-defined variables such as
>> prefix, eprefix, sbindir, bindir, etc... One would need in addition
>> some variables for python files (.py, pyc and pyo). Then, each scheme
>> would just be different mapping for those variables. For example,
>> --user option just means changing prefix to $HOME/.local on linux,
>> gobolinux (which as a similar file organization as mac os x) could
>> easily package as they do for autoconf packages.
> 
> +1
> 
> But don't expect all package maintainers to suddenly classify their
> datafiles properly.  Last time this got discussed there was
> significant opposition to this, maintainers did not see the point and
> didn't want *any* change to their code at all unless it made *their*
> lives easier.

I think the atual sentiment was that package authors / maintainers
weren't willingly going to add those "librarian" categories themselves,
but would mostly be happy to take patches which added them, likely from
OS distribution packagers.

> Personally I think the API you proposed is reasonable and would love
> to see something like that in distutils so at least distros could
> patch upstream (and hopefully get it accepted).

Excactly.

> The only issue I can see it that I'd like the API to retrieve the
> data-files to work both when running from a development repository as
> from an installed distribution.  Making the API methods of the
> Distribution class would force you to have an installed Distribution
> on you PYTHONPATH I think (as would relying on a file in the egg-info
> directory).

Static metadata, anyone?



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkr9wL0ACgkQ+gerLs4ltQ5rjACfcvmCOjkP5leBxbmnhjzq9wwE
weEAoL7nmzwBtt/TtadM6ZguZbJP8jXW
=b7es
-----END PGP SIGNATURE-----


From cool-rr at cool-rr.com  Fri Nov 13 22:10:27 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Fri, 13 Nov 2009 21:10:27 +0000 (UTC)
Subject: [Distutils] Deleting build directory
Message-ID: <loom.20091113T220808-850@post.gmane.org>

What's the best way to delete the "build" directory both before and after  
installation, using Distribute? I want to do it so the user will only need to do 
"setup.py install", and the build directory will be pre- and post-deleted.

Thanks,
Ram.


From cool-rr at cool-rr.com  Fri Nov 13 21:55:02 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Fri, 13 Nov 2009 22:55:02 +0200
Subject: [Distutils] =?windows-1252?q?=60pkg=5Fresources=2Erequire=28=29?=
	=?windows-1252?q?=60_doesn=92t_see_wxPython?=
Message-ID: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com>

I'm having trouble using Distribute's pkg_resources.require(). I posted a
question on StackOverflow:

http://stackoverflow.com/questions/1728676/pkgresources-require-doesnt-see-wxpython

I hope someone here has an answer.

Thanks, Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091113/9ff76190/attachment.htm>

From pav at iki.fi  Wed Nov 11 23:49:43 2009
From: pav at iki.fi (Pauli Virtanen)
Date: Thu, 12 Nov 2009 00:49:43 +0200
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
Message-ID: <1257979783.4524.105.camel@idol>

ke, 2009-11-11 kello 21:42 +0100, Tarek Ziad? kirjoitti:
[clip]
> Do you mean, an Extension that would require several compilers ?
> 
> I was thinking of a one-to-one relation between an Extension and a
> compiler type, even if there are are multiple source files (in different languages)
> for this extension.
> 
> Meaning that *one* compiler would have to handle of those files. Does that fit
> the cython use case?

It might. The distutils Cython compiler would however need to be smart
enough to handle all languages used in the other source files by itself,
or somehow delegate this back to distutils (which still sounds doable).

Also, the f2py compiler does similar things. Ditto for Pyrex. How much
code duplication would be needed between these?

Source file preprocessing / generation is one thing that also comes
useful. For example, Numpy has its own templated C code generator file
format, and also generates some of the source files automatically from
scratch during the build. These are compiled and linked as a part of
ordinary C extension module.

Currently, as I understand it, numpy.distutils does source generation in
a separate build_src command.

> > Also, the option of determining the necessary compiler and compiler
> > options from file extensions does not always work. For example, Fortran
> > 90 files come in two flavors, fixed-form and free-form, which often are
> > not differentiated by the file extension. However, they may require
> > different compiler flags to compile. (Hopefully, however, nobody uses
> > the fixed-form for F90 any more...)
> 
> How do they do then? Is file extensions, inline options in setup.py,
> and the environ, are enough?

numpy.distutils's Fortran compiler class reads the file and tries to
analyze whether it's fixed-form or not. And chooses appropriate compiler
flags. This is sort of a special "worst case" scenario for compiler
selection, of course.

    ***

Just to throw some wild, perhaps obvious, and definitely unasked-for
ideas in the air (especially as I can't promise I can give any sustained
help here :/ ):

I suppose one option would be to factor *everything* related to
extension module building into build_ext (and abolish build_clib): the
rest of distutils would just say to build_ext

        """
        Here's some info about the environment:

        dict(python_lib_path=/usr/lib/..., optimize=yes/no,
             python_lib_name=..., python_includes=/usr/include/...,
             install_prefix=/usr/lib/python2.6, ...
             ..., python_extension=..., build_temp_dir=...)

        Look the rest up from sysconfig.

        Please build any extensions and data files you like,
        and tell the file and directory names where you placed them
        and where (relative paths) they should go.
        """

Information needed how to build each component would be passed to
build_ext subcomponent directly from setup.py or from a config file.
Now, perhaps it is already like this --- I don't really know the
internals of distutils --- and the build subcomponent is insulated from
the others. In any case, something like this could make refactoring the
build system easier.

I think this idea quickly boils down more or less to David's idea about
a pluggable build system -- implementing a good one takes a lot of work,
so it might make sense to refactor distutils so that it would be
possible [1] to use some of the existing ones (scons, waf, whatever,
heck, even autoconf+make). The *default* build system could be a simple
one, and backwards compatible. Especially so, since it seems to me that
building extension modules is orthogonal to what distutils and
setuptools want to do -- package and install Python modules in a
Python-specific way, not really worry about how to properly call
different compilers on obscure platforms.

Anyway, even in the case pluggability is a bad idea, refactoring the
build system out from the rest of distutils might make sense.

.. [1] possible and easy -- I understand numpyscons is a stab
       at the possible, but it sounds like it was not easy to do.

-- 
Pauli Virtanen




From david.lyon at preisshare.net  Fri Nov 13 23:39:32 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Fri, 13 Nov 2009 17:39:32 -0500
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <hdkfc2$86n$1@ger.gmane.org>
References: <20091111152414.GC7023@kinakuta.local>	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113130447.GC32483@laurie.devork> <hdkfc2$86n$1@ger.gmane.org>
Message-ID: <1275be45d6c4519eeb5affe08103b7a2@preisshare.net>

On Fri, 13 Nov 2009 15:25:38 -0500, Tres Seaver <tseaver at palladion.com>
wrote:
>>>> * configuration     etc
>>>> * shared data       usr/share/foo/
>>>> * readme            usr/share/foo/README        README, TODO, ...
could
>>>>                                                be automatically
>>>>                                                discovered
>>>> * examples          usr/share/foo/examples
>>>> * documentation     usr/share/foo/doc
>>>> * man files         usr/share/man
>>>> * variable          var/lib/foo
>>>> * ...
>>>>
> Static metadata, anyone?

That's like trying to get a prius into a bikeshed and
work on it.

There simply doesn't seem to be the management mechanisms 
available to handle outside-the-square topics like that in 
distutils presently.

Let's just wait for the current work to be done with 
distutils and after that see what transpires.

David










 



From ben+python at benfinney.id.au  Fri Nov 13 23:59:47 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 14 Nov 2009 09:59:47 +1100
Subject: [Distutils] Deleting build directory
References: <loom.20091113T220808-850@post.gmane.org>
Message-ID: <877htut4ik.fsf@benfinney.id.au>

Ram Rachum <cool-rr at cool-rr.com> writes:

> What's the best way to delete the "build" directory both before and
> after installation, using Distribute? I want to do it so the user will
> only need to do "setup.py install", and the build directory will be
> pre- and post-deleted.

Does the ?clean? command do what you want? I would have thought it
should remove any files generated by ?build?.


-- 
 \     ?We are no more free to believe whatever we want about God than |
  `\         we are free to adopt unjustified beliefs about science or |
_o__)              history [?].? ?Sam Harris, _The End of Faith_, 2004 |
Ben Finney


From cool-rr at cool-rr.com  Sat Nov 14 00:28:47 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Fri, 13 Nov 2009 23:28:47 +0000 (UTC)
Subject: [Distutils] Deleting build directory
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
Message-ID: <loom.20091114T002739-269@post.gmane.org>

Ben Finney <ben+python <at> benfinney.id.au> writes:

> 
> Ram Rachum <cool-rr <at> cool-rr.com> writes:
> 
> > What's the best way to delete the "build" directory both before and
> > after installation, using Distribute? I want to do it so the user will
> > only need to do "setup.py install", and the build directory will be
> > pre- and post-deleted.
> 
> Does the ?clean? command do what you want? I would have thought it
> should remove any files generated by ?build?.
> 

Would I have to type it when calling setup.py?
I want something that won't require that, and
would work when the user does the normal "setup.py
install".

Ram.


From cournape at gmail.com  Sat Nov 14 00:30:41 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 14 Nov 2009 08:30:41 +0900
Subject: [Distutils] Install time prefixes and data files
In-Reply-To: <20091113130447.GC32483@laurie.devork>
References: <20091111152414.GC7023@kinakuta.local>
	<5b8d13220911130122u40938dceu8508e50d5a090e67@mail.gmail.com>
	<20091113130447.GC32483@laurie.devork>
Message-ID: <5b8d13220911131530r6cbff022q14bb32741ff86732@mail.gmail.com>

On Fri, Nov 13, 2009 at 10:04 PM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
> On Fri, Nov 13, 2009 at 06:22:06PM +0900, David Cournapeau wrote:
>> On Thu, Nov 12, 2009 at 12:24 AM, Wolodja Wentland
>> <wentland at cl.uni-heidelberg.de> wrote:
>>
>> >
>> > The FHS differentiates between various classes of files and defines
>> > proper location for them. We could define platform dependent
>> > standard infixes for the following data file classes for distribution
>> > foo:
>> >
>> > * configuration ? ? etc
>> > * shared data ? ? ? usr/share/foo/
>> > * readme ? ? ? ? ? ?usr/share/foo/README ? ? ? ?README, TODO, ... could
>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?be automatically
>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?discovered
>> > * examples ? ? ? ? ?usr/share/foo/examples
>> > * documentation ? ? usr/share/foo/doc
>> > * man files ? ? ? ? usr/share/man
>> > * variable ? ? ? ? ?var/lib/foo
>> > * ...
>> >
>>
>> One could start from all the autoconf-defined variables such as
>> prefix, eprefix, sbindir, bindir, etc... One would need in addition
>> some variables for python files (.py, pyc and pyo). Then, each scheme
>> would just be different mapping for those variables. For example,
>> --user option just means changing prefix to $HOME/.local on linux,
>> gobolinux (which as a similar file organization as mac os x) could
>> easily package as they do for autoconf packages.
>
> +1
>
> But don't expect all package maintainers to suddenly classify their
> datafiles properly. ?Last time this got discussed there was
> significant opposition to this, maintainers did not see the point and
> didn't want *any* change to their code at all unless it made *their*
> lives easier.

Oh, I don't expect much on that front. But if it becomes at least
possible, pressuring people to do so is more doable. And you could in
theory enforce some kind of policy on Pypi, but it seems there is a
strong opposition to do so - but again, if you are willing to have
your "own" Pypi for a subcommunity, it is at least possible to do so.

> The only issue I can see it that I'd like the API to retrieve the
> data-files to work both when running from a development repository as
> from an installed distribution. ?Making the API methods of the
> Distribution class would force you to have an installed Distribution
> on you PYTHONPATH I think (as would relying on a file in the egg-info
> directory).

As long as python itself does not support this, I don't see a way to
do data retrieving reliably. It may be useful to see how other
languages do it (haskell and ruby, and I am sure others, have the same
issues).

David

From cournape at gmail.com  Sat Nov 14 00:39:37 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 14 Nov 2009 08:39:37 +0900
Subject: [Distutils] Deleting build directory
In-Reply-To: <loom.20091114T002739-269@post.gmane.org>
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
	<loom.20091114T002739-269@post.gmane.org>
Message-ID: <5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>

On Sat, Nov 14, 2009 at 8:28 AM, Ram Rachum <cool-rr at cool-rr.com> wrote:

> Would I have to type it when calling setup.py?
> I want something that won't require that, and
> would work when the user does the normal "setup.py
> install".

You cannot delete the build directory automatically by default, that
would force you to rebuild everytime you call setup.py. Why do you
want to delete the build directory everytime ?

David

From setuptools at bugs.python.org  Sat Nov 14 00:40:53 2009
From: setuptools at bugs.python.org (Benjamin Riggs)
Date: Fri, 13 Nov 2009 23:40:53 +0000
Subject: [Distutils] [issue96] Invalid setup.py flag in srpm spec file
	(0.6c11)
In-Reply-To: <1258155653.67.0.781173221201.issue96@psf.upfronthosting.co.za>
Message-ID: <1258155653.67.0.781173221201.issue96@psf.upfronthosting.co.za>


New submission from Benjamin Riggs <riggs at umn.edu>:

line 191-2:
%install
python setup.py install --single-version-externally-managed
--root=$RPM_BUILD_ROOT --record-rpm=INSTALLED_OBJECTS

causes:
error: option --record-rpm not recognized

----------
messages: 463
nosy: riggs
priority: bug
status: unread
title: Invalid setup.py flag in srpm spec file (0.6c11)

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue96>
_______________________________________________

From cool-rr at cool-rr.com  Sat Nov 14 00:43:54 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Sat, 14 Nov 2009 01:43:54 +0200
Subject: [Distutils] Deleting build directory
In-Reply-To: <5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au> 
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
Message-ID: <a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>

On Sat, Nov 14, 2009 at 1:39 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Sat, Nov 14, 2009 at 8:28 AM, Ram Rachum <cool-rr at cool-rr.com> wrote:
>
> > Would I have to type it when calling setup.py?
> > I want something that won't require that, and
> > would work when the user does the normal "setup.py
> > install".
>
> You cannot delete the build directory automatically by default, that
> would force you to rebuild everytime you call setup.py. Why do you
> want to delete the build directory everytime ?
>
> David
>

What I really want is never having to worry about the build directory being
around after doing any actions with `setup.py`. Do you have any other
suggestion? Also, how will it hurt me that `setup.py` rebuilds every time?

Ram.

-- 
Sincerely,
Ram Rachum
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091114/40943b2a/attachment-0001.htm>

From cournape at gmail.com  Sat Nov 14 00:53:50 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 14 Nov 2009 08:53:50 +0900
Subject: [Distutils] Deleting build directory
In-Reply-To: <a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>
Message-ID: <5b8d13220911131553u42cafd7pca1a6c81e927c127@mail.gmail.com>

On Sat, Nov 14, 2009 at 8:43 AM, cool-RR <cool-rr at cool-rr.com> wrote:

> What I really want is never having to worry about the build directory being
> around after doing any actions with `setup.py`.

Then create a script to do so.

David

From cool-rr at cool-rr.com  Sat Nov 14 00:56:44 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Sat, 14 Nov 2009 01:56:44 +0200
Subject: [Distutils] Deleting build directory
In-Reply-To: <5b8d13220911131553u42cafd7pca1a6c81e927c127@mail.gmail.com>
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au> 
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com> 
	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com> 
	<5b8d13220911131553u42cafd7pca1a6c81e927c127@mail.gmail.com>
Message-ID: <a422b72d0911131556i21560101v509eb7bf6a076ccb@mail.gmail.com>

On Sat, Nov 14, 2009 at 1:53 AM, David Cournapeau <cournape at gmail.com>wrote:

> On Sat, Nov 14, 2009 at 8:43 AM, cool-RR <cool-rr at cool-rr.com> wrote:
>
> > What I really want is never having to worry about the build directory
> being
> > around after doing any actions with `setup.py`.
>
> Then create a script to do so.
>
> David
>

I used to have one, which relied on distutils' `dir_util` module, but I
understand that doesn't exist in Distribute, so what do I do?

-- 
Sincerely,
Ram Rachum
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091114/d4370567/attachment.htm>

From ben+python at benfinney.id.au  Sat Nov 14 05:06:48 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 14 Nov 2009 15:06:48 +1100
Subject: [Distutils] Deleting build directory
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>
Message-ID: <87y6m9sqav.fsf@benfinney.id.au>

cool-RR <cool-rr at cool-rr.com> writes:

> What I really want is never having to worry about the build directory
> being around after doing any actions with `setup.py`. Do you have any
> other suggestion?

If you have determined that ?setup.py install clean? does what you want,
then you can create a user-specific ?setup.cfg? with this section:

    [aliases]
    install = install clean

-- 
 \           ?Special cocktails for the ladies with nuts.? ?bar, Tokyo |
  `\                                                                   |
_o__)                                                                  |
Ben Finney


From ziade.tarek at gmail.com  Sat Nov 14 10:42:47 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 14 Nov 2009 10:42:47 +0100
Subject: [Distutils] Deleting build directory
In-Reply-To: <87y6m9sqav.fsf@benfinney.id.au>
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>
	<87y6m9sqav.fsf@benfinney.id.au>
Message-ID: <94bdd2610911140142pfd73a11v22a07e3047866079@mail.gmail.com>

On Sat, Nov 14, 2009 at 5:06 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> cool-RR <cool-rr at cool-rr.com> writes:
>
>> What I really want is never having to worry about the build directory
>> being around after doing any actions with `setup.py`. Do you have any
>> other suggestion?
>
> If you have determined that ?setup.py install clean? does what you want,
> then you can create a user-specific ?setup.cfg? with this section:
>
> ? ?[aliases]
> ? ?install = install clean


Semi-related:
I was wondering if a pre/post install hook to the install command
could be useful.

Tarek

From alex.gronholm at nextday.fi  Sat Nov 14 10:45:34 2009
From: alex.gronholm at nextday.fi (=?windows-1252?Q?Alex_Gr=F6nholm?=)
Date: Sat, 14 Nov 2009 11:45:34 +0200
Subject: [Distutils] Deleting build directory
In-Reply-To: <94bdd2610911140142pfd73a11v22a07e3047866079@mail.gmail.com>
References: <loom.20091113T220808-850@post.gmane.org>	<877htut4ik.fsf@benfinney.id.au>	<loom.20091114T002739-269@post.gmane.org>	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>	<87y6m9sqav.fsf@benfinney.id.au>
	<94bdd2610911140142pfd73a11v22a07e3047866079@mail.gmail.com>
Message-ID: <4AFE7C3E.2000005@nextday.fi>

Tarek Ziad? kirjoitti:
> On Sat, Nov 14, 2009 at 5:06 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
>   
>> cool-RR <cool-rr at cool-rr.com> writes:
>>
>>     
>>> What I really want is never having to worry about the build directory
>>> being around after doing any actions with `setup.py`. Do you have any
>>> other suggestion?
>>>       
>> If you have determined that ?setup.py install clean? does what you want,
>> then you can create a user-specific ?setup.cfg? with this section:
>>
>>    [aliases]
>>    install = install clean
>>     
>
>
> Semi-related:
> I was wondering if a pre/post install hook to the install command
> could be useful.
>   
Distribute itself uses something like that, doesn't it? Having 
standardized pre/post-install hooks might be worth considering.
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>   


From ziade.tarek at gmail.com  Sat Nov 14 10:46:49 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 14 Nov 2009 10:46:49 +0100
Subject: [Distutils]
	=?windows-1252?q?=60pkg=5Fresources=2Erequire=28=29?=
	=?windows-1252?q?=60_doesn=92t_see_wxPython?=
In-Reply-To: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com>
References: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com>
Message-ID: <94bdd2610911140146id3bbff3yb28343275f97538b@mail.gmail.com>

On Fri, Nov 13, 2009 at 9:55 PM, cool-RR <cool-rr at cool-rr.com> wrote:
> I'm having trouble using Distribute's pkg_resources.require(). I posted a
> question on StackOverflow:
> http://stackoverflow.com/questions/1728676/pkgresources-require-doesnt-see-wxpython
> I hope someone here has an answer.
> Thanks, Ram.

If pkg_resources doesn't find it, it might be that it doesn't have
somehow an egg-info folder or file in site-packages,
or that the project name is not what you are expecting ('wxpython')

Look for that egg-info and search for the name of the project. In
*egg-info/PKG-INFO or in egg-info itself.

Tarek

From ziade.tarek at gmail.com  Sat Nov 14 10:51:06 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 14 Nov 2009 10:51:06 +0100
Subject: [Distutils] Deleting build directory
In-Reply-To: <4AFE7C3E.2000005@nextday.fi>
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>
	<87y6m9sqav.fsf@benfinney.id.au>
	<94bdd2610911140142pfd73a11v22a07e3047866079@mail.gmail.com>
	<4AFE7C3E.2000005@nextday.fi>
Message-ID: <94bdd2610911140151i5d4219a4s2b6ed8d73b22f2f6@mail.gmail.com>

2009/11/14 Alex Gr?nholm <alex.gronholm at nextday.fi>:
[..]
>> Semi-related:
>> I was wondering if a pre/post install hook to the install command
>> could be useful.
>>
>
> Distribute itself uses something like that, doesn't it? Having standardized
> pre/post-install hooks might be worth considering.

And that could probably be generalized to any command if we define a new global
option, that would point to a callable for pre or post.

setup(
  ...
  post_install_hooks={'install': [hook1, hook2],},
  ...
)

From cool-rr at cool-rr.com  Sat Nov 14 14:46:11 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Sat, 14 Nov 2009 13:46:11 +0000 (UTC)
Subject: [Distutils] =?utf-8?b?YHBrZ19yZXNvdXJjZXMucmVxdWlyZSgpYGRvZXNu?=
	=?utf-8?q?=E2=80=99t_see_wxPython?=
References: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com>
	<94bdd2610911140146id3bbff3yb28343275f97538b@mail.gmail.com>
Message-ID: <loom.20091114T143334-146@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:

> 
> On Fri, Nov 13, 2009 at 9:55 PM, cool-RR <cool-rr <at> cool-rr.com> wrote:
> > I'm having trouble using Distribute's pkg_resources.require(). I posted a
> > question on StackOverflow:
> > http://stackoverflow.com/questions/1728676/pkgresources-require-doesnt-see-
wxpython
> > I hope someone here has an answer.
> > Thanks, Ram.
> 
> If pkg_resources doesn't find it, it might be that it doesn't have
> somehow an egg-info folder or file in site-packages,
> or that the project name is not what you are expecting ('wxpython')
> 
> Look for that egg-info and search for the name of the project. In
> *egg-info/PKG-INFO or in egg-info itself.
> 
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 
> 

Well, about the unexpected name thing, I don't think so, cause I printed the 
entire working_set and looked at the names one-by-one, and wxPython wasn't there 
in any kind of different name.

I haven't investigated that egg-info thing, because it doesn't matter much to me 
what is the technical reason for this. The fact that `require` gave a false 
positive, and not even under any special circumstances, just a standard Windows 
installation of wxPython, is reason enough not to use it in my project.

Also, before I came to this conclusion I noticed the kind of error that 
pkg_resources raises:

    pkg_resources.DistributionNotFound: my-dependency

That is not too helpful. Yes, I noticed you have a todo comment on that error to 
make it more informative:

    raise DistributionNotFound(req)  # XXX put more info here

So here's a patch for you:

    raise DistributionNotFound('''The distribution %s is needed for this \
program, but it was not found on this system. Please search for it on the \
internet, then download and install it, then try this program again.''' % req)

Because when a potential user has just downloaded my program and he's trying it 
out, the last thing I want to do is scare him away with an obscure error 
message.

Ram.



From pje at telecommunity.com  Sat Nov 14 16:19:35 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 14 Nov 2009 10:19:35 -0500
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.co
 m>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<4AFA1CA0.3010209@ar.media.kyoto-u.ac.jp>
	<4AFA5652.8080708@canterbury.ac.nz>
	<4AFA540F.3050205@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110111p5d30a701sf225e613dd01106c@mail.gmail.com>
	<4AFA880F.8020801@ar.media.kyoto-u.ac.jp>
	<94bdd2610911110225l95820d4hfbd7ef6a3e1f9096@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
Message-ID: <20091114151936.EB6633A415F@sparrow.telecommunity.com>

At 03:13 PM 11/11/2009 +0100, Tarek Ziad? wrote:
>But you call it with "install" in your example, meaning that is is
>called at install time, right ?
>
>Or it is just that you want to get the "--prefix" value finalized and
>computed by the install
>command. If it's the later, I guess you will be able to use the
>upcoming "sysconfig" module,
>that gives you the install schemes, depending on sys.prefix/sys.exec_prefix.

The issue is that setup.py can accept multiple commands on the 
command line, and in principle "build_clib" might be being called as 
a subcommand of build (and thus of install).  So, he needs the 
*active* --prefix, either from the command line, config file(s), or 
defaults.  Simply having an API to get the defaults won't help this.

Really, getting a finalized "install" command object is the only way 
to do this correctly in distutils at the moment, and the sysconfig 
API won't change that.


From pje at telecommunity.com  Sat Nov 14 16:31:43 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 14 Nov 2009 10:31:43 -0500
Subject: [Distutils] Improving distutils vs redesigning it (was people
 want CPAN)
In-Reply-To: <hdgad0$neg$1@ger.gmane.org>
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<hdgad0$neg$1@ger.gmane.org>
Message-ID: <20091114153144.C3DE23A411A@sparrow.telecommunity.com>

At 12:36 AM 11/12/2009 -0600, Robert Kern wrote:
>Sorry, I edited out the bit at the last minute where I explained 
>that it would be great to have a centralized option-managing object 
>such that any command can ask what options were set on any other 
>regardless of the dependencies between commands.

Actually, that such a thing is needed in the first place is evidence 
of one of the deepest design flaws in the distutils -- the fact that 
things which are fundamentally system or project-level configuration 
values are defined in terms of options to commands!

The distutils is a definite case of "superficial design flaws being 
so annoying as to keep most people from noticing the fundamental 
design flaws".  ;-)


From ziade.tarek at gmail.com  Sat Nov 14 19:00:33 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 14 Nov 2009 19:00:33 +0100
Subject: [Distutils]
	=?windows-1252?q?=60pkg=5Fresources=2Erequire=28=29?=
	=?windows-1252?q?=60doesn=92t_see_wxPython?=
In-Reply-To: <loom.20091114T143334-146@post.gmane.org>
References: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com>
	<94bdd2610911140146id3bbff3yb28343275f97538b@mail.gmail.com>
	<loom.20091114T143334-146@post.gmane.org>
Message-ID: <94bdd2610911141000r6223beddp77d2c2335b19f40@mail.gmail.com>

On Sat, Nov 14, 2009 at 2:46 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
[..]
>
> I haven't investigated that egg-info thing, because it doesn't matter much to me
> what is the technical reason for this. The fact that `require` gave a false
> positive, and not even under any special circumstances, just a standard Windows
> installation of wxPython, is reason enough not to use it in my project.

What is your Python version ? Starting at Python 2.5, all
distutils-based installers add an egg-info
file or folder alongside the packages it installs in your system.

If it's not present it could be a partial installation, or a bug. I
would be interested in your feedback
if you do investigate some more.


>
> Also, before I came to this conclusion I noticed the kind of error that
> pkg_resources raises:
>
> ? ?pkg_resources.DistributionNotFound: my-dependency
>
> That is not too helpful. Yes, I noticed you have a todo comment on that error to
> make it more informative:
>
> ? ?raise DistributionNotFound(req) ?# XXX put more info here
>
> So here's a patch for you:
>
> ? ?raise DistributionNotFound('''The distribution %s is needed for this \
> program, but it was not found on this system. Please search for it on the \
> internet, then download and install it, then try this program again.''' % req)
>
> Because when a potential user has just downloaded my program and he's trying it
> out, the last thing I want to do is scare him away with an obscure error
> message.

Sure, that's helpfull. I am adding this in Distribute, for the 0.6.9
upcoming release.

Tarek

From cool-rr at cool-rr.com  Sat Nov 14 19:10:37 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Sat, 14 Nov 2009 20:10:37 +0200
Subject: [Distutils]
	=?windows-1252?q?=60pkg=5Fresources=2Erequire=28=29?=
	=?windows-1252?q?=60doesn=92t_see_wxPython?=
In-Reply-To: <94bdd2610911141000r6223beddp77d2c2335b19f40@mail.gmail.com>
References: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com> 
	<94bdd2610911140146id3bbff3yb28343275f97538b@mail.gmail.com> 
	<loom.20091114T143334-146@post.gmane.org>
	<94bdd2610911141000r6223beddp77d2c2335b19f40@mail.gmail.com>
Message-ID: <a422b72d0911141010l3e0c362aj29bc8dbbcf0214b6@mail.gmail.com>

>
>
> What is your Python version ? Starting at Python 2.5, all
> distutils-based installers add an egg-info
> file or folder alongside the packages it installs in your system.
>
> If it's not present it could be a partial installation, or a bug. I
> would be interested in your feedback
> if you do investigate some more.
>

I did it on 2.6, but then I tried also on 2.5 and 2.4, and it happened the
same. I looked in site-packages and didn't find a egg, but I don't know much
about eggs so I don't know.

The way I installed wxPython is from the standard Windows installer that I
downloaded from wxpython.org. You can give it a try yourself, install using
their installer (Assuming you have access to a Windows box) and do the
require thing. If it works for you then there will be reason to think
something is fucked up in my system, and I'll investigate.



> Sure, that's helpfull. I am adding this in Distribute, for the 0.6.9
> upcoming release.
>
>
Great :)

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091114/2a1311a7/attachment.htm>

From ziade.tarek at gmail.com  Sat Nov 14 19:21:29 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 14 Nov 2009 19:21:29 +0100
Subject: [Distutils]
	=?windows-1252?q?=60pkg=5Fresources=2Erequire=28=29?=
	=?windows-1252?q?=60doesn=92t_see_wxPython?=
In-Reply-To: <a422b72d0911141010l3e0c362aj29bc8dbbcf0214b6@mail.gmail.com>
References: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com>
	<94bdd2610911140146id3bbff3yb28343275f97538b@mail.gmail.com>
	<loom.20091114T143334-146@post.gmane.org>
	<94bdd2610911141000r6223beddp77d2c2335b19f40@mail.gmail.com>
	<a422b72d0911141010l3e0c362aj29bc8dbbcf0214b6@mail.gmail.com>
Message-ID: <94bdd2610911141021i55ff52b1x670f74bfa009245f@mail.gmail.com>

On Sat, Nov 14, 2009 at 7:10 PM, cool-RR <cool-rr at cool-rr.com> wrote:
[..]
>
> I did it on 2.6, but then I tried also on 2.5 and 2.4, and it happened the
> same. I looked in site-packages and didn't find a egg, but I don't know much
> about eggs so I don't know.

we are looking for a file or a folder named "wxPython*.egg-info" or
something like that,
that is added alongside a "wxpython" package I guess.

If you are not sure where to look at, do this in your prompt
(supposing the package is called "wxpython")

>>> import wxpython; wxpython

You will get the path of the package, and the egg-info is supposely
located alongside this package.


> The way I installed wxPython is from the standard Windows installer that I
> downloaded from wxpython.org. You can give it a try yourself, install using
> their installer (Assuming you have access to a Windows box) and do the
> require thing. If it works for you then there will be reason to think
> something is fucked up in my system, and I'll investigate.

I'll try in my VM when I get a chance.

>
>>
>> Sure, that's helpfull. I am adding this in Distribute, for the 0.6.9
>> upcoming release.
>>
>
> Great :)
>
> Ram.
> _______________________________________________
> Distutils-SIG maillist ?- ?Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>



-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From cool-rr at cool-rr.com  Sat Nov 14 19:33:49 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Sat, 14 Nov 2009 20:33:49 +0200
Subject: [Distutils]
	=?windows-1252?q?=60pkg=5Fresources=2Erequire=28=29?=
	=?windows-1252?q?=60doesn=92t_see_wxPython?=
In-Reply-To: <94bdd2610911141021i55ff52b1x670f74bfa009245f@mail.gmail.com>
References: <a422b72d0911131255m5f7edc41xdff3d82c3b950ca2@mail.gmail.com> 
	<94bdd2610911140146id3bbff3yb28343275f97538b@mail.gmail.com> 
	<loom.20091114T143334-146@post.gmane.org>
	<94bdd2610911141000r6223beddp77d2c2335b19f40@mail.gmail.com> 
	<a422b72d0911141010l3e0c362aj29bc8dbbcf0214b6@mail.gmail.com> 
	<94bdd2610911141021i55ff52b1x670f74bfa009245f@mail.gmail.com>
Message-ID: <a422b72d0911141033k50f7f910nb3e52be56e406d6a@mail.gmail.com>

>
>
> >>> import wxpython; wxpython
>
> You will get the path of the package, and the egg-info is supposely
> located alongside this package.
>
>
>
(It's actually `wx` when you use it, not `wxpython`).

I did that and got the path and searched around, but didn't find any egg.

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091114/91d71081/attachment.htm>

From ben+python at benfinney.id.au  Sun Nov 15 00:27:07 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 15 Nov 2009 10:27:07 +1100
Subject: [Distutils] Improving distutils vs redesigning it
References: <5b8d13220911111518yae0c7fbv1599099e8f950134@mail.gmail.com>
	<94bdd2610911111604r22723c73m6db3e21e52e541f@mail.gmail.com>
	<hdfjlk$6km$1@ger.gmane.org>
	<94bdd2610911111648s745e3e06l4e07449f99a5beb1@mail.gmail.com>
	<hdfml9$fbp$1@ger.gmane.org>
	<94bdd2610911111714r76533e0cl484d1f435fd8ccb1@mail.gmail.com>
	<hdfu07$un4$1@ger.gmane.org>
	<A5EED46D-48D7-4BD5-B7DE-7C01B9F06CFB@twistedmatrix.com>
	<4AFB9700.4030207@ar.media.kyoto-u.ac.jp>
	<B71492AC-7592-4D3D-89C5-6FBF8E10B937@twistedmatrix.com>
	<hdgad0$neg$1@ger.gmane.org>
	<20091114153144.C3DE23A411A@sparrow.telecommunity.com>
Message-ID: <87bpj4sn5g.fsf@benfinney.id.au>

"P.J. Eby" <pje at telecommunity.com> writes:

> The distutils is a definite case of "superficial design flaws being so
> annoying as to keep most people from noticing the fundamental design
> flaws". ;-)

Douglas Adams, RIP.

-- 
 \         ?Nature hath given men one tongue but two ears, that we may |
  `\          hear from others twice as much as we speak.? ?Epictetus, |
_o__)                                                      _Fragments_ |
Ben Finney


From ziade.tarek at gmail.com  Sun Nov 15 01:09:01 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 01:09:01 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
Message-ID: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>

Hello

I've update PEP 345 with a first draft about the markers,

http://python.org/dev/peps/pep-0345/#environment-markers

PEP 390 is being reworked accordingly, but I guess we can have a new round of
comments on PEP 345 and PEP 386, as they can be accepted and added in Python
independently from the other PEPs.

There's also a "Requires-External" field that was added in the first
update of 345, that we need to discuss.

The idea of this field is to be able to define a non-Python dependency
in the metadata. It's based on a list stored and managed at PyPI.

Values could be things like "libxslt", "libpng", etc..

Last, "Requires-Python" is introduced to define the version of Python.
I am not sure this is required anymore since Martin has added a Trove
classifier
for this. But in the meantime, this is stronger than a "simple"
classifier I think.

Any comments ?

Regards

Tarek

-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From ziade.tarek at gmail.com  Sun Nov 15 01:14:05 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 01:14:05 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <200911130746.19487.jeremy.kloth@gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<5b8d13220911130452i549e22a0obf51ec3d84735416@mail.gmail.com>
	<94bdd2610911130544r49aab04fne844a7cb468c90c3@mail.gmail.com>
	<200911130746.19487.jeremy.kloth@gmail.com>
Message-ID: <94bdd2610911141614w4e50d6cfs5ef49a4000dbc59f@mail.gmail.com>

On Fri, Nov 13, 2009 at 3:46 PM, Jeremy Kloth <jeremy.kloth at gmail.com> wrote:
[..]
>
> The exact thing being described has been done in 4Suite for 6 years (along
> with many other distutils improvements). Feel free to take or discuss or
> request help with any of the features/additions (like FHS layout of files) in
> the 4Suite distutils extensions. The code is available for browsing:
> http://cvs.4suite.org/viewcvs/4Suite/Ft/Lib/DistExt/

That's great, thanks for the pointer ! I'll look into it asap

>
> The way we've impl'd 'config' was as a prerequisite for 'build', just as
> 'build' is for 'install'. ?If any of the options stored by the 'config'
> command are overridden via 'build' or 'install' options, the 'config' command
> would be re-run to store the "new" choices.
>
> Any questions or just simple help with integrating a similar system, just let
> me know. Been there, done that.

So did you end up changing the way options are passed to the commands,
or do you just have a specific "config" command that looks over other options
passed to the other commands ?

Tarek

From ziade.tarek at gmail.com  Sun Nov 15 01:31:46 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 01:31:46 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <1257979783.4524.105.camel@idol>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<1257979783.4524.105.camel@idol>
Message-ID: <94bdd2610911141631i166bc305o11de254d5724f904@mail.gmail.com>

On Wed, Nov 11, 2009 at 11:49 PM, Pauli Virtanen <pav at iki.fi> wrote:
[..]
>
> Just to throw some wild, perhaps obvious, and definitely unasked-for
> ideas in the air (especially as I can't promise I can give any sustained
> help here :/ ):
>
> I suppose one option would be to factor *everything* related to
> extension module building into build_ext (and abolish build_clib): the
> rest of distutils would just say to build_ext
>
> ? ? ? ?"""
> ? ? ? ?Here's some info about the environment:
>
> ? ? ? ?dict(python_lib_path=/usr/lib/..., optimize=yes/no,
> ? ? ? ? ? ? python_lib_name=..., python_includes=/usr/include/...,
> ? ? ? ? ? ? install_prefix=/usr/lib/python2.6, ...
> ? ? ? ? ? ? ..., python_extension=..., build_temp_dir=...)
>
> ? ? ? ?Look the rest up from sysconfig.
>
> ? ? ? ?Please build any extensions and data files you like,
> ? ? ? ?and tell the file and directory names where you placed them
> ? ? ? ?and where (relative paths) they should go.
> ? ? ? ?"""

I ought to be something like that, but what is unclear to me is how to
describe which compiler to use for what files.

I had this "one extension == one compiler type" pattern in my head,
but it seems more complex than that. IOW an extension can invoke
several compilers and tools to be built.

So "one extension == one extension builder" might best describe it.
and I am wondering
if we can't define a simple interface for these extension builders,
from the simplest case (one tool uses one compiler) to the weirdest
one (one tool uses a complex toolchain to create the extension)

So at the end we would have:

- Extension (the existing extension class, that takes a
"extension_builder_type")
- ExtensionBuilder (class in charge of creating an extension)
- a registery for ExtensionBuilder subclasses

And have the community create new ExtensionBuilder subclasses that
could be registered like command.

build_ext would then become an empty shell, just in charge of looping
through the extensions,
so each extension invokes its builder.


[..]
> I think this idea quickly boils down more or less to David's idea about
> a pluggable build system -- implementing a good one takes a lot of work,
> so it might make sense to refactor distutils so that it would be
> possible [1] to use some of the existing ones (scons, waf, whatever,
> heck, even autoconf+make). The *default* build system could be a simple
> one, and backwards compatible. Especially so, since it seems to me that
> building extension modules is orthogonal to what distutils and
> setuptools want to do -- package and install Python modules in a
> Python-specific way, not really worry about how to properly call
> different compilers on obscure platforms.
>
> Anyway, even in the case pluggability is a bad idea, refactoring the
> build system out from the rest of distutils might make sense.

Agreed. it seems that the addition of the "configure" command, and the
refactoring
of the "build_ext" one, are the right things to do, together with the addition
of the "sysconfig" stdlib module (which allows configure to get more info that
what distutils.sysconfig provides)

Now, if we can take back the work done in 4suite as suggested, or in scons, etc,
that's even better.


Tarek

From jeremy.kloth at gmail.com  Sun Nov 15 03:44:18 2009
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Sat, 14 Nov 2009 19:44:18 -0700
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911141614w4e50d6cfs5ef49a4000dbc59f@mail.gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<200911130746.19487.jeremy.kloth@gmail.com>
	<94bdd2610911141614w4e50d6cfs5ef49a4000dbc59f@mail.gmail.com>
Message-ID: <200911141944.18925.jeremy.kloth@gmail.com>

On Saturday 14 November 2009 05:14:05 pm Tarek Ziad? wrote:
> On Fri, Nov 13, 2009 at 3:46 PM, Jeremy Kloth <jeremy.kloth at gmail.com>
> > The way we've impl'd 'config' was as a prerequisite for 'build', just as
> > 'build' is for 'install'.  If any of the options stored by the 'config'
> > command are overridden via 'build' or 'install' options, the 'config'
> > command would be re-run to store the "new" choices.
> >
> > Any questions or just simple help with integrating a similar system, just
> > let me know. Been there, done that.
> 
> So did you end up changing the way options are passed to the commands,
> or do you just have a specific "config" command that looks over other
>  options passed to the other commands ?

It is done with the 'config' command having all the options used by the other 
commands.  The other commands would then look up their options' values from 
'config' (if not supplied on the command-line). If, for example, `--prefix` 
was supplied to 'install', the 'install' command would then cause the 'config' 
to redo its stored options.

If at all possible, I would eliminate the redundant options on 
build_*/install_* and leave them solely on 'config' as it greatly simplifies 
the interaction of the commands.

Jeremy
-- 
Jeremy Kloth
http://4suite.org/

From kevin at bud.ca  Sun Nov 15 07:18:41 2009
From: kevin at bud.ca (Kevin Teague)
Date: Sat, 14 Nov 2009 22:18:41 -0800
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
Message-ID: <7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>


On Nov 14, 2009, at 4:09 PM, Tarek Ziad? wrote:

>
> Last, "Requires-Python" is introduced to define the version of Python.
> I am not sure this is required anymore since Martin has added a Trove
> classifier
> for this. But in the meantime, this is stronger than a "simple"
> classifier I think.
>

+1 for Requires-Python.

This is a simpler field to understand than scanning through a list of  
Trove classifiers. Both for filling out the metadata, and for  
displaying on PyPI, since it would (arguably) be easier to read:

Requires Python: >2.4, <2.7

Than:

Categories:
   Programming Language :: Python :: 2.4
   Programming Language :: Python :: 2.5
   Programming Language :: Python :: 2.6

Especially for packages with a large number of categories. Although  
the easiest to read would be:

Requires Python: 2.4, 2.5, 2.6

At least I find it easier to map "2.4, 2.5, 2.6" to "requires Python  
2.4, or Python 2.5 or Python 2.6" than translating from the >< ranges  
into specific Python releases supported (which the Trove categories  
already naturally suports). Perhaps just making a note in the PEP or  
Distutils docs recommending this format "2.4, 2.5, 2.6", or at least  
making it the primary example of using field, and recommending other  
uses for special-case packages.

Erm, actually, re-reading the Requires Python section in PEP 345 I  
guess the conditional operator is required, so it would actually be:

Requires Python: ==2.4, ==2.5, ==2.6

Is that right? Perhaps in the absence of a conditional operator, then  
== should be the default, since I think it'd be easy to misunderstand  
this field upon casual usage. But then it should also make explicit in  
the PEP that ==2.5 is taken to mean only 2.5.0 or the entire sereis of  
releases in the 2.5 line.

But then arguably there aren't even enough special cases to justify  
the extra syntax to support things such as ">= 2.5.2"? I would imagine  
that almost all packages, when they release just test against the  
latest releases of whatever versions of Python that project supports?  
Also for the minor releases the other Python implementations, they  
aren't lock step with the CPython minor releases are they? e.g. Jython  
has a 2.5.0 release and 2.5.1 release, but that 2.5.1 isn't the same  
as CPython 2.5.1 (I could be wrong about this, haven't looked in depth).

Finally, it actually only makes sense to tag specific distributions  
for which versions of Python they support. So for example the Grok  
project:

Grok 1.0:
   Requires Python: 2.4, 2.5

Grok 1.1:
   Requires Python: 2.5, 2.6

Which means that they PyPI would say "Grok supports Python 2.5 and  
Python 2.6" after 1.1 is released. Which might not be true if there  
was still maintenance releases happening in the 1.0.x line. So they  
what should the project specify here?

So maybe I'm -0 on this field, since it does seems ambigous or the  
information is specific to the distributions and not the overall  
project.







From greg.ewing at canterbury.ac.nz  Sun Nov 15 07:24:52 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 15 Nov 2009 19:24:52 +1300
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911141631i166bc305o11de254d5724f904@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<loom.20091111T124450-168@post.gmane.org>
	<94bdd2610911110516w791f7103ibffbcaaa41f6fff8@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<1257979783.4524.105.camel@idol>
	<94bdd2610911141631i166bc305o11de254d5724f904@mail.gmail.com>
Message-ID: <4AFF9EB4.1030205@canterbury.ac.nz>

Tarek Ziad? wrote:

> And have the community create new ExtensionBuilder subclasses that
> could be registered like command.

I don't see a need for registering anything. You should
just be able to explicitly say what tool to use for each
stage of the process.

I envisage something like this:

   from distutils import Extension, CCompile
   from pyrex.distutils import PyrexCompile

   foo_ext = Extension("foo",
     CCompile(
       PyrexCompile("foo.pyx"),
       "somelib.c"))

Here Extension, CCompile and PyrexCompile are constructors
for dependency graph nodes. Their responsibilities are:

   Extension -- takes compiled object files and libraries
     and links them into a Python extension.

   CCompile -- takes C source files and turns them into
     object files.

   PyrexCompile -- takes Pyrex source files and turns
     them into C source files.

They would of course also take other relevant arguments
such as compiler flags, search paths, etc.

-- 
Greg

From cool-rr at cool-rr.com  Sun Nov 15 17:14:14 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Sun, 15 Nov 2009 16:14:14 +0000 (UTC)
Subject: [Distutils] Including package data with Distribute
Message-ID: <loom.20091115T171241-589@post.gmane.org>

Hey guys,

Please take a look at my question about including package data:

http://stackoverflow.com/questions/1734373/including-package-data-with-distribute/

(I don't care whether you answer here or on StackOverflow.)

Ram.


From ziade.tarek at gmail.com  Sun Nov 15 17:15:33 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 17:15:33 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <200911141944.18925.jeremy.kloth@gmail.com>
References: <4AFB8F5D.4070302@ar.media.kyoto-u.ac.jp>
	<200911130746.19487.jeremy.kloth@gmail.com>
	<94bdd2610911141614w4e50d6cfs5ef49a4000dbc59f@mail.gmail.com>
	<200911141944.18925.jeremy.kloth@gmail.com>
Message-ID: <94bdd2610911150815y5e2dd12cu285bbe04fc49451d@mail.gmail.com>

On Sun, Nov 15, 2009 at 3:44 AM, Jeremy Kloth <jeremy.kloth at gmail.com> wrote:
[..]
>> So did you end up changing the way options are passed to the commands,
>> or do you just have a specific "config" command that looks over other
>> ?options passed to the other commands ?
>
> It is done with the 'config' command having all the options used by the other
> commands. ?The other commands would then look up their options' values from
> 'config' (if not supplied on the command-line). If, for example, `--prefix`
> was supplied to 'install', the 'install' command would then cause the 'config'
> to redo its stored options.
>
> If at all possible, I would eliminate the redundant options on
> build_*/install_* and leave them solely on 'config' as it greatly simplifies
> the interaction of the commands.

So, pratically speaking, if:

$ python setup.py install

is called, the install command will instanciate the configure command, that will
return options that were stored in some file, in a previous call ?

But, for the option redundancy problem, the simplest way I can see to
have the "configure" options in "install", or to let the end user pass
them along, would be
to make "configure" the base class for all commands that are part of
the configure-make-install story,
so when they run they can read and write options in the stored file
and use if needed the options
passed in the command line.

Tarek

From ziade.tarek at gmail.com  Sun Nov 15 17:22:56 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 17:22:56 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <4AFF9EB4.1030205@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<1257979783.4524.105.camel@idol>
	<94bdd2610911141631i166bc305o11de254d5724f904@mail.gmail.com>
	<4AFF9EB4.1030205@canterbury.ac.nz>
Message-ID: <94bdd2610911150822y5284110cvf19e8ace569110af@mail.gmail.com>

On Sun, Nov 15, 2009 at 7:24 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Tarek Ziad? wrote:
>
>> And have the community create new ExtensionBuilder subclasses that
>> could be registered like command.
>
> I don't see a need for registering anything. You should
> just be able to explicitly say what tool to use for each
> stage of the process.
>
> I envisage something like this:
>
> ?from distutils import Extension, CCompile
> ?from pyrex.distutils import PyrexCompile
>
> ?foo_ext = Extension("foo",
> ? ?CCompile(
> ? ? ?PyrexCompile("foo.pyx"),
> ? ? ?"somelib.c"))
>
> Here Extension, CCompile and PyrexCompile are constructors
> for dependency graph nodes. Their responsibilities are:
>
> ?Extension -- takes compiled object files and libraries
> ? ?and links them into a Python extension.
>
> ?CCompile -- takes C source files and turns them into
> ? ?object files.
>
> ?PyrexCompile -- takes Pyrex source files and turns
> ? ?them into C source files.
>
> They would of course also take other relevant arguments
> such as compiler flags, search paths, etc.

The advantage of the registery is that a project can provide a compiler type,
let's say "Pyrex". Then you can use in your own project setup.py this compiler
without explicitely importing something.

But the result is similar, and explicit imports should work too, so
maybe registeries
are just sugar on the top of something we first need to make work.

Tarek

From ziade.tarek at gmail.com  Sun Nov 15 17:37:52 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 17:37:52 +0100
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <loom.20091115T171241-589@post.gmane.org>
References: <loom.20091115T171241-589@post.gmane.org>
Message-ID: <94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>

On Sun, Nov 15, 2009 at 5:14 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> Hey guys,
>
> Please take a look at my question about including package data:
>
> http://stackoverflow.com/questions/1734373/including-package-data-with-distribute/
>
> (I don't care whether you answer here or on StackOverflow.)
>

I've change that behavior in Distutils (in Python trunk (2.7/3.2) )

Now all files mentioned in package_data will be included by default
without having to
write a MANIFEST.in file, and without having to use the magic behavior
based on DVCS.

Until then, I would recommend using an explicit MANIFEST.in and stick
with plain Distutils options,
so you don't rely on any VCS, and you don't add files by accidents,
that are in your
repository, but that you don't want to see added in your release.

Distribute 0.7.x will probably stick with Distutils upcoming default behavior.

Tarek

From greg.ewing at canterbury.ac.nz  Sun Nov 15 22:33:06 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 16 Nov 2009 10:33:06 +1300
Subject: [Distutils] People want CPAN
In-Reply-To: <94bdd2610911150822y5284110cvf19e8ace569110af@mail.gmail.com>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<5b8d13220911110548m7b3624aoa7a9716d8aeaf956@mail.gmail.com>
	<94bdd2610911110613s73c496efuae31b46bac1e1a3e@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<1257979783.4524.105.camel@idol>
	<94bdd2610911141631i166bc305o11de254d5724f904@mail.gmail.com>
	<4AFF9EB4.1030205@canterbury.ac.nz>
	<94bdd2610911150822y5284110cvf19e8ace569110af@mail.gmail.com>
Message-ID: <4B007392.30907@canterbury.ac.nz>

Tarek Ziad? wrote:

> But the result is similar, and explicit imports should work too, so
> maybe registeries
> are just sugar on the top of something we first need to make work.

It's completely unnecessary sugar, if you ask me.
I don't see what's bad about importing functionality
you want to use.

Where and how do you intend the registration to happen,
anyway? Would it be done by the setup.py script? In
that case I don't see how it saves you anything, since
you would have to first import the thing you want to
register anyway.

Or are you envisaging that Pyrex or whatever tool is
involved would somehow patch itself into distutils
when you install it? I don't like that idea much,
since it smacks of the kind of monkeypatching that
setuptools is reviled for.

-- 
Greg

From wentland at cl.uni-heidelberg.de  Sun Nov 15 23:17:03 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Sun, 15 Nov 2009 23:17:03 +0100
Subject: [Distutils] Wiki page for ideas and proposals
Message-ID: <20091115221703.GA9776@kinakuta.local>

Hi all,

I have created a Wiki page that gives an overview on the PEPs that are
currently drafted and some of the proposals that were mentioned here
on the list.

You can find it at:

http://wiki.python.org/moin/Distutils/DiscussionOverview

I would be happy if you could add your own proposals/ideas or comment on
those that are already present.

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091115/ef42b577/attachment.pgp>

From ziade.tarek at gmail.com  Sun Nov 15 23:22:48 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 15 Nov 2009 23:22:48 +0100
Subject: [Distutils] People want CPAN
In-Reply-To: <4B007392.30907@canterbury.ac.nz>
References: <ca471dc20911060953v1db91bbei992dcb964b6ab88b@mail.gmail.com>
	<5b8d13220911110647y3a8b5de8p4a830ff337138a2@mail.gmail.com>
	<94bdd2610911110747p61fa1d67y17fda0a65aa3bbe5@mail.gmail.com>
	<1257970956.4524.31.camel@idol>
	<94bdd2610911111242q4502f0ddr843df4090ea977e8@mail.gmail.com>
	<1257979783.4524.105.camel@idol>
	<94bdd2610911141631i166bc305o11de254d5724f904@mail.gmail.com>
	<4AFF9EB4.1030205@canterbury.ac.nz>
	<94bdd2610911150822y5284110cvf19e8ace569110af@mail.gmail.com>
	<4B007392.30907@canterbury.ac.nz>
Message-ID: <94bdd2610911151422l4d5f048ekf453ef27ad383ca0@mail.gmail.com>

On Sun, Nov 15, 2009 at 10:33 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Tarek Ziad? wrote:
>
>> But the result is similar, and explicit imports should work too, so
>> maybe registeries
>> are just sugar on the top of something we first need to make work.
>
> It's completely unnecessary sugar, if you ask me.
> I don't see what's bad about importing functionality
> you want to use.
>
> Where and how do you intend the registration to happen,
> anyway? Would it be done by the setup.py script? In
> that case I don't see how it saves you anything, since
> you would have to first import the thing you want to
> register anyway.
>
> Or are you envisaging that Pyrex or whatever tool is
> involved would somehow patch itself into distutils
> when you install it? I don't like that idea much,
> since it smacks of the kind of monkeypatching that
> setuptools is reviled for.
>

Patching ?

No, I was thinking about a basic plugin registery, exactly like what we
have *now* for commands with distutils.cfg, which is a simple configparser
file where you can point packages that contains commands, so they are loaded
when Distutils is run. (that's the "command-packages" option)

So, using the same technique, we can explicitely list in such .cfg
what are the compilers and where they are:

  [compilers]
  pyrex=pyrex.distutils:PyrexCompile


Tarek

From cool-rr at cool-rr.com  Mon Nov 16 02:47:55 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 01:47:55 +0000 (UTC)
Subject: [Distutils] Deleting build directory
References: <loom.20091113T220808-850@post.gmane.org>
	<877htut4ik.fsf@benfinney.id.au>
	<loom.20091114T002739-269@post.gmane.org>
	<5b8d13220911131539j6e4f68a4hc2e74147a70cc0fa@mail.gmail.com>
	<a422b72d0911131543m7c605d94t1b11bdf7b52aa531@mail.gmail.com>
	<87y6m9sqav.fsf@benfinney.id.au>
Message-ID: <loom.20091116T024649-483@post.gmane.org>

Ben Finney <ben+python <at> benfinney.id.au> writes:

> 
> cool-RR <cool-rr <at> cool-rr.com> writes:
> 
> > What I really want is never having to worry about the build directory
> > being around after doing any actions with `setup.py`. Do you have any
> > other suggestion?
> 
> If you have determined that ?setup.py install clean? does what you want,
> then you can create a user-specific ?setup.cfg? with this section:
> 
>     [aliases]
>     install = install clean
> 


I just tried `install clean` and it leaves a
build directory. I'll be using distutils to
delete it until someone comes up with a
Distribute-provided solution.

Ram.


From cool-rr at cool-rr.com  Mon Nov 16 03:28:10 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 02:28:10 +0000 (UTC)
Subject: [Distutils] Including package data with Distribute
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
Message-ID: <loom.20091116T032551-371@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:


> Until then, I would recommend using an explicit MANIFEST.in and stick
> with plain Distutils options,


You said on StackOverflow to look in distutils' docs to read about the template 
language for MANIFEST.in.

http://docs.python.org/distutils/commandref.html?highlight=manifest

Is this it?

It's pretty lame. Does this really not support Windows? Please help.

Ram.


From cournape at gmail.com  Mon Nov 16 03:44:17 2009
From: cournape at gmail.com (David Cournapeau)
Date: Mon, 16 Nov 2009 11:44:17 +0900
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <loom.20091116T032551-371@post.gmane.org>
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<loom.20091116T032551-371@post.gmane.org>
Message-ID: <5b8d13220911151844k7c5c5995v6a45e02ad00710a@mail.gmail.com>

On Mon, Nov 16, 2009 at 11:28 AM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> Tarek Ziad? <ziade.tarek <at> gmail.com> writes:
>
>
>> Until then, I would recommend using an explicit MANIFEST.in and stick
>> with plain Distutils options,
>
>
> You said on StackOverflow to look in distutils' docs to read about the template
> language for MANIFEST.in.
>
> http://docs.python.org/distutils/commandref.html?highlight=manifest
>
> Is this it?
>
> It's pretty lame.

Yes it is, but other options are not better ATM. Including data files
in a predictible and simple way is a recurrent problem with distutils.

> Does this really not support Windows?

It should. I don't remember having problems on windows with
MANIFEST.in. There also used to be problems where the MANIFEST was not
regenerated correctly - I almost always remove MANIFEST everytime in
my setup.py for most of my packages because of this.

The problem is known, and is being discussed right now for distribute,
so hopefully things will improve.

David

From ziade.tarek at gmail.com  Mon Nov 16 10:38:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 10:38:11 +0100
Subject: [Distutils] [Python-Dev] PyPI comments and ratings, *really*?
In-Reply-To: <bad82a80911140734i822938fm8b57d15db5364c66@mail.gmail.com>
References: <4AFC9064.20700@v.loewis.de>
	<20091113221234.GA15732@amk-desktop.matrixgroup.net>
	<87bpj6t4mp.fsf@benfinney.id.au>
	<200911141018.13536.steve@pearwood.info> <hdksh9$jkt$1@ger.gmane.org>
	<20091114152921.GA28003@laurie.devork>
	<bad82a80911140734i822938fm8b57d15db5364c66@mail.gmail.com>
Message-ID: <94bdd2610911160138k3f973a15i78d49cd36534df23@mail.gmail.com>

On Sat, Nov 14, 2009 at 4:34 PM, Arc Riley <arcriley at gmail.com> wrote:
>
>> +1
>>
>> Having a "Repository-URL", "Repository-Browse-URL" and a
>> "Bug-Tracker-URL" field in PyPI would be a lot more usefule then
>> comments and ratings.
>>
>
> +1

You mean in the Metadata ? We are currenty working on adding fields in them,
for an upcoming 1.2 version (see PEP 345)

I am crossposting to distutils-sig so we can discuss this point over
there, because I think that would
be a great triple of fields to add in 1.2

Tarek

From mal at egenix.com  Mon Nov 16 11:55:44 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 16 Nov 2009 11:55:44 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
Message-ID: <4B012FB0.30407@egenix.com>

Tarek Ziad? wrote:
> Hello
> 
> I've update PEP 345 with a first draft about the markers,
> 
> http://python.org/dev/peps/pep-0345/#environment-markers

Thanks for adding this.

One detail to add:

python_full_version = sys.version.split()[0]

This is important, since a package may well only support Python
2.5 starting with patch level release 2.5.2 and python_version
only includes the major.minor version information.

> PEP 390 is being reworked accordingly, but I guess we can have a new round of
> comments on PEP 345 and PEP 386, as they can be accepted and added in Python
> independently from the other PEPs.
> 
> There's also a "Requires-External" field that was added in the first
> update of 345, that we need to discuss.
> 
> The idea of this field is to be able to define a non-Python dependency
> in the metadata. It's based on a list stored and managed at PyPI.
> 
> Values could be things like "libxslt", "libpng", etc..

Am I right in understanding this as informational field only ?

Different systems have different ways of naming such external
dependencies, so it's unlikely that we can come up with our own
little standard set of names for everything, e.g. you could
use any of these names for a dependency on zlib and its header
files: "zlib", "libz", "zlib-devel", "zlib-dev".

It is also not clear where to draw the line and how to manage
multiple mentions with slightly different focus, e.g. would you have
both "zlib" and "zlib-devel" (extensions may require the binary
and/or the lib and header files), or would "zlib" include the header
files dependency ?

Given the complexity this adds, I'm not sure whether it's worth
trying to come up with a fixed list of names for external
dependencies.

> Last, "Requires-Python" is introduced to define the version of Python.
> I am not sure this is required anymore since Martin has added a Trove
> classifier
> for this. But in the meantime, this is stronger than a "simple"
> classifier I think.

This may be useful for cases like the one mentioned above (patch
level requirements), which are not covered by the trove classifiers.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 16 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ozarow at gmail.com  Mon Nov 16 13:31:49 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Mon, 16 Nov 2009 13:31:49 +0100
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
Message-ID: <20091116123149.GM2966@piotro.eu>

[Tarek Ziad?, 2009-11-15]
> On Sun, Nov 15, 2009 at 5:14 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> > Hey guys,
> >
> > Please take a look at my question about including package data:
> >
> > http://stackoverflow.com/questions/1734373/including-package-data-with-distribute/
> >
> > (I don't care whether you answer here or on StackOverflow.)
> >
> 
> I've change that behavior in Distutils (in Python trunk (2.7/3.2) )

semi related:

What's the preferred way to notify module (at install stage) about new
data location when someone overrides --install-data
(--install-data=/usr/share/foo/)?
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From cool-rr at cool-rr.com  Mon Nov 16 14:46:58 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 13:46:58 +0000 (UTC)
Subject: [Distutils] Including package data with Distribute
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<loom.20091116T032551-371@post.gmane.org>
	<5b8d13220911151844k7c5c5995v6a45e02ad00710a@mail.gmail.com>
Message-ID: <loom.20091116T144325-406@post.gmane.org>

David Cournapeau <cournape <at> gmail.com> writes:

> 
> On Mon, Nov 16, 2009 at 11:28 AM, Ram Rachum <cool-rr <at> cool-rr.com> wrote:

> >
> >
> > You said on StackOverflow to look in distutils' docs to read about the 
template
> > language for MANIFEST.in.
> >
> > http://docs.python.org/distutils/commandref.html?highlight=manifest
> >
> > Is this it?
> >
> > It's pretty lame.
> 
> Yes it is, but other options are not better ATM. Including data files
> in a predictible and simple way is a recurrent problem with distutils.
> 
> > Does this really not support Windows?
> 
> It should. I don't remember having problems on windows with
> MANIFEST.in. There also used to be problems where the MANIFEST was not
> regenerated correctly - I almost always remove MANIFEST everytime in
> my setup.py for most of my packages because of this.
> 
> The problem is known, and is being discussed right now for distribute,
> so hopefully things will improve.
> 
> David


I'm really lost here.

Can someone answer these question? Will MANIFEST.in work on Windows? Do I have 
to delete the MANIFEST file every time? Is this the sanest way right now to 
include data files?

Ram.



From ziade.tarek at gmail.com  Mon Nov 16 15:00:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 15:00:32 +0100
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <loom.20091116T144325-406@post.gmane.org>
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<loom.20091116T032551-371@post.gmane.org>
	<5b8d13220911151844k7c5c5995v6a45e02ad00710a@mail.gmail.com>
	<loom.20091116T144325-406@post.gmane.org>
Message-ID: <94bdd2610911160600s4db26ddft52bd17d2fd193549@mail.gmail.com>

On Mon, Nov 16, 2009 at 2:46 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
>> > http://docs.python.org/distutils/commandref.html?highlight=manifest
>> >
>> > Is this it?
>> >
>> > It's pretty lame.

Contribution to the doc is always welcome.

> I'm really lost here.
>
> Can someone answer these question? Will MANIFEST.in work on Windows?

yes.

> Do I have to delete the MANIFEST file every time?

no. if you change MANIFEST.in, MANIFEST is rebuilt. You have to re-run
the sdist command though.

> Is this the sanest way right now to include data files?

yes.

From ziade.tarek at gmail.com  Mon Nov 16 15:01:29 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 15:01:29 +0100
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <20091116123149.GM2966@piotro.eu>
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<20091116123149.GM2966@piotro.eu>
Message-ID: <94bdd2610911160601y2f347cfble16835a198fe16e1@mail.gmail.com>

On Mon, Nov 16, 2009 at 1:31 PM, Piotr Ozarowski <ozarow at gmail.com> wrote:
[..]
>> I've change that behavior in Distutils (in Python trunk (2.7/3.2) )
>
> semi related:
>
> What's the preferred way to notify module (at install stage) about new
> data location when someone overrides --install-data
> (--install-data=/usr/share/foo/)?

What is "module" here ? what's your use case ?

From cool-rr at cool-rr.com  Mon Nov 16 15:09:13 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 14:09:13 +0000 (UTC)
Subject: [Distutils] Including package data with Distribute
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<loom.20091116T032551-371@post.gmane.org>
	<5b8d13220911151844k7c5c5995v6a45e02ad00710a@mail.gmail.com>
	<loom.20091116T144325-406@post.gmane.org>
	<94bdd2610911160600s4db26ddft52bd17d2fd193549@mail.gmail.com>
Message-ID: <loom.20091116T150701-668@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:


> no. if you change MANIFEST.in, MANIFEST is rebuilt. You have to re-run
> the sdist command though.
> 

Okay, new development. I've written a MAINFEST.in file like this:

recursive-include garlicsim_wx *
recursive-exclude garlicsim_wx *.pyc

When I do an sdist, I see all is good, it includes my images in the package.
But, when I do a bdist_msi and run the installer, it doesn't copy the images
to the garlicsim_wx folder in site-packages.

Why? What do I do?


From cool-rr at cool-rr.com  Mon Nov 16 15:25:19 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 14:25:19 +0000 (UTC)
Subject: [Distutils] Including package data with Distribute
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<loom.20091116T032551-371@post.gmane.org>
	<5b8d13220911151844k7c5c5995v6a45e02ad00710a@mail.gmail.com>
	<loom.20091116T144325-406@post.gmane.org>
	<94bdd2610911160600s4db26ddft52bd17d2fd193549@mail.gmail.com>
	<loom.20091116T150701-668@post.gmane.org>
Message-ID: <loom.20091116T152439-230@post.gmane.org>

Ram Rachum <cool-rr <at> cool-rr.com> writes:

> 

> Okay, new development. I've written a MAINFEST.in file like this:
> 
> recursive-include garlicsim_wx *
> recursive-exclude garlicsim_wx *.pyc
> 
> When I do an sdist, I see all is good, it includes my images in the package.
> But, when I do a bdist_msi and run the installer, it doesn't copy the images
> to the garlicsim_wx folder in site-packages.
> 
> Why? What do I do?


Sorry, please ignore. I forgot to do include_package_data=True.

Ram.


From ziade.tarek at gmail.com  Mon Nov 16 15:32:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 15:32:25 +0100
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <loom.20091116T150701-668@post.gmane.org>
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<loom.20091116T032551-371@post.gmane.org>
	<5b8d13220911151844k7c5c5995v6a45e02ad00710a@mail.gmail.com>
	<loom.20091116T144325-406@post.gmane.org>
	<94bdd2610911160600s4db26ddft52bd17d2fd193549@mail.gmail.com>
	<loom.20091116T150701-668@post.gmane.org>
Message-ID: <94bdd2610911160632v1f8c112ak24dda30929019943@mail.gmail.com>

On Mon, Nov 16, 2009 at 3:09 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> Tarek Ziad? <ziade.tarek <at> gmail.com> writes:
>
>
>> no. if you change MANIFEST.in, MANIFEST is rebuilt. You have to re-run
>> the sdist command though.
>>
>
> Okay, new development. I've written a MAINFEST.in file like this:
>
> recursive-include garlicsim_wx *
> recursive-exclude garlicsim_wx *.pyc
>
> When I do an sdist, I see all is good, it includes my images in the package.
> But, when I do a bdist_msi and run the installer, it doesn't copy the images
> to the garlicsim_wx folder in site-packages.
>
> Why? What do I do?

Did you include them in "package_data"  in your setup.py ?

From tseaver at palladion.com  Mon Nov 16 15:41:20 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 16 Nov 2009 09:41:20 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <4B012FB0.30407@egenix.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com>
Message-ID: <hdroah$dke$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

M.-A. Lemburg wrote:
> Tarek Ziad? wrote:
>> Hello
>>
>> I've update PEP 345 with a first draft about the markers,
>>
>> http://python.org/dev/peps/pep-0345/#environment-markers
> 
> Thanks for adding this.
> 
> One detail to add:
> 
> python_full_version = sys.version.split()[0]
> 
> This is important, since a package may well only support Python
> 2.5 starting with patch level release 2.5.2 and python_version
> only includes the major.minor version information.
> 
>> PEP 390 is being reworked accordingly, but I guess we can have a new round of
>> comments on PEP 345 and PEP 386, as they can be accepted and added in Python
>> independently from the other PEPs.
>>
>> There's also a "Requires-External" field that was added in the first
>> update of 345, that we need to discuss.
>>
>> The idea of this field is to be able to define a non-Python dependency
>> in the metadata. It's based on a list stored and managed at PyPI.
>>
>> Values could be things like "libxslt", "libpng", etc..
> 
> Am I right in understanding this as informational field only ?

Yes.  The primary consumers will be OS packagers, who will map them onto
their own package names.  As with other metadata they care about, we
hope that the packagers will suggest packages to make these names
"uniform" acrrss Python distributions.  I imagine there will be some
jockeying among them to find the "common" name for such things, which
they will then need to map.

> Different systems have different ways of naming such external
> dependencies, so it's unlikely that we can come up with our own
> little standard set of names for everything, e.g. you could
> use any of these names for a dependency on zlib and its header
> files: "zlib", "libz", "zlib-devel", "zlib-dev".
> 
> It is also not clear where to draw the line and how to manage
> multiple mentions with slightly different focus, e.g. would you have
> both "zlib" and "zlib-devel" (extensions may require the binary
> and/or the lib and header files), or would "zlib" include the header
> files dependency ?
> 
> Given the complexity this adds, I'm not sure whether it's worth
> trying to come up with a fixed list of names for external
> dependencies.

Again, this is mostly a place to put information as requested by the
downstream packagers.

>> Last, "Requires-Python" is introduced to define the version of Python.
>> I am not sure this is required anymore since Martin has added a Trove
>> classifier
>> for this. But in the meantime, this is stronger than a "simple"
>> classifier I think.
> 
> This may be useful for cases like the one mentioned above (patch
> level requirements), which are not covered by the trove classifiers.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksBZJAACgkQ+gerLs4ltQ7MuACeNPPxDLG+6bqQwxKxqhYfIQo9
js8AmwZy/Du9lMMoPmptzJBk7zfIVVEn
=dhZl
-----END PGP SIGNATURE-----


From ozarow at gmail.com  Mon Nov 16 16:10:06 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Mon, 16 Nov 2009 16:10:06 +0100
Subject: [Distutils] Including package data with Distribute
In-Reply-To: <94bdd2610911160601y2f347cfble16835a198fe16e1@mail.gmail.com>
References: <loom.20091115T171241-589@post.gmane.org>
	<94bdd2610911150837n18c1e120pf0bfa584325f099f@mail.gmail.com>
	<20091116123149.GM2966@piotro.eu>
	<94bdd2610911160601y2f347cfble16835a198fe16e1@mail.gmail.com>
Message-ID: <20091116151006.GN2966@piotro.eu>

[Tarek Ziad?, 2009-11-16]
> On Mon, Nov 16, 2009 at 1:31 PM, Piotr Ozarowski <ozarow at gmail.com> wrote:
> [..]
> >> I've change that behavior in Distutils (in Python trunk (2.7/3.2) )
> >
> > semi related:
> >
> > What's the preferred way to notify module (at install stage) about new
> > data location when someone overrides --install-data
> > (--install-data=/usr/share/foo/)?
> 
> What is "module" here ? what's your use case ?

any library/package/extension/whatever that sits in site-packages
directory

I use --install-data to install non .py/.so files outside site-packages
but then I have to patch the module to look for these files in the
right directory as it still looks for them in site-packages.

Wolodja mentioned something similar on [1] - if $data can be overridden
in .egg-info (using --install-data), then that's what I'm looking for.
It's just a proposal, though - I'm wondering how should I use
--install-data now.

[1] http://wiki.python.org/moin/Distutils/DiscussionOverview
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From tseaver at palladion.com  Mon Nov 16 16:53:01 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 16 Nov 2009 10:53:01 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <hdroah$dke$1@ger.gmane.org>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	<4B012FB0.30407@egenix.com>
	<hdroah$dke$1@ger.gmane.org>
Message-ID: <hdrsgt$uh4$2@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tres Seaver wrote:
> M.-A. Lemburg wrote:
>> Tarek Ziad? wrote:

>>> Values could be things like "libxslt", "libpng", etc..
>> Am I right in understanding this as informational field only ?
> 
> Yes.  The primary consumers will be OS packagers, who will map them onto
> their own package names.  As with other metadata they care about, we
> hope that the packagers will suggest packages to make these names

- --------------------------------------^ s/packages/patches/

> "uniform" acrrss Python distributions.  I imagine there will be some
> jockeying among them to find the "common" name for such things, which
> they will then need to map.

Dang muscle memory typed it for me.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksBdV0ACgkQ+gerLs4ltQ5UYQCg20vlmpBjAFdKeaUuNPOjyKaF
bnUAoMn43mmuYsem98rpis7GzEYmGBfb
=YuMe
-----END PGP SIGNATURE-----


From cool-rr at cool-rr.com  Mon Nov 16 17:10:43 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 16:10:43 +0000 (UTC)
Subject: [Distutils] Making a tar.gz source distribution
Message-ID: <loom.20091116T170945-424@post.gmane.org>

I'm trying to make a tar.gz source distribution. I saw in the docs that a tar 
program is needed. I am on Windows. Where do I get one?

Ram.


From ssteinerx at gmail.com  Mon Nov 16 17:19:20 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 16 Nov 2009 11:19:20 -0500
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <loom.20091116T170945-424@post.gmane.org>
References: <loom.20091116T170945-424@post.gmane.org>
Message-ID: <4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>


On Nov 16, 2009, at 11:10 AM, Ram Rachum wrote:

> I'm trying to make a tar.gz source distribution. I saw in the docs  
> that a tar
> program is needed. I am on Windows. Where do I get one?

You've got to be joking.

Try Google.

http://www.google.com/search?client=safari&rls=en&q=windows+tar+program&ie=UTF-8&oe=UTF-8

S


From ozarow at gmail.com  Mon Nov 16 17:24:37 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Mon, 16 Nov 2009 17:24:37 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External"
	and	"Requires-Python"
In-Reply-To: <hdroah$dke$1@ger.gmane.org>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>
Message-ID: <20091116162437.GO2966@piotro.eu>

[Tres Seaver, 2009-11-16]
> >> Values could be things like "libxslt", "libpng", etc..
> > 
> > Am I right in understanding this as informational field only ?
> 
> Yes.  The primary consumers will be OS packagers, who will map them onto
> their own package names.  As with other metadata they care about, we
> hope that the packagers will suggest packages to make these names
> "uniform" acrrss Python distributions.  I imagine there will be some
> jockeying among them to find the "common" name for such things, which
> they will then need to map.

How about putting there what we (distro maintainers) can pass to
ctypes.util.find_library()?

It would be great if there would be another field for build
dependencies, though. F.e. enchant extension required enchant headers to
build enchant.so (so libenchant-dev package was required at build
stage), but now it uses ctypes only and enchant.h is not needed at all.

Previously I used ldd (or dpkg-shlibdeps to be exact) to get runtime
dependencies from .so extension (so once I figured out the right build
dependencies, runtime dependencies were generated automatically),
now I use:

 from ctypes.util import find_library; print find_library("enchant")

to generate right depenencies, but I had to read the sources to figure
out what to pass to find_library.
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From ziade.tarek at gmail.com  Mon Nov 16 17:30:37 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 17:30:37 +0100
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
Message-ID: <94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com>

On Mon, Nov 16, 2009 at 5:19 PM, ssteinerX at gmail.com
<ssteinerx at gmail.com> wrote:
>
> On Nov 16, 2009, at 11:10 AM, Ram Rachum wrote:
>
>> I'm trying to make a tar.gz source distribution. I saw in the docs that a
>> tar
>> program is needed. I am on Windows. Where do I get one?
>
> You've got to be joking.
>
> Try Google.
>
> http://www.google.com/search?client=safari&rls=en&q=windows+tar+program&ie=UTF-8&oe=UTF-8

Notice that the upcoming version does not rely on externals program
anymore, and will use
the stdlib  module to work with the tar engine.

Regards

From cool-rr at cool-rr.com  Mon Nov 16 17:31:41 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 16:31:41 +0000 (UTC)
Subject: [Distutils] Making a tar.gz source distribution
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
Message-ID: <loom.20091116T172940-229@post.gmane.org>

ssteinerX <at> gmail.com <ssteinerx <at> gmail.com> writes:

> 
> 
> On Nov 16, 2009, at 11:10 AM, Ram Rachum wrote:
> 
> > I'm trying to make a tar.gz source distribution. I saw in the docs  
> > that a tar
> > program is needed. I am on Windows. Where do I get one?
> 
> You've got to be joking.
> 
> Try Google.
> 
> http://www.google.com/search?
client=safari&rls=en&q=windows+tar+program&ie=UTF-8&oe=UTF-8
> 
> S

No, I am not joking. I'm a windows user who wants to distribute his project in 
tar.gz, and I don't know how I'm supposed to operate this tar thing, or whether 
gzip is like tar, because that's the top result in Google. Also, even if I had 
that tar program, I wouldn't know how to make distutils use it.

So what am I supposed to do?

Ram.







From tseaver at palladion.com  Mon Nov 16 17:32:05 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 16 Nov 2009 11:32:05 -0500
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <loom.20091116T170945-424@post.gmane.org>
References: <loom.20091116T170945-424@post.gmane.org>
Message-ID: <hdruq5$7vu$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ram Rachum wrote:
> I'm trying to make a tar.gz source distribution. I saw in the docs that a tar 
> program is needed. I am on Windows. Where do I get one?

I consider it a bug that the distutils.archive_utils module wants to
spawn an external program to make a tarball / zipfile, rather than using
the perfectly good stdlib modules.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksBfn4ACgkQ+gerLs4ltQ58AQCgk9COc7i/mIwpJdUzpir/kpS1
MDoAn0ExIwxK/yha1JONkZ6XFoQ+VRa8
=dbAx
-----END PGP SIGNATURE-----


From cool-rr at cool-rr.com  Mon Nov 16 17:33:09 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 16 Nov 2009 16:33:09 +0000 (UTC)
Subject: [Distutils] Making a tar.gz source distribution
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com>
Message-ID: <loom.20091116T173221-764@post.gmane.org>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes:


> Notice that the upcoming version does not rely on externals program
> anymore, and will use
> the stdlib  module to work with the tar engine.
> 
> Regards
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG <at> python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 
> 


That's awesome, thanks. I hope it gets finished and adopted soon. But a solution 
for now is needed...

Ram.


From arve.knudsen at gmail.com  Mon Nov 16 17:37:43 2009
From: arve.knudsen at gmail.com (Arve Knudsen)
Date: Mon, 16 Nov 2009 17:37:43 +0100
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <loom.20091116T173221-764@post.gmane.org>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com>
	<loom.20091116T173221-764@post.gmane.org>
Message-ID: <a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com>

On Mon, Nov 16, 2009 at 5:33 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:

> Tarek Ziad? <ziade.tarek <at> gmail.com> writes:
>
>
> > Notice that the upcoming version does not rely on externals program
> > anymore, and will use
> > the stdlib  module to work with the tar engine.
> >
> > Regards
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG <at> python.org
> > http://mail.python.org/mailman/listinfo/distutils-sig
> >
> >
>
>
> That's awesome, thanks. I hope it gets finished and adopted soon. But a
> solution
> for now is needed...


Try this: http://gnuwin32.sourceforge.net/packages/libarchive.htm.

Arve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091116/81ce31ee/attachment.htm>

From ziade.tarek at gmail.com  Mon Nov 16 17:39:10 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 17:39:10 +0100
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <hdruq5$7vu$1@ger.gmane.org>
References: <loom.20091116T170945-424@post.gmane.org>
	<hdruq5$7vu$1@ger.gmane.org>
Message-ID: <94bdd2610911160839u37ac780fvdb708aca42b042cd@mail.gmail.com>

On Mon, Nov 16, 2009 at 5:32 PM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ram Rachum wrote:
>> I'm trying to make a tar.gz source distribution. I saw in the docs that a tar
>> program is needed. I am on Windows. Where do I get one?
>
> I consider it a bug that the distutils.archive_utils module wants to
> spawn an external program to make a tarball / zipfile, rather than using
> the perfectly good stdlib modules.

That's historical and not true anymore. -> the current trunk uses tarfile, etc.

Tarek.

From cool-rr at cool-rr.com  Mon Nov 16 17:45:12 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 16 Nov 2009 18:45:12 +0200
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com> 
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com> 
	<loom.20091116T173221-764@post.gmane.org>
	<a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com>
Message-ID: <a422b72d0911160845v5ce611f1s904207ec31624f2c@mail.gmail.com>

Arve Knudsen <arve.knudsen <at> gmail.com> writes: > Try this:
http://gnuwin32.sourceforge.net/packages/libarchive.htm. > > Arve  > I
downloaded it, but I have no idea how to make distutils use it. Ram
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091116/1be2f66b/attachment.htm>

From fwierzbicki at gmail.com  Mon Nov 16 17:48:29 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Mon, 16 Nov 2009 11:48:29 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
Message-ID: <4dab5f760911160848ib9526aeocbebd0811b7a197f@mail.gmail.com>

On Sun, Nov 15, 2009 at 1:18 AM, Kevin Teague <kevin at bud.ca> wrote:
> Also for the minor
> releases the other Python implementations, they aren't lock step with the
> CPython minor releases are they? e.g. Jython has a 2.5.0 release and 2.5.1
> release, but that 2.5.1 isn't the same as CPython 2.5.1 (I could be wrong
> about this, haven't looked in depth).
For Jython at least, this is correct. The 2.5 part tells you which
CPython we are targeting, but the x in 2.5.x is entirely Jython
specific.

-Frank

From ziade.tarek at gmail.com  Mon Nov 16 17:48:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 17:48:44 +0100
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <a422b72d0911160845v5ce611f1s904207ec31624f2c@mail.gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com>
	<loom.20091116T173221-764@post.gmane.org>
	<a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com>
	<a422b72d0911160845v5ce611f1s904207ec31624f2c@mail.gmail.com>
Message-ID: <94bdd2610911160848h3131cadalc8d45ec170cc70c2@mail.gmail.com>

On Mon, Nov 16, 2009 at 5:45 PM, cool-RR <cool-rr at cool-rr.com> wrote:
> Arve Knudsen <arve.knudsen <at> gmail.com> writes: > Try
> this:?http://gnuwin32.sourceforge.net/packages/libarchive.htm. > > Arve? > I
> downloaded it, but I have no idea how to make distutils use it. Ram

Distutils will try to find it in the PATH.

As far as I remember, the GnuWin32 tools adds themselves in the path, so just
try to call "tar" in a prompt, if it works, you are fine.

If not, add it in the PATH environment variables

Tarek

From cool-rr at cool-rr.com  Mon Nov 16 18:00:04 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 16 Nov 2009 19:00:04 +0200
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <94bdd2610911160848h3131cadalc8d45ec170cc70c2@mail.gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com> 
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com> 
	<loom.20091116T173221-764@post.gmane.org>
	<a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com> 
	<a422b72d0911160845v5ce611f1s904207ec31624f2c@mail.gmail.com> 
	<94bdd2610911160848h3131cadalc8d45ec170cc70c2@mail.gmail.com>
Message-ID: <a422b72d0911160900t380f37a3tbba50ad55308acba@mail.gmail.com>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes: > Distutils will try to
find it in the PATH. > > As far as I remember, the GnuWin32 tools adds
themselves in the path, so just > try to call "tar" in a prompt, if it
works, you are fine. > > If not, add it in the PATH environment variables >
> Tarek > _______________________________________________ Okay, I installed,
added to the PATH cause it didn't add it itself, made a copy of bsdtar.exe
named tar.exe. But when I ran sdist it crashed. Fuck it. I'll just compress
with WinRar. Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091116/352499db/attachment.htm>

From ziade.tarek at gmail.com  Mon Nov 16 18:21:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 18:21:25 +0100
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <a422b72d0911160900t380f37a3tbba50ad55308acba@mail.gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com>
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com>
	<loom.20091116T173221-764@post.gmane.org>
	<a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com>
	<a422b72d0911160845v5ce611f1s904207ec31624f2c@mail.gmail.com>
	<94bdd2610911160848h3131cadalc8d45ec170cc70c2@mail.gmail.com>
	<a422b72d0911160900t380f37a3tbba50ad55308acba@mail.gmail.com>
Message-ID: <94bdd2610911160921p33848079w656f0b220bb5627e@mail.gmail.com>

On Mon, Nov 16, 2009 at 6:00 PM, cool-RR <cool-rr at cool-rr.com> wrote:
> Okay, I installed, added to the PATH cause it didn't add it itself, made a copy of bsdtar.exe
> named tar.exe. But when I ran sdist it crashed. Fuck it. I'll just compress
> with WinRar.

bsdtar.exe ? you shouldn't have to rename anything, it's probably the problem.

I use myself  cygwin under windows, and it works fine without any renaming.
What was your traceback ?

and what's your tar version ?

$ tar --version

Distutils uses tar like this, so it might be that "bsdtar.exe" doesn't
work the same way.:

$ tar.ex -cf dist\archive.tar built_dir

Tarek

From cool-rr at cool-rr.com  Mon Nov 16 18:41:45 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 16 Nov 2009 19:41:45 +0200
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <94bdd2610911160921p33848079w656f0b220bb5627e@mail.gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>
	<4C57F0E5-1D3F-44B5-BC19-B5BA07E763A9@gmail.com> 
	<94bdd2610911160830y45c82835v24dc6a075f1e296@mail.gmail.com> 
	<loom.20091116T173221-764@post.gmane.org>
	<a0d6258d0911160837x460821abtb1273ab2eef850eb@mail.gmail.com> 
	<a422b72d0911160845v5ce611f1s904207ec31624f2c@mail.gmail.com> 
	<94bdd2610911160848h3131cadalc8d45ec170cc70c2@mail.gmail.com> 
	<a422b72d0911160900t380f37a3tbba50ad55308acba@mail.gmail.com> 
	<94bdd2610911160921p33848079w656f0b220bb5627e@mail.gmail.com>
Message-ID: <a422b72d0911160941k13e65d00g2275a3d2bec89720@mail.gmail.com>

Tarek Ziad? <ziade.tarek <at> gmail.com> writes: > bsdtar.exe ? you
shouldn't have to rename anything, it's probably the problem. > > I use
myself cygwin under windows, and it works fine without any renaming. > What
was your traceback ? > > and what's your tar version ? > > $ tar --version >
> Distutils uses tar like this, so it might be that "bsdtar.exe" doesn't >
work the same way.: > > $ tar.ex -cf dist\archive.tar built_dir > > Tarek
Never mind, this is too much work. I'll just use WinRAR.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091116/deaa3a64/attachment.htm>

From tseaver at palladion.com  Mon Nov 16 19:08:54 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Mon, 16 Nov 2009 13:08:54 -0500
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <94bdd2610911160839u37ac780fvdb708aca42b042cd@mail.gmail.com>
References: <loom.20091116T170945-424@post.gmane.org>	<hdruq5$7vu$1@ger.gmane.org>
	<94bdd2610911160839u37ac780fvdb708aca42b042cd@mail.gmail.com>
Message-ID: <hds4fm$ss4$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarek Ziad? wrote:
> On Mon, Nov 16, 2009 at 5:32 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Ram Rachum wrote:
>>> I'm trying to make a tar.gz source distribution. I saw in the docs that a tar
>>> program is needed. I am on Windows. Where do I get one?
>> I consider it a bug that the distutils.archive_utils module wants to
>> spawn an external program to make a tarball / zipfile, rather than using
>> the perfectly good stdlib modules.
> 
> That's historical and not true anymore. -> the current trunk uses tarfile, etc.

I didn't know that you had fixed it on the trunk. AFAIK, it is still
true that  every released version of Python has this bug.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksBlTIACgkQ+gerLs4ltQ6HOwCgxLJipF2/U18b2drfLL5EJx5E
pMIAnjAqCiKjwxszHDQmhJHHKj4ww0bc
=jhvI
-----END PGP SIGNATURE-----


From floris.bruynooghe at gmail.com  Mon Nov 16 19:22:22 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Mon, 16 Nov 2009 18:22:22 +0000
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <4B012FB0.30407@egenix.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com>
Message-ID: <20091116182222.GA29345@laurie.devork>

On Mon, Nov 16, 2009 at 11:55:44AM +0100, M.-A. Lemburg wrote:
> Tarek Ziad? wrote:
> One detail to add:
> 
> python_full_version = sys.version.split()[0]
> 
> This is important, since a package may well only support Python
> 2.5 starting with patch level release 2.5.2 and python_version
> only includes the major.minor version information.

I still like hex(sys.hexversion) for this.  IIRC this is what the docs
recommend to use inside python code too for checking versions.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ziade.tarek at gmail.com  Mon Nov 16 19:50:36 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 16 Nov 2009 19:50:36 +0100
Subject: [Distutils] Making a tar.gz source distribution
In-Reply-To: <hds4fm$ss4$1@ger.gmane.org>
References: <loom.20091116T170945-424@post.gmane.org>
	<hdruq5$7vu$1@ger.gmane.org>
	<94bdd2610911160839u37ac780fvdb708aca42b042cd@mail.gmail.com>
	<hds4fm$ss4$1@ger.gmane.org>
Message-ID: <94bdd2610911161050q63f708ecgce55cec70e30bddc@mail.gmail.com>

On Mon, Nov 16, 2009 at 7:08 PM, Tres Seaver <tseaver at palladion.com> wrote:
[..]
>>
>> That's historical and not true anymore. -> the current trunk uses tarfile, etc.
>
> I didn't know that you had fixed it on the trunk. AFAIK, it is still
> true that ?every released version of Python has this bug.
>

Well, it's not a bug. It's an enhancement. Python 2.4, 2.5 and 2.6
work as advertised:

http://docs.python.org/distutils/sourcedist.html

"...requires external utilities: tar and possibly one of gzip, bzip2,
or compress..."

That's why it wasn't backported to the 2.6 series.

Now there's no more need for such external programs:

http://docs.python.org/2.7/distutils/sourcedist.html

(except the "compress" program but this compression format will be abandoned and
 comes with a deprecation warning in 2.7)


Tarek

From chris at simplistix.co.uk  Tue Nov 17 12:30:13 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 17 Nov 2009 11:30:13 +0000
Subject: [Distutils] PEP 345 update + RFC on "Requires-External"
	and	"Requires-Python"
In-Reply-To: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
Message-ID: <4B028945.3060908@simplistix.co.uk>

Tarek Ziad? wrote:
> Last, "Requires-Python" is introduced to define the version of Python.
> I am not sure this is required anymore since Martin has added a Trove
> classifier
> for this. 

I don't think Trove classifiers have anything to do with this.
Otherwise, we could move *all* the requirements stuff into trove 
classification which buys us nothing more than having to maintain a huge 
list of trove classifiers.

I like the idea of Requires-Python, but only because I don't see why 
Python can't just be specified in the equivalent of setuptools' 
install_requires and its ilk.

Why can't python just be treated as another package on while a package 
my have a particular version dependency?

Chris


From chris at simplistix.co.uk  Tue Nov 17 12:32:33 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 17 Nov 2009 11:32:33 +0000
Subject: [Distutils] PEP 345 update + RFC on
	"Requires-External"	and	"Requires-Python"
In-Reply-To: <20091116162437.GO2966@piotro.eu>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	<4B012FB0.30407@egenix.com>
	<hdroah$dke$1@ger.gmane.org> <20091116162437.GO2966@piotro.eu>
Message-ID: <4B0289D1.8000307@simplistix.co.uk>

Piotr Ozarowski wrote:
> [Tres Seaver, 2009-11-16]
>>>> Values could be things like "libxslt", "libpng", etc..
>>> Am I right in understanding this as informational field only ?
>> Yes.  The primary consumers will be OS packagers, who will map them onto
>> their own package names.  As with other metadata they care about, we
>> hope that the packagers will suggest packages to make these names
>> "uniform" acrrss Python distributions.  I imagine there will be some
>> jockeying among them to find the "common" name for such things, which
>> they will then need to map.
> 
> How about putting there what we (distro maintainers) can pass to
> ctypes.util.find_library()?

How cross platform is that?

Chris

From mal at egenix.com  Tue Nov 17 12:44:50 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 17 Nov 2009 12:44:50 +0100
Subject: [Distutils] PEP 345 update + RFC on
	"Requires-External"	and	"Requires-Python"
In-Reply-To: <4B028945.3060908@simplistix.co.uk>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B028945.3060908@simplistix.co.uk>
Message-ID: <4B028CB2.8010604@egenix.com>

Chris Withers wrote:
> Tarek Ziad? wrote:
>> Last, "Requires-Python" is introduced to define the version of Python.
>> I am not sure this is required anymore since Martin has added a Trove
>> classifier
>> for this. 
> 
> I don't think Trove classifiers have anything to do with this.
> Otherwise, we could move *all* the requirements stuff into trove
> classification which buys us nothing more than having to maintain a huge
> list of trove classifiers.
> 
> I like the idea of Requires-Python, but only because I don't see why
> Python can't just be specified in the equivalent of setuptools'
> install_requires and its ilk.
> 
> Why can't python just be treated as another package on while a package
> my have a particular version dependency?

Perhaps because a package manager cannot go and resolve that
dependency by installing another Python version for you ? ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 17 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Tue Nov 17 12:45:03 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 12:45:03 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <4B028945.3060908@simplistix.co.uk>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B028945.3060908@simplistix.co.uk>
Message-ID: <94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>

On Tue, Nov 17, 2009 at 12:30 PM, Chris Withers <chris at simplistix.co.uk> wrote:
[..]
> I like the idea of Requires-Python, but only because I don't see why Python
> can't just be specified in the equivalent of setuptools' install_requires
> and its ilk.
>
> Why can't python just be treated as another package on while a package my
> have a particular version dependency?

But it turns it into a chicken and egg problem:  what happens if your
add "python>3" in "Requires-Dist" (the equivalent of install_requires)
?

Pip or easy_install is going to look for Python 3, and install it ?
then install itself in Python 3 and relaunch itself ?

I think Python is not to be treated as a package dependency, but as a
specific pre-request that has to be met for your application to run.
That's why it has to be specified separately from Requires-Dist.

Tarek

From chris at simplistix.co.uk  Tue Nov 17 12:49:51 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 17 Nov 2009 11:49:51 +0000
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	
	<4B028945.3060908@simplistix.co.uk>
	<94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>
Message-ID: <4B028DDF.6080808@simplistix.co.uk>

Tarek Ziad? wrote:
> On Tue, Nov 17, 2009 at 12:30 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> [..]
>> I like the idea of Requires-Python, but only because I don't see why Python
>> can't just be specified in the equivalent of setuptools' install_requires
>> and its ilk.
>>
>> Why can't python just be treated as another package on while a package my
>> have a particular version dependency?
> 
> But it turns it into a chicken and egg problem:  what happens if your
> add "python>3" in "Requires-Dist" (the equivalent of install_requires)
> ?

True. Oh well. Requires-Python it is then...

> Pip or easy_install is going to look for Python 3, and install it ?
> then install itself in Python 3 and relaunch itself ?

That's what buildout does is you specify a version of itself ;-)

Chris

From cournape at gmail.com  Tue Nov 17 13:56:41 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 17 Nov 2009 21:56:41 +0900
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <4B0289D1.8000307@simplistix.co.uk>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>
	<20091116162437.GO2966@piotro.eu> <4B0289D1.8000307@simplistix.co.uk>
Message-ID: <5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>

On Tue, Nov 17, 2009 at 8:32 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> Piotr Ozarowski wrote:
>>
>> [Tres Seaver, 2009-11-16]
>>>>>
>>>>> Values could be things like "libxslt", "libpng", etc..
>>>>
>>>> Am I right in understanding this as informational field only ?
>>>
>>> Yes. ?The primary consumers will be OS packagers, who will map them onto
>>> their own package names. ?As with other metadata they care about, we
>>> hope that the packagers will suggest packages to make these names
>>> "uniform" acrrss Python distributions. ?I imagine there will be some
>>> jockeying among them to find the "common" name for such things, which
>>> they will then need to map.
>>
>> How about putting there what we (distro maintainers) can pass to
>> ctypes.util.find_library()?
>
> How cross platform is that?

near none. I don't see much point in a field for this feature: that's
a typical example where every os vendor is different, and it would
bring more confusion that it worth IMHO.

If there is a need for this feature within tools like buildout or pip,
I think having a wrapped, fake python package for each library works
better (and is actually portable). For helping OS integrators, I think
a readme for packagers is as helpful.

cheers,

David

From mal at egenix.com  Tue Nov 17 14:04:40 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 17 Nov 2009 14:04:40 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External"
	and	"Requires-Python"
In-Reply-To: <20091116182222.GA29345@laurie.devork>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	<4B012FB0.30407@egenix.com>
	<20091116182222.GA29345@laurie.devork>
Message-ID: <4B029F68.6000802@egenix.com>

Floris Bruynooghe wrote:
> On Mon, Nov 16, 2009 at 11:55:44AM +0100, M.-A. Lemburg wrote:
>> Tarek Ziad? wrote:
>> One detail to add:
>>
>> python_full_version = sys.version.split()[0]
>>
>> This is important, since a package may well only support Python
>> 2.5 starting with patch level release 2.5.2 and python_version
>> only includes the major.minor version information.
> 
> I still like hex(sys.hexversion) for this.  IIRC this is what the docs
> recommend to use inside python code too for checking versions.

Fine with me, but that format doesn't work too well with the
(new) standard for distutils version numbers which is supposed
to be used for these version comparisons.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 17 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From wentland at cl.uni-heidelberg.de  Tue Nov 17 14:20:44 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Tue, 17 Nov 2009 14:20:44 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <4B028DDF.6080808@simplistix.co.uk>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B028945.3060908@simplistix.co.uk>
	<94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>
	<4B028DDF.6080808@simplistix.co.uk>
Message-ID: <20091117132044.GK21154@kinakuta.local>

On Tue, Nov 17, 2009 at 11:49 +0000, Chris Withers wrote:

> True. Oh well. Requires-Python it is then...

Can anyone add that proposal to the wiki so it won't be forgotten?
Furthermore it can be seen by people that try to get an overview of the
changes that might be integrated.

http://wiki.python.org/moin/Distutils/DiscussionOverview

TIA!
-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091117/ed990aaf/attachment.pgp>

From ziade.tarek at gmail.com  Tue Nov 17 14:20:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 14:20:44 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <20091117132044.GK21154@kinakuta.local>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B028945.3060908@simplistix.co.uk>
	<94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>
	<4B028DDF.6080808@simplistix.co.uk>
	<20091117132044.GK21154@kinakuta.local>
Message-ID: <94bdd2610911170520k38f3ff7bs786653e34fb117f1@mail.gmail.com>

On Tue, Nov 17, 2009 at 2:20 PM, Wolodja Wentland
<wentland at cl.uni-heidelberg.de> wrote:
> On Tue, Nov 17, 2009 at 11:49 +0000, Chris Withers wrote:
>
>> True. Oh well. Requires-Python it is then...
>
> Can anyone add that proposal to the wiki so it won't be forgotten?
> Furthermore it can be seen by people that try to get an overview of the
> changes that might be integrated.
>
> http://wiki.python.org/moin/Distutils/DiscussionOverview

It's already in PEP 345

From wentland at cl.uni-heidelberg.de  Tue Nov 17 14:25:50 2009
From: wentland at cl.uni-heidelberg.de (Wolodja Wentland)
Date: Tue, 17 Nov 2009 14:25:50 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <94bdd2610911170520k38f3ff7bs786653e34fb117f1@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B028945.3060908@simplistix.co.uk>
	<94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>
	<4B028DDF.6080808@simplistix.co.uk>
	<20091117132044.GK21154@kinakuta.local>
	<94bdd2610911170520k38f3ff7bs786653e34fb117f1@mail.gmail.com>
Message-ID: <20091117132550.GL21154@kinakuta.local>

On Tue, Nov 17, 2009 at 14:20 +0100, Tarek Ziad? wrote:
> On Tue, Nov 17, 2009 at 2:20 PM, Wolodja Wentland
> It's already in PEP 345

Even better :-D

-- 
  .''`.     Wolodja Wentland    <wentland at cl.uni-heidelberg.de> 
 : :'  :    
 `. `'`     4096R/CAF14EFC 
   `-       081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091117/5928ee7e/attachment.pgp>

From ziade.tarek at gmail.com  Tue Nov 17 14:29:52 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 14:29:52 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <20091117132550.GL21154@kinakuta.local>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B028945.3060908@simplistix.co.uk>
	<94bdd2610911170345o3b10f3a6v38f100cfbede837f@mail.gmail.com>
	<4B028DDF.6080808@simplistix.co.uk>
	<20091117132044.GK21154@kinakuta.local>
	<94bdd2610911170520k38f3ff7bs786653e34fb117f1@mail.gmail.com>
	<20091117132550.GL21154@kinakuta.local>
Message-ID: <94bdd2610911170529s167e821fy8cf1880bac38edcf@mail.gmail.com>

On Tue, Nov 17, 2009 at 2:25 PM, Wolodja Wentland
<wentland at cl.uni-heidelberg.de> wrote:
> On Tue, Nov 17, 2009 at 14:20 +0100, Tarek Ziad? wrote:
>> On Tue, Nov 17, 2009 at 2:20 PM, Wolodja Wentland
>> It's already in PEP 345
>
> Even better :-D
>

Although, the wiki page you set up is great to gather the ideas being discussed,
so if Requires-Python as it is now in the PEP 345 turns out to be controversial,
we can write down the various options in the wiki

Tarek

From ziade.tarek at gmail.com  Tue Nov 17 14:51:39 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 14:51:39 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
Message-ID: <94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>

On Sun, Nov 15, 2009 at 7:18 AM, Kevin Teague <kevin at bud.ca> wrote:
[..]
> Erm, actually, re-reading the Requires Python section in PEP 345 I guess the
> conditional operator is required, so it would actually be:
>
> Requires Python: ==2.4, ==2.5, ==2.6
>
> Is that right? Perhaps in the absence of a conditional operator, then ==
> should be the default, since I think it'd be easy to misunderstand this
> field upon casual usage.

That's a multi-line field, so at the end, Distutils will return a list
with the value of all lines.

For "==" I guess we can specify in the field that it's the default
operator, I'll add that.


> But then it should also make explicit in the PEP
> that ==2.5 is taken to mean only 2.5.0 or the entire sereis of releases in
> the 2.5 line.

I think that ==2.5 should target the entire 2.5.x series.


> But then arguably there aren't even enough special cases to justify the
> extra syntax to support things such as ">= 2.5.2"? I would imagine that
> almost all packages, when they release just test against the latest releases
> of whatever versions of Python that project supports? Also for the minor
> releases the other Python implementations, they aren't lock step with the
> CPython minor releases are they? e.g. Jython has a 2.5.0 release and 2.5.1
> release, but that 2.5.1 isn't the same as CPython 2.5.1 (I could be wrong
> about this, haven't looked in depth).

Answering to Frank here too:

We can suppose that a program that works with Jython, can safely use jython
versions in Requires-Python,

unless it's possible for a given program to run on CPython *and* Jython
and that it's impossible to use a common MAJOR.MINOR version ?

>
> Finally, it actually only makes sense to tag specific distributions for
> which versions of Python they support. So for example the Grok project:
>
> Grok 1.0:
> ?Requires Python: 2.4, 2.5
>
> Grok 1.1:
> ?Requires Python: 2.5, 2.6
>
> Which means that they PyPI would say "Grok supports Python 2.5 and Python
> 2.6" after 1.1 is released. Which might not be true if there was still
> maintenance releases happening in the 1.0.x line. So they what should the
> project specify here?
>
> So maybe I'm -0 on this field, since it does seems ambigous or the
> information is specific to the distributions and not the overall project.

I am not sure why you say there's an ambiguity, because PyPI doesn't
give the metadata
of a project, but of a distribution. (and as a matter of fact, some
PyPI UIs will automatically
use the latest distribution of the project, but that's just a default behavior)

So the "Grok supports Python 2.5 and Python 2.6" sentence doesn't
exists anywhere.
It's "Grok 1.1 supports Python 2.5 and Python 2.6"
or
"Grok LATEST_VERSION supports Python 2.5 and Python 2.6"

Tarek

From ziade.tarek at gmail.com  Tue Nov 17 14:58:08 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 14:58:08 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <4B029F68.6000802@egenix.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <20091116182222.GA29345@laurie.devork>
	<4B029F68.6000802@egenix.com>
Message-ID: <94bdd2610911170558h3da65ee2l759e48b6160c0757@mail.gmail.com>

On Tue, Nov 17, 2009 at 2:04 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Floris Bruynooghe wrote:
>> On Mon, Nov 16, 2009 at 11:55:44AM +0100, M.-A. Lemburg wrote:
>>> Tarek Ziad? wrote:
>>> One detail to add:
>>>
>>> python_full_version = sys.version.split()[0]
>>>
>>> This is important, since a package may well only support Python
>>> 2.5 starting with patch level release 2.5.2 and python_version
>>> only includes the major.minor version information.
>>
>> I still like hex(sys.hexversion) for this. ?IIRC this is what the docs
>> recommend to use inside python code too for checking versions.
>
> Fine with me, but that format doesn't work too well with the
> (new) standard for distutils version numbers which is supposed
> to be used for these version comparisons.

+1

Using sys.hexversion would bypass the PEP 386-compatible standard
we want to provide.

(I am adding python_full_version in the PEP btw)

Tarek

From ziade.tarek at gmail.com  Tue Nov 17 15:08:22 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 15:08:22 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>
	<20091116162437.GO2966@piotro.eu> <4B0289D1.8000307@simplistix.co.uk>
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
Message-ID: <94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>

On Tue, Nov 17, 2009 at 1:56 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Tue, Nov 17, 2009 at 8:32 PM, Chris Withers <chris at simplistix.co.uk> wrote:
>> Piotr Ozarowski wrote:
>>>
>>> [Tres Seaver, 2009-11-16]
>>>>>>
>>>>>> Values could be things like "libxslt", "libpng", etc..
>>>>>
>>>>> Am I right in understanding this as informational field only ?
>>>>
>>>> Yes. ?The primary consumers will be OS packagers, who will map them onto
>>>> their own package names. ?As with other metadata they care about, we
>>>> hope that the packagers will suggest packages to make these names
>>>> "uniform" acrrss Python distributions. ?I imagine there will be some
>>>> jockeying among them to find the "common" name for such things, which
>>>> they will then need to map.
>>>
>>> How about putting there what we (distro maintainers) can pass to
>>> ctypes.util.find_library()?
>>
>> How cross platform is that?
>
> near none. I don't see much point in a field for this feature: that's
> a typical example where every os vendor is different, and it would
> bring more confusion that it worth IMHO.

I agree.

And I can think about a few scenarii where the names managment at PyPI
will be a nightmare.

The PyPI manager will have to decide unilaterally the names to keep,
the names to change,  and he will need to maintain aliases for names
that points to the same lib, to make both camps happy,
and possibly keep backward compatibilities for those names.

I am proposing to remove the Requires-External completely from PEP
345, unless someone wants it badly. (And he/she will need to go into
greater details in these scenarii)

Tarek

From kevin at bud.ca  Tue Nov 17 17:20:23 2009
From: kevin at bud.ca (Kevin Teague)
Date: Tue, 17 Nov 2009 08:20:23 -0800
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
	<94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
Message-ID: <10AB82CC-AB1A-4DF4-9049-39F1CA648479@bud.ca>

>>
>> So maybe I'm -0 on this field, since it does seems ambigous or the
>> information is specific to the distributions and not the overall  
>> project.
>
> I am not sure why you say there's an ambiguity, because PyPI doesn't
> give the metadata
> of a project, but of a distribution. (and as a matter of fact, some
> PyPI UIs will automatically
> use the latest distribution of the project, but that's just a  
> default behavior)
>

Erm, yeah ... ignore that last bit I wrote, it's just flat out wrong.  
I think I said it because I quit drinking coffee and it's made my head  
foggy :P

(although this idea probably came from PyPI's default behaviour, is  
there a way to list all distributions for a project on PyPI? I guess  
that's a question for catalog-sig)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091117/5dbf6285/attachment.htm>

From ziade.tarek at gmail.com  Tue Nov 17 17:27:50 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 17:27:50 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <10AB82CC-AB1A-4DF4-9049-39F1CA648479@bud.ca>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
	<94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
	<10AB82CC-AB1A-4DF4-9049-39F1CA648479@bud.ca>
Message-ID: <94bdd2610911170827y7c1768d7t5911d37f24d3b29c@mail.gmail.com>

On Tue, Nov 17, 2009 at 5:20 PM, Kevin Teague <kevin at bud.ca> wrote:
[...]
> (although this idea probably came from PyPI's default behaviour, is there a
> way to list all distributions for a project on PyPI? I guess that's a
> question for catalog-sig)

The XML-RPC API offers this through "package_releases", see:

http://wiki.python.org/moin/PyPiXmlRpc

From tseaver at palladion.com  Tue Nov 17 17:44:26 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 17 Nov 2009 11:44:26 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	<4B012FB0.30407@egenix.com>
	<hdroah$dke$1@ger.gmane.org>	<20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
Message-ID: <hdujta$4in$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarek Ziad? wrote:

>> near none. I don't see much point in a field for this feature: that's
>> a typical example where every os vendor is different, and it would
>> bring more confusion that it worth IMHO.
> 
> I agree.
> 
> And I can think about a few scenarii where the names managment at PyPI
> will be a nightmare.
> 
> The PyPI manager will have to decide unilaterally the names to keep,
> the names to change,  and he will need to maintain aliases for names
> that points to the same lib, to make both camps happy,
> and possibly keep backward compatibilities for those names.
> 
> I am proposing to remove the Requires-External completely from PEP
> 345, unless someone wants it badly. (And he/she will need to go into
> greater details in these scenarii)

- -1. I don't see a problem keeping it:  it is "advisory" only.  PyPI is
not a consumer of this field, and should *not* be doing any enforcement
/ normalization / mapping of its values.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksC0uUACgkQ+gerLs4ltQ55JQCbBDH7gyWpWvw4+Lp+43GYV0QN
qwwAmgICi+HzQCj9lt44e1lRY143Dj/U
=EN8s
-----END PGP SIGNATURE-----


From cool-rr at cool-rr.com  Tue Nov 17 17:50:07 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Tue, 17 Nov 2009 16:50:07 +0000 (UTC)
Subject: [Distutils] Please help me use PyPI
Message-ID: <loom.20091117T174729-113@post.gmane.org>

Hi all,

I'm trying to release my project to PyPI today.

I have three problems:

(1.) I have 4 different forks for my project for Python versions 2.4, 2.5, 2.6 
and 3.1. Should they all be on the same name in PyPI?

(2.) I'm trying to upload an MSI. I'm doing `setup.py bdist_msi register 
upload`. It builds the project and the setup script finishes, but then I get:

    error: garlicsim-0.1: No such file or directory

And no .msi file is uploaded to PyPi. Why?

(3.) I tried uploading my 2.5 fork. It asks me to identify, I do, it gives an OK 
response (which doesn't happen unless the identification is right), but then it 
claims I didn't identify! Why?


    [...]
    removing 'build\bdist.win32\egg' (and everything under it)
    running register
    We need to know who you are, so please choose either:
     1. use your existing login,
     2. register as a new user,
     3. have the server generate a new password for you (and email it to you), 
or
     4. quit
    Your selection [default 1]:  1
    Username: coolRR
    Password:
    Server response (200): OK
    running upload
    Submitting dist\garlicsim-0.1.zip to http://pypi.python.org/pypi
    Upload failed (401): You must be identified to edit package information
    removing 'build' (and everything under it)
    error: garlicsim-0.1: No such file or directory



Thanks,
Ram.


From chris at simplistix.co.uk  Tue Nov 17 18:27:51 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 17 Nov 2009 17:27:51 +0000
Subject: [Distutils] Please help me use PyPI
In-Reply-To: <loom.20091117T174729-113@post.gmane.org>
References: <loom.20091117T174729-113@post.gmane.org>
Message-ID: <4B02DD17.5010408@simplistix.co.uk>

Ram Rachum wrote:
> (1.) I have 4 different forks for my project for Python versions 2.4, 2.5, 2.6 
> and 3.1. Should they all be on the same name in PyPI?

Yes.

> (2.) I'm trying to upload an MSI. I'm doing `setup.py bdist_msi register 
> upload`. It builds the project and the setup script finishes, but then I get:
> 
>     error: garlicsim-0.1: No such file or directory
> 
> And no .msi file is uploaded to PyPi. Why?

It doesn't look like your project built successfully.
Just do setup.py bdist_msi. When you find a .msi in your build 
directory, then try the upload command.

> (3.) I tried uploading my 2.5 fork. It asks me to identify, I do, it gives an OK 
> response (which doesn't happen unless the identification is right), but then it 
> claims I didn't identify! Why?
> 
> 
>     [...]
>     removing 'build\bdist.win32\egg' (and everything under it)
>     running register
>     We need to know who you are, so please choose either:
>      1. use your existing login,
>      2. register as a new user,
>      3. have the server generate a new password for you (and email it to you), 
> or
>      4. quit
>     Your selection [default 1]:  1
>     Username: coolRR
>     Password:
>     Server response (200): OK
>     running upload
>     Submitting dist\garlicsim-0.1.zip to http://pypi.python.org/pypi
>     Upload failed (401): You must be identified to edit package information

Dunno on this one. Have you since solved this?
I see your .zip there:
http://pypi.python.org/pypi/garlicsim

cheers,

Chris

PS: catalog-sig is arguably the correct list for your problems...

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From cool-rr at cool-rr.com  Tue Nov 17 18:33:10 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Tue, 17 Nov 2009 17:33:10 +0000 (UTC)
Subject: [Distutils] Please help me use PyPI
References: <loom.20091117T174729-113@post.gmane.org>
	<4B02DD17.5010408@simplistix.co.uk>
Message-ID: <loom.20091117T182826-778@post.gmane.org>

Comments inline.

Chris Withers <chris <at> simplistix.co.uk> writes:

> 
> Ram Rachum wrote:
> > (1.) I have 4 different forks for my project for Python versions 2.4, 2.5, 
2.6 
> > and 3.1. Should they all be on the same name in PyPI?
> 
> Yes.
> 

Sounds good. But I am confused. I see in my PyPI page here:

http://pypi.python.org/pypi/garlicsim/0.1

That I have a source distribution "garlicsim-0.1.zip". How can anyone know 
whether it's the source distribution for 2.5 or 2.6?


> > (2.) I'm trying to upload an MSI. I'm doing `setup.py bdist_msi register 
> > upload`. It builds the project and the setup script finishes, but then I 
get:
> > 
> >     error: garlicsim-0.1: No such file or directory
> > 
> > And no .msi file is uploaded to PyPi. Why?
> 
> It doesn't look like your project built successfully.
> Just do setup.py bdist_msi. When you find a .msi in your build 
> directory, then try the upload command.
> 

I'm not getting an .msi file in my build directory, but I do get a "garlicsim-
0.1.win32-py2.6.msi" in my dist directory. What's going on?

> > (3.) I tried uploading my 2.5 fork. It asks me to identify, I do, it gives 
an OK 
> > response (which doesn't happen unless the identification is right), but then 
it 
> > claims I didn't identify! Why?
> > 
> > 
> >     [...]
> >     removing 'build\bdist.win32\egg' (and everything under it)
> >     running register
> >     We need to know who you are, so please choose either:
> >      1. use your existing login,
> >      2. register as a new user,
> >      3. have the server generate a new password for you (and email it to 
you), 
> > or
> >      4. quit
> >     Your selection [default 1]:  1
> >     Username: coolRR
> >     Password:
> >     Server response (200): OK
> >     running upload
> >     Submitting dist\garlicsim-0.1.zip to http://pypi.python.org/pypi
> >     Upload failed (401): You must be identified to edit package information
> 
> Dunno on this one. Have you since solved this?
> I see your .zip there:
> http://pypi.python.org/pypi/garlicsim
> 

I haven't, this problems happens on the 2.5 fork. (Haven't tried the others 
yet.)

> cheers,
> 
> Chris
> 
> PS: catalog-sig is arguably the correct list for your problems...
> 





From ziade.tarek at gmail.com  Tue Nov 17 18:53:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 18:53:11 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <hdujta$4in$1@ger.gmane.org>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>
	<20091116162437.GO2966@piotro.eu> <4B0289D1.8000307@simplistix.co.uk>
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
	<hdujta$4in$1@ger.gmane.org>
Message-ID: <94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>

On Tue, Nov 17, 2009 at 5:44 PM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Tarek Ziad? wrote:
>
>>> near none. I don't see much point in a field for this feature: that's
>>> a typical example where every os vendor is different, and it would
>>> bring more confusion that it worth IMHO.
>>
>> I agree.
>>
>> And I can think about a few scenarii where the names managment at PyPI
>> will be a nightmare.
>>
>> The PyPI manager will have to decide unilaterally the names to keep,
>> the names to change, ?and he will need to maintain aliases for names
>> that points to the same lib, to make both camps happy,
>> and possibly keep backward compatibilities for those names.
>>
>> I am proposing to remove the Requires-External completely from PEP
>> 345, unless someone wants it badly. (And he/she will need to go into
>> greater details in these scenarii)
>
> - -1. I don't see a problem keeping it: ?it is "advisory" only. ?PyPI is
> not a consumer of this field, and should *not* be doing any enforcement
> / normalization / mapping of its values.

In that case, I am not sure to see what's the point of keeping an
external reference registery at PyPI,
as a free place where anyone can put anything. The only gain I was
seeing it normalization.

If no normalization is done, then a plain README.txt file with this
info in the project itself is sufficient,  or maybe making the field
hold these (name, description, url) info directly.

Tarek

From floris.bruynooghe at gmail.com  Tue Nov 17 18:59:35 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Tue, 17 Nov 2009 17:59:35 +0000
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <hdujta$4in$1@ger.gmane.org>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>
	<20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
	<hdujta$4in$1@ger.gmane.org>
Message-ID: <20091117175935.GA15496@laurie.devork>

On Tue, Nov 17, 2009 at 11:44:26AM -0500, Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Tarek Ziad? wrote:
> 
> >> near none. I don't see much point in a field for this feature: that's
> >> a typical example where every os vendor is different, and it would
> >> bring more confusion that it worth IMHO.
> > 
> > I agree.
> > 
> > And I can think about a few scenarii where the names managment at PyPI
> > will be a nightmare.
> > 
> > The PyPI manager will have to decide unilaterally the names to keep,
> > the names to change,  and he will need to maintain aliases for names
> > that points to the same lib, to make both camps happy,
> > and possibly keep backward compatibilities for those names.
> > 
> > I am proposing to remove the Requires-External completely from PEP
> > 345, unless someone wants it badly. (And he/she will need to go into
> > greater details in these scenarii)
> 
> - -1. I don't see a problem keeping it:  it is "advisory" only.  PyPI is
> not a consumer of this field, and should *not* be doing any enforcement
> / normalization / mapping of its values.

IMHO the only sane way of keeping the field is regarding it as a
"readme" and hence just make it completely free form text to be
interpreted by humans only.

I'm +0 on removing it since the long description can probably describe
this easier.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From fwierzbicki at gmail.com  Tue Nov 17 19:59:25 2009
From: fwierzbicki at gmail.com (Frank Wierzbicki)
Date: Tue, 17 Nov 2009 13:59:25 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
	<94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
Message-ID: <4dab5f760911171059o3a923c7cx1a7563e2139e639d@mail.gmail.com>

On Tue, Nov 17, 2009 at 8:51 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Answering to Frank here too:
>
> We can suppose that a program that works with Jython, can safely use jython
> versions in Requires-Python,
>
> unless it's possible for a given program to run on CPython *and* Jython
> and that it's impossible to use a common MAJOR.MINOR version ?
I think it will be very rare to find a program that works on both but
only on different MAJOR.MINOR versions. It is *possible* - for
example, we have an ast.py in Jython 2.5 even though CPython does not
get this until 2.6 (ast.py as a replacement for pyc bytecode
manipulation was just too compelling for us to ignore).

-Frank

From tseaver at palladion.com  Tue Nov 17 20:51:23 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 17 Nov 2009 14:51:23 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	<4B012FB0.30407@egenix.com>
	<hdroah$dke$1@ger.gmane.org>	<20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>	<hdujta$4in$1@ger.gmane.org>
	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
Message-ID: <hduurr$ds3$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarek Ziad? wrote:
> On Tue, Nov 17, 2009 at 5:44 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Tarek Ziad? wrote:
>>
>>>> near none. I don't see much point in a field for this feature: that's
>>>> a typical example where every os vendor is different, and it would
>>>> bring more confusion that it worth IMHO.
>>> I agree.
>>>
>>> And I can think about a few scenarii where the names managment at PyPI
>>> will be a nightmare.
>>>
>>> The PyPI manager will have to decide unilaterally the names to keep,
>>> the names to change,  and he will need to maintain aliases for names
>>> that points to the same lib, to make both camps happy,
>>> and possibly keep backward compatibilities for those names.
>>>
>>> I am proposing to remove the Requires-External completely from PEP
>>> 345, unless someone wants it badly. (And he/she will need to go into
>>> greater details in these scenarii)
>> - -1. I don't see a problem keeping it:  it is "advisory" only.  PyPI is
>> not a consumer of this field, and should *not* be doing any enforcement
>> / normalization / mapping of its values.
> 
> In that case, I am not sure to see what's the point of keeping an
> external reference registery at PyPI,
> as a free place where anyone can put anything. The only gain I was
> seeing it normalization.
> 
> If no normalization is done, then a plain README.txt file with this
> info in the project itself is sufficient,  or maybe making the field
> hold these (name, description, url) info directly.

This isn't *about* PyPI:  it is about putting the information in a
standard place for downstream packagers (*they* requested it).  PyPI
should either ignore the field or just pass it through untouched:
downstream packagers are going to examine the PKG_INFO file in the
tarball they work with, and might write scripts which do the mapping of
the machine-readable file to their own dependency names.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksC/rsACgkQ+gerLs4ltQ4+rgCgiyIVoFCReG8KNE/BixtxCWH/
gOAAnjN8v8I4rS7MUiok3Xb64Lq89dW9
=OIBJ
-----END PGP SIGNATURE-----


From tseaver at palladion.com  Tue Nov 17 20:31:58 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 17 Nov 2009 14:31:58 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>	
	<20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>	
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>	
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>	
	<hdujta$4in$1@ger.gmane.org>
	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
Message-ID: <4B02FA2E.8030101@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarek Ziad? wrote:
> On Tue, Nov 17, 2009 at 5:44 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Tarek Ziad? wrote:
>>
>>>> near none. I don't see much point in a field for this feature: that's
>>>> a typical example where every os vendor is different, and it would
>>>> bring more confusion that it worth IMHO.
>>> I agree.
>>>
>>> And I can think about a few scenarii where the names managment at PyPI
>>> will be a nightmare.
>>>
>>> The PyPI manager will have to decide unilaterally the names to keep,
>>> the names to change,  and he will need to maintain aliases for names
>>> that points to the same lib, to make both camps happy,
>>> and possibly keep backward compatibilities for those names.
>>>
>>> I am proposing to remove the Requires-External completely from PEP
>>> 345, unless someone wants it badly. (And he/she will need to go into
>>> greater details in these scenarii)
>> - -1. I don't see a problem keeping it:  it is "advisory" only.  PyPI is
>> not a consumer of this field, and should *not* be doing any enforcement
>> / normalization / mapping of its values.
> 
> In that case, I am not sure to see what's the point of keeping an
> external reference registery at PyPI,
> as a free place where anyone can put anything. The only gain I was
> seeing it normalization.
> 
> If no normalization is done, then a plain README.txt file with this
> info in the project itself is sufficient,  or maybe making the field
> hold these (name, description, url) info directly.

This isn't *about* PyPI:  it is about putting the information in a
standard place for downstream packagers (*they* requested it).  PyPI
should either ignore the field or just pass it through untouched:
downstream packagers are going to examine the PKG_INFO file in the
tarball they work with, and might write scripts which do the mapping of
the machine-readable file to their own dependency names.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksC+ikACgkQ+gerLs4ltQ6C6wCcCjgmVQAh//TyqcKSMtfboZzg
TKMAoLHMh/uWl6b8LzAGKx1i52Sm6TX2
=PfNw
-----END PGP SIGNATURE-----

From ziade.tarek at gmail.com  Tue Nov 17 21:40:25 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 21:40:25 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <4B02FA2E.8030101@palladion.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<4B012FB0.30407@egenix.com> <hdroah$dke$1@ger.gmane.org>
	<20091116162437.GO2966@piotro.eu> <4B0289D1.8000307@simplistix.co.uk>
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
	<hdujta$4in$1@ger.gmane.org>
	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
	<4B02FA2E.8030101@palladion.com>
Message-ID: <94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>

On Tue, Nov 17, 2009 at 8:31 PM, Tres Seaver <tseaver at palladion.com> wrote:
[..]
>> If no normalization is done, then a plain README.txt file with this
>> info in the project itself is sufficient, ?or maybe making the field
>> hold these (name, description, url) info directly.
>
> This isn't *about* PyPI: ?it is about putting the information in a
> standard place for downstream packagers (*they* requested it). ?PyPI
> should either ignore the field or just pass it through untouched:
> downstream packagers are going to examine the PKG_INFO file in the
> tarball they work with, and might write scripts which do the mapping of
> the machine-readable file to their own dependency names.

And again, that's why I don't see *any* benefit of having an external
registery at PyPI.

You say that PyPI should ignore it, but in the meantime, the "External
References Registry"
implies that we will manage and maintain, urls and descriptions for
each one of those Requires-External value.

How this is going to work ?

Why can't I put everything in my setup.py in the first place ? I have
to go to PyPI first, to add
an entry with an url and a description, then get back to my package to
add just the name ?

What happens if the name is taken, but I don't want the url and the
description someone already put there ?  I'll just go for another name
?

And the packagers will have to look to my PKG-INFO, *and* to the PyPI
registery ?

That registery thing at PyPI doesn't make sense as it is currently
described in the PEP because
it implies that the metadata that comes with a project distribution
are *incomplete* and that you
need to look into a registery in some website to complete them.

Although, having a structured way to describe external dependencies in
the metadata is
interesting.

That's why I said that 100% of the information could stay in PKG-INFO
(==the metadata),
instead of having to maintain something else at PyPI.

Developers can add their things in it, and packagers look at them.

So, that means that "Require-Dist" field would need the url and the
description, besides the name,
like I've suggested.

Tarek

From ziade.tarek at gmail.com  Tue Nov 17 21:41:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 21:41:44 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<hdroah$dke$1@ger.gmane.org> <20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
	<hdujta$4in$1@ger.gmane.org>
	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
	<4B02FA2E.8030101@palladion.com>
	<94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>
Message-ID: <94bdd2610911171241jccc3099s1678dd722d67379b@mail.gmail.com>

On Tue, Nov 17, 2009 at 9:40 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
[..]
> So, that means that "Require-Dist" field would need the url and the
> description, besides the name,

typo: Requires-Externals

From tseaver at palladion.com  Tue Nov 17 22:28:37 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 17 Nov 2009 16:28:37 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	<4B012FB0.30407@egenix.com>
	<hdroah$dke$1@ger.gmane.org>	<20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>	<hdujta$4in$1@ger.gmane.org>	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>	<4B02FA2E.8030101@palladion.com>
	<94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>
Message-ID: <4B031585.5040900@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarek Ziad? wrote:

> That's why I said that 100% of the information could stay in PKG-INFO
> (==the metadata),
> instead of having to maintain something else at PyPI.

Apoligies for misunderstanding:  I thought you were arguing to remove
the field from PKG-INFO.  I'm fine with removing any mandate for a
registry from the PEP.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksDFYEACgkQ+gerLs4ltQ7PdQCfep1dxMMWOFRECaFQ5rPHbvGc
pEYAniAFT21xe5s4cCr3e+2+BEFXix7J
=pffI
-----END PGP SIGNATURE-----


From tseaver at palladion.com  Tue Nov 17 23:01:50 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 17 Nov 2009 17:01:50 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911171353q316285dbu80435469c50c364f@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>	
	<20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>	
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>	
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>	
	<hdujta$4in$1@ger.gmane.org>	
	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>	
	<4B02FA2E.8030101@palladion.com>	
	<94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>	
	<4B031585.5040900@palladion.com>
	<94bdd2610911171353q316285dbu80435469c50c364f@mail.gmail.com>
Message-ID: <4B031D4E.6020901@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarek Ziad? wrote:
> On Tue, Nov 17, 2009 at 10:28 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Tarek Ziad? wrote:
>>
>>> That's why I said that 100% of the information could stay in PKG-INFO
>>> (==the metadata),
>>> instead of having to maintain something else at PyPI.
>> Apoligies for misunderstanding:  I thought you were arguing to remove
>> the field from PKG-INFO.  I'm fine with removing any mandate for a
>> registry from the PEP.
> 
> That's fine, probably related to my frenglish too.
> 
> There's one thing though: do we want the url and description data with
> the name ?

I would say we don't need anything.  This field is like the "data type"
stuff (not yet in a PEP):  we want the downstream folks from Fedora /
Debian to suggest values to the upstream (PyPI) package maintainer.
I think we can get "enough" traction from just having a well-understood
name ('zlib', 'libxslt', etc.) in this field.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksDHUkACgkQ+gerLs4ltQ5PbQCeOV980LniLELp1PROe8Vyyh1u
bEkAoMv135Mhk/V4w8u1Og++JJNbUr56
=v5Kr
-----END PGP SIGNATURE-----


From ziade.tarek at gmail.com  Tue Nov 17 23:40:46 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 23:40:46 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<hdroah$dke$1@ger.gmane.org> <20091116162437.GO2966@piotro.eu>
	<4B0289D1.8000307@simplistix.co.uk>
	<5b8d13220911170456s1f59a8e2mdfe13be344f10a6f@mail.gmail.com>
	<94bdd2610911170608q3e87b13wef308e800b8e78b5@mail.gmail.com>
	<hdujta$4in$1@ger.gmane.org>
	<94bdd2610911170953x27afbae0ud32b9b7e3ae2d98@mail.gmail.com>
	<4B02FA2E.8030101@palladion.com>
	<94bdd2610911171240r671e6c4epba295e959742c145@mail.gmail.com>
Message-ID: <94bdd2610911171440q3e53354wd1df094ccbd0728@mail.gmail.com>

On Tue, Nov 17, 2009 at 9:40 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
[..]
>
> That's why I said that 100% of the information could stay in PKG-INFO
> (==the metadata),
> instead of having to maintain something else at PyPI.

As discussed live with Tres, we agreed that the Requires-External
field is enough, as it is
defined now, and that we can remove the Registry stuff from PEP 345.

Tarek

From ziade.tarek at gmail.com  Tue Nov 17 23:47:16 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 17 Nov 2009 23:47:16 +0100
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
	"Requires-Python"
In-Reply-To: <4dab5f760911171059o3a923c7cx1a7563e2139e639d@mail.gmail.com>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
	<94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
	<4dab5f760911171059o3a923c7cx1a7563e2139e639d@mail.gmail.com>
Message-ID: <94bdd2610911171447p3abffd7ex3f8ccf91441a6995@mail.gmail.com>

On Tue, Nov 17, 2009 at 7:59 PM, Frank Wierzbicki <fwierzbicki at gmail.com> wrote:
> On Tue, Nov 17, 2009 at 8:51 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> Answering to Frank here too:
>>
>> We can suppose that a program that works with Jython, can safely use jython
>> versions in Requires-Python,
>>
>> unless it's possible for a given program to run on CPython *and* Jython
>> and that it's impossible to use a common MAJOR.MINOR version ?

> I think it will be very rare to find a program that works on both but
> only on different MAJOR.MINOR versions. It is *possible* - for
> example, we have an ast.py in Jython 2.5 even though CPython does not
> get this until 2.6 (ast.py as a replacement for pyc bytecode
> manipulation was just too compelling for us to ignore).
>

So do you think Requires-Python is good enough in general as it is
described now,
and that this case you mention can be handled by some code at install
time, raising an error in
case the conditions are not met ?


Tarek

From david.lyon at preisshare.net  Wed Nov 18 03:16:51 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Tue, 17 Nov 2009 21:16:51 -0500
Subject: [Distutils] PEP 345 update + RFC on "Requires-External" and
 "Requires-Python"
In-Reply-To: <10AB82CC-AB1A-4DF4-9049-39F1CA648479@bud.ca>
References: <94bdd2610911141609i6d46a785m1d5d07d6b0d532ed@mail.gmail.com>
	<7987BDB7-CC98-44CF-BCEF-6DB71D7FC92C@bud.ca>
	<94bdd2610911170551h98ce93yf258929ae47d10d7@mail.gmail.com>
	<10AB82CC-AB1A-4DF4-9049-39F1CA648479@bud.ca>
Message-ID: <a8113a6c57d685aaa80ad9374a8f0382@preisshare.net>

On Tue, 17 Nov 2009 08:20:23 -0800, Kevin Teague <kevin at bud.ca> wrote:

> (although this idea probably came from PyPI's default behaviour, is  
> there a way to list all distributions for a project on PyPI? 

Yeah of course, see:

  http://wiki.python.org/moin/PyPiXmlRpc

David

From setuptools at bugs.python.org  Wed Nov 18 19:24:22 2009
From: setuptools at bugs.python.org (David Hart)
Date: Wed, 18 Nov 2009 18:24:22 +0000
Subject: [Distutils] [issue97] Windows installer causes different behavior
In-Reply-To: <1258568661.99.0.701106398287.issue97@psf.upfronthosting.co.za>
Message-ID: <1258568661.99.0.701106398287.issue97@psf.upfronthosting.co.za>


New submission from David Hart <dbhart at sandia.gov>:

Using the windows .exe installer for setuptools0.6c11 causes an error that is
not present when using the .tar.gz and installing from source (python setup.py
install). easy_install.exe behaves differently between the two methods.

Using the .exe, you can no longer download the setuptools package (see attached)
- it finds it somewhere cached, and then tries to overwrite itself. Using the
.tar.gz build, it behaves correctly.

----------
messages: 467
nosy: dbhart
priority: bug
status: unread
title: Windows installer causes different behavior

_______________________________________________
Setuptools tracker <setuptools at bugs.python.org>
<http://bugs.python.org/setuptools/issue97>
_______________________________________________

From pj2192 at columbia.edu  Wed Nov 18 20:59:21 2009
From: pj2192 at columbia.edu (Pavol Juhas)
Date: Wed, 18 Nov 2009 14:59:21 -0500
Subject: [Distutils] obtaining easy_install prefix in setup.py
Message-ID: <20091118195921.GA3562@columbia.edu>

Hello,

I searched and read the recent discussions on the distutils-sig
list and I hope someone can help me with the following problem.

Is there a way to recover the prefix value inside pkgfoo/setup.py,
when it is installed as

    easy_install --prefix=value pkgfoo
    (pkgfoo is a directory with setup.py)

I can get the prefix, when pkgfoo is installed with

    python setup.py install --prefix=value

by overloading the bdist_egg class and passing it via
cmdclass argument to the setup function:

http://danse.us/trac/diffraction/browser/diffraction/diffpy.lib/trunk/setup.py
http://danse.us/trac/diffraction/browser/diffraction/diffpy.lib/trunk/setup_scons.py

However, this does not work under easy_install, because
easy_install already creates its own distribution object
before sourcing pkgfoo/setup.py, in addition setup.py
is executed in a sandbox with mangled sys.argv.

Has anybody, perhaps David Cournapeau, figured out how to wrestle
the prefix value out of easy_install?

Thank you all,

Pavol

PS: Let me give you some background for what I am trying to do.

    I am working on several Python packages that need to be distributed
    together with a shared C++ library (libdiffpy.so).  So far I
    have used scons to build and install the library and that has
    been working really well.  However, I would like to have a
    dependency resolution for any client packages as available in the
    setuptools.  After wading through the "People want CPAN" thread
    I realized that converting my scons scripts to distutils would be
    a hopeless task.  So my solution is to package the library sources
    in a phony python package diffpy.lib, where its setup.py is extended
    to execute scons that actually takes care of the library build
    and installation.  I have decided to ignore all setup.py options
    and take care of the prefix under which the library should be
    installed.  However to do so, I need to recover the prefix value
    and pass it to scons.

From pje at telecommunity.com  Wed Nov 18 22:16:19 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Wed, 18 Nov 2009 16:16:19 -0500
Subject: [Distutils] obtaining easy_install prefix in setup.py
In-Reply-To: <20091118195921.GA3562@columbia.edu>
References: <20091118195921.GA3562@columbia.edu>
Message-ID: <20091118211627.EB9563A407A@sparrow.telecommunity.com>

At 02:59 PM 11/18/2009 -0500, Pavol Juhas wrote:
>Hello,
>
>I searched and read the recent discussions on the distutils-sig
>list and I hope someone can help me with the following problem.
>
>Is there a way to recover the prefix value inside pkgfoo/setup.py,
>when it is installed as
>
>     easy_install --prefix=value pkgfoo
>     (pkgfoo is a directory with setup.py)

Short of walking the stack in some truly hideous fashion, no.  This 
is something that needs a configure step, and is unlikely to be 
supportable in setuptools until after distutils has its own configure command.


From david.lyon at preisshare.net  Thu Nov 19 01:45:34 2009
From: david.lyon at preisshare.net (David Lyon)
Date: Wed, 18 Nov 2009 19:45:34 -0500
Subject: [Distutils]
 =?utf-8?q?obtaining_easy=5Finstall_prefix_in_setup=2E?=
 =?utf-8?q?py?=
In-Reply-To: <20091118195921.GA3562@columbia.edu>
References: <20091118195921.GA3562@columbia.edu>
Message-ID: <eeff631dd646e791a3344f7705030848@preisshare.net>

On Wed, 18 Nov 2009 14:59:21 -0500, Pavol Juhas <pj2192 at columbia.edu>
wrote:
> Hello,
> 
> I searched and read the recent discussions on the distutils-sig
> list and I hope someone can help me with the following problem.
> 
> Is there a way to recover the prefix value inside pkgfoo/setup.py,
> when it is installed as
> 
>     easy_install --prefix=value pkgfoo
>     (pkgfoo is a directory with setup.py)
> 
> I can get the prefix, when pkgfoo is installed with
> 
>     python setup.py install --prefix=value
> 
> by overloading the bdist_egg class and passing it via
> cmdclass argument to the setup function:
> 
>
http://danse.us/trac/diffraction/browser/diffraction/diffpy.lib/trunk/setup.py
>
http://danse.us/trac/diffraction/browser/diffraction/diffpy.lib/trunk/setup_scons.py
> 
> However, this does not work under easy_install, because
> easy_install already creates its own distribution object
> before sourcing pkgfoo/setup.py, in addition setup.py
> is executed in a sandbox with mangled sys.argv.
> 
> Has anybody, perhaps David Cournapeau, figured out how to wrestle
> the prefix value out of easy_install?
> 
> Thank you all,

Hi Pavol,

Is this only for the linux/mac version? I tried the windows version
of diffpy and it uses a nsis installer and that all seemed to work fine
on my test machine.

So, you want to find the "full path" of the install directory for
post install processing? (using non-distutils installation terminology)

I'm sure somebody will correct me, but it is possible to get the
exact install directory from setuptools using the a workingset api
call.

But I've never tried integrating that into a setup call in setup.py
but I'm wondering if that is the sort of thing that you're after?

David







From ndbecker2 at gmail.com  Thu Nov 19 03:13:48 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Wed, 18 Nov 2009 21:13:48 -0500
Subject: [Distutils] distribute traceback
Message-ID: <he29ku$ehv$1@ger.gmane.org>

sudo easy_install -U openopt
Processing openopt
Running OOPy/setup.py -q bdist_egg --dist-dir 
/home/nbecker/openopt/OOPy/egg-dist-tmp-fTMbRS
Adding openopt 0.24.dev to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/openopt-0.24.dev-py2.6.egg
Processing dependencies for openopt==0.24.dev
Finished processing dependencies for openopt==0.24.dev
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib64/python2.6/multiprocessing/util.py", line 258, in 
_exit_function
    info('process shutting down')
TypeError: 'NoneType' object is not callable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib64/python2.6/multiprocessing/util.py", line 258, in 
_exit_function
    info('process shutting down')
TypeError: 'NoneType' object is not callable

At least, I think it's distribute (I just did update fedora 11->12, so maybe 
something got screwed up).  I did
easy_install -U distribute after the update.
sudo easy_install -U distribute
Searching for distribute                             
Reading http://pypi.python.org/simple/distribute/    
Reading http://packages.python.org/distribute        
Best match: distribute 0.6.8                         
Processing distribute-0.6.8-py2.6.egg                
distribute 0.6.8 is already the active version in easy-install.pth
Installing easy_install script to /usr/bin                        
Installing easy_install-2.6 script to /usr/bin                    

Using /usr/lib/python2.6/site-packages/distribute-0.6.8-py2.6.egg
Processing dependencies for distribute                           
Finished processing dependencies for distribute                  



From ziade.tarek at gmail.com  Thu Nov 19 06:57:08 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 19 Nov 2009 06:57:08 +0100
Subject: [Distutils] distribute traceback
In-Reply-To: <he29ku$ehv$1@ger.gmane.org>
References: <he29ku$ehv$1@ger.gmane.org>
Message-ID: <94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>

On Thu, Nov 19, 2009 at 3:13 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> sudo easy_install -U openopt
> Processing openopt
> Running OOPy/setup.py -q bdist_egg --dist-dir
> /home/nbecker/openopt/OOPy/egg-dist-tmp-fTMbRS
> Adding openopt 0.24.dev to easy-install.pth file
>
> Installed /usr/lib/python2.6/site-packages/openopt-0.24.dev-py2.6.egg
> Processing dependencies for openopt==0.24.dev
> Finished processing dependencies for openopt==0.24.dev
> Error in atexit._run_exitfuncs:
> Traceback (most recent call last):
> ?File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
> ? ?func(*targs, **kargs)
> ?File "/usr/lib64/python2.6/multiprocessing/util.py", line 258, in
> _exit_function
> ? ?info('process shutting down')
> TypeError: 'NoneType' object is not callable
> Error in sys.exitfunc:
> Traceback (most recent call last):
> ?File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
> ? ?func(*targs, **kargs)
> ?File "/usr/lib64/python2.6/multiprocessing/util.py", line 258, in
> _exit_function
> ? ?info('process shutting down')
> TypeError: 'NoneType' object is not callable
>
> At least, I think it's distribute (I just did update fedora 11->12, so maybe
> something got screwed up). ?I did
> easy_install -U distribute after the update.
> sudo easy_install -U distribute
> Searching for distribute
> Reading http://pypi.python.org/simple/distribute/
> Reading http://packages.python.org/distribute
> Best match: distribute 0.6.8
> Processing distribute-0.6.8-py2.6.egg
> distribute 0.6.8 is already the active version in easy-install.pth
> Installing easy_install script to /usr/bin
> Installing easy_install-2.6 script to /usr/bin
>
> Using /usr/lib/python2.6/site-packages/distribute-0.6.8-py2.6.egg
> Processing dependencies for distribute
> Finished processing dependencies for distribute
>


That's a known bug in multiprocessing apparently,

http://bugs.python.org/issue4106

I am not sure why it's triggered in Distribute, but it happens when
the garbage collector has staretd to remove some refs. Notice that the
work is over, and everything is installed.

I am not sure we can do something clean on Distribute side until it's
fixed in Python,

Can you reproduce this problem *everytime* ?

Regards,
Tarek

ps: I am putting Jesse in the loop, he might have an idea

From pj2192 at columbia.edu  Thu Nov 19 06:59:56 2009
From: pj2192 at columbia.edu (Pavol Juhas)
Date: Thu, 19 Nov 2009 00:59:56 -0500
Subject: [Distutils] obtaining easy_install prefix in setup.py
In-Reply-To: <eeff631dd646e791a3344f7705030848@preisshare.net>
References: <20091118195921.GA3562@columbia.edu>
	<eeff631dd646e791a3344f7705030848@preisshare.net>
Message-ID: <20091119055956.GA4437@columbia.edu>

On Wed, Nov 18, 2009 at 07:45:34PM -0500, David Lyon wrote:
...
> Is this only for the linux/mac version? I tried the windows version
> of diffpy and it uses a nsis installer and that all seemed to work fine
> on my test machine.

Hi David,

The shared C++ library is necessary for new packages that are not
yet released on diffpy.org, but they should work under both linux/mac
and windows.  Our current release has source packages for linux/mac
that are deployed by easy_install, for windows we roll out an NSIS
bundle, that contains a dedicated Python with all 3rd-party
dependencies (numpy, matplotlib, etc.) and binary eggs for any
packages that have extension modules.  The idea was to allow
easy_install package upgrades in the same way on both Linux and
Windows and also allow our software to be added to an existing Windows
Python tree.

Deploying shared library on Windows opens a whole new can of worms.
It can be included in an NSIS bundle, but I don't see it possible
to support upgrades with easy_install.  Should not be a big deal
as the Windows users probably prefer the NSIS bundle anyway.

> So, you want to find the "full path" of the install directory for
> post install processing? (using non-distutils installation terminology)

Yes, that is correct.  I don't know how to start a post-install
processing from setup.py, apart from overloading one of the
distutils/setuptools commands.  As PJE mentioned, it is indeed
possible to walk the stack trace and rip out the original sys.argv
(and its '--prefix' element) from the easy_install main.  It is an
atrocity, but perhaps the only thing left to do.

Anybody else out there with a need to deploy a shared C library
with a Python package?

Thank you,

Pavol

From david at ar.media.kyoto-u.ac.jp  Thu Nov 19 06:52:51 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 19 Nov 2009 14:52:51 +0900
Subject: [Distutils] obtaining easy_install prefix in setup.py
In-Reply-To: <20091118195921.GA3562@columbia.edu>
References: <20091118195921.GA3562@columbia.edu>
Message-ID: <4B04DD33.8010606@ar.media.kyoto-u.ac.jp>

Pavol Juhas wrote:
> easy_install already creates its own distribution object
> before sourcing pkgfoo/setup.py, in addition setup.py
> is executed in a sandbox with mangled sys.argv.
>   

At least for the Distribution class, what you can do is to detect
whether you are run under setuptools/easy_install or not, and
conditionally inherit from there. I don't know if it is enough to get
--prefix from easy_install - I never use easy_install.

David

From christophe.pradal at cirad.fr  Thu Nov 19 12:18:52 2009
From: christophe.pradal at cirad.fr (Christophe Pradal)
Date: Thu, 19 Nov 2009 12:18:52 +0100
Subject: [Distutils] obtaining easy_install prefix in setup.py
In-Reply-To: <20091119055956.GA4437@columbia.edu>
References: <20091118195921.GA3562@columbia.edu>	<eeff631dd646e791a3344f7705030848@preisshare.net>
	<20091119055956.GA4437@columbia.edu>
Message-ID: <4B05299C.2060608@cirad.fr>

Pavol Juhas a ?crit :
...
> Anybody else out there with a need to deploy a shared C library
> with a Python package?
>   
Yes. To deploy Python libraries with extension modules, or even shared 
libs on different OS (Linux, Mac and Windows) we have implemented 
OpenAlea.Deploy as an extension of setuptools. (see
http://openalea.gforge.inria.fr/wiki/doku.php?id=packages:compilation_installation:deploy:deploy).

OpenAlea.Deploy call scons scripts which copy in a build directory libs, 
headers and bin.
3 files are added to the egg (lib_dirs.txt, bin_dirs.txt and include_dirs).

When a new egg containing shared libs is added, all the shared libs are 
copied in a share directory to avoid conflicts. This allow us to install 
all our dependencies as egg and to share libs on Windows.

We have also extended SCons to retrieve the path of the libs installed 
with eggs
(see OpenAlea.SConsX).

There is a support for post install scripts.

However, we have modified easy_install into alea_install to call 
build_py command after build_ext ones.



From jnoller at gmail.com  Thu Nov 19 12:51:38 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 19 Nov 2009 06:51:38 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
Message-ID: <A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>



On Nov 19, 2009, at 12:57 AM, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

> On Thu, Nov 19, 2009 at 3:13 AM, Neal Becker <ndbecker2 at gmail.com>  
> wrote:
>> sudo easy_install -U openopt
>> Processing openopt
>> Running OOPy/setup.py -q bdist_egg --dist-dir
>> /home/nbecker/openopt/OOPy/egg-dist-tmp-fTMbRS
>> Adding openopt 0.24.dev to easy-install.pth file
>>
>> Installed /usr/lib/python2.6/site-packages/openopt-0.24.dev-py2.6.egg
>> Processing dependencies for openopt==0.24.dev
>> Finished processing dependencies for openopt==0.24.dev
>> Error in atexit._run_exitfuncs:
>> Traceback (most recent call last):
>>  File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
>>    func(*targs, **kargs)
>>  File "/usr/lib64/python2.6/multiprocessing/util.py", line 258, in
>> _exit_function
>>    info('process shutting down')
>> TypeError: 'NoneType' object is not callable
>> Error in sys.exitfunc:
>> Traceback (most recent call last):
>>  File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
>>    func(*targs, **kargs)
>>  File "/usr/lib64/python2.6/multiprocessing/util.py", line 258, in
>> _exit_function
>>    info('process shutting down')
>> TypeError: 'NoneType' object is not callable
>>
>> At least, I think it's distribute (I just did update fedora 11->12,  
>> so maybe
>> something got screwed up).  I did
>> easy_install -U distribute after the update.
>> sudo easy_install -U distribute
>> Searching for distribute
>> Reading http://pypi.python.org/simple/distribute/
>> Reading http://packages.python.org/distribute
>> Best match: distribute 0.6.8
>> Processing distribute-0.6.8-py2.6.egg
>> distribute 0.6.8 is already the active version in easy-install.pth
>> Installing easy_install script to /usr/bin
>> Installing easy_install-2.6 script to /usr/bin
>>
>> Using /usr/lib/python2.6/site-packages/distribute-0.6.8-py2.6.egg
>> Processing dependencies for distribute
>> Finished processing dependencies for distribute
>>
>
>
> That's a known bug in multiprocessing apparently,
>
> http://bugs.python.org/issue4106
>
> I am not sure why it's triggered in Distribute, but it happens when
> the garbage collector has staretd to remove some refs. Notice that the
> work is over, and everything is installed.
>
> I am not sure we can do something clean on Distribute side until it's
> fixed in Python,
>
> Can you reproduce this problem *everytime* ?
>
> Regards,
> Tarek
>
> ps: I am putting Jesse in the loop, he might have an idea

No - I don't. And while the error may look similiar to that bug, I  
think that's a red herring.

I have no idea why installing distribute - or any other third party  
module would trigger a bug in a different module in the stdlib, unless  
the module being installed was actively using multiprocessing.

So, can anyone tell me if something in this chain is actually  
importing and running multiprocessing?

From cournape at gmail.com  Thu Nov 19 13:08:03 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 19 Nov 2009 21:08:03 +0900
Subject: [Distutils] distribute traceback
In-Reply-To: <A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
Message-ID: <5b8d13220911190408s6e509e9bv806a459d662558c5@mail.gmail.com>

On Thu, Nov 19, 2009 at 8:51 PM, Jesse Noller <jnoller at gmail.com> wrote:
>
> No - I don't. And while the error may look similiar to that bug, I think
> that's a red herring.
>
> I have no idea why installing distribute - or any other third party module
> would trigger a bug in a different module in the stdlib, unless the module
> being installed was actively using multiprocessing.

When I have random problem caused by setuptools, it is often caused by
the develop feature. I would first try to uninstall any package in
develop mode

cheers,

David

From ziade.tarek at gmail.com  Thu Nov 19 13:05:29 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 19 Nov 2009 13:05:29 +0100
Subject: [Distutils] distribute traceback
In-Reply-To: <A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
Message-ID: <94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>

On Thu, Nov 19, 2009 at 12:51 PM, Jesse Noller <jnoller at gmail.com> wrote:
[..]
>
> No - I don't. And while the error may look similiar to that bug, I think
> that's a red herring.
>
> I have no idea why installing distribute - or any other third party module
> would trigger a bug in a different module in the stdlib, unless the module
> being installed was actively using multiprocessing.

It's more related to the way Distribute shutdowns the process, and the
fact that multiprocess
adds an entry to the atexit registry.

>
> So, can anyone tell me if something in this chain is actually importing and
> running multiprocessing?

Yes it does, thus multiprocessing adds an entry to atexit when its
util.get_logger() function
gets called.

IOW when atexit is triggered multiprocess.util.info is already
dereferenced by the GC.

My guess is that  the _exit_function should be careful about the
variables it uses because they are not garanteed to still be present
when things shutdown. (But I am not an atexit expert)

Tarek

From jnoller at gmail.com  Thu Nov 19 13:52:48 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 19 Nov 2009 07:52:48 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
	<94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>
Message-ID: <7088F261-06A5-4F28-A547-2E9016DF0D01@gmail.com>



On Nov 19, 2009, at 7:05 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> On Thu, Nov 19, 2009 at 12:51 PM, Jesse Noller <jnoller at gmail.com>  
> wrote:
> [..]
>>
>> No - I don't. And while the error may look similiar to that bug, I  
>> think
>> that's a red herring.
>>
>> I have no idea why installing distribute - or any other third party  
>> module
>> would trigger a bug in a different module in the stdlib, unless the  
>> module
>> being installed was actively using multiprocessing.
>
> It's more related to the way Distribute shutdowns the process, and the
> fact that multiprocess
> adds an entry to the atexit registry.
>
>>
>> So, can anyone tell me if something in this chain is actually  
>> importing and
>> running multiprocessing?
>
> Yes it does, thus multiprocessing adds an entry to atexit when its
> util.get_logger() function
> gets called.
>
> IOW when atexit is triggered multiprocess.util.info is already
> dereferenced by the GC.
>
> My guess is that  the _exit_function should be careful about the
> variables it uses because they are not garanteed to still be present
> when things shutdown. (But I am not an atexit expert)
>
> Tarek

Distribute imports multiprocessing? That's surprising

From ziade.tarek at gmail.com  Thu Nov 19 14:26:29 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 19 Nov 2009 14:26:29 +0100
Subject: [Distutils] distribute traceback
In-Reply-To: <7088F261-06A5-4F28-A547-2E9016DF0D01@gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
	<94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>
	<7088F261-06A5-4F28-A547-2E9016DF0D01@gmail.com>
Message-ID: <94bdd2610911190526g57668d6fk9d50d2e9cf790a6a@mail.gmail.com>

(Sorry mobile device)

s/yes it does/the distribution installed by distribute does

In any case the problem is about the order of execution when things shuts
down. The function in multiprocess via atexit makes the assumption it is
called before the gc has started removing objects.

Tarek

On Nov 19, 2009 2:16 PM, "Jesse Noller" <jnoller at gmail.com> wrote:

On Nov 19, 2009, at 7:05 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote: > On
Thu, Nov 19, 2009 at...
Distribute imports multiprocessing? That's surprising
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091119/a4029eb7/attachment-0001.htm>

From jnoller at gmail.com  Thu Nov 19 14:38:44 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 19 Nov 2009 08:38:44 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <94bdd2610911190526g57668d6fk9d50d2e9cf790a6a@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
	<94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>
	<7088F261-06A5-4F28-A547-2E9016DF0D01@gmail.com>
	<94bdd2610911190526g57668d6fk9d50d2e9cf790a6a@mail.gmail.com>
Message-ID: <4222a8490911190538y1e7d602asadc3d909174a01f8@mail.gmail.com>

On Thu, Nov 19, 2009 at 8:26 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> (Sorry mobile device)
>
> s/yes it does/the distribution installed by distribute does
>
> In any case the problem is about the order of execution when things shuts
> down. The function in multiprocess via atexit makes the assumption it is
> called before the gc has started removing objects.
>
> Tarek

No problem :) I was replying from my phone too.

Yeah, Antoine recently added the looping capability I wanted into
regrtest.py - I'll see if I can scratch up some time to let it spin -
the last time I looked at this it happened so rarely as to be nearly
impossible to debug.

jesse

From pje at telecommunity.com  Thu Nov 19 15:30:08 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 19 Nov 2009 09:30:08 -0500
Subject: [Distutils] obtaining easy_install prefix in setup.py
In-Reply-To: <20091119055956.GA4437@columbia.edu>
References: <20091118195921.GA3562@columbia.edu>
	<eeff631dd646e791a3344f7705030848@preisshare.net>
	<20091119055956.GA4437@columbia.edu>
Message-ID: <20091119143017.E2FCC3A407A@sparrow.telecommunity.com>

At 12:59 AM 11/19/2009 -0500, Pavol Juhas wrote:
>As PJE mentioned, it is indeed
>possible to walk the stack trace and rip out the original sys.argv
>(and its '--prefix' element) from the easy_install main.

Technically, I meant it's possible to walk back until you find a 
'self' that's a different Distribution instance, at which point you 
can ask for an easy_install command and check for a prefix.  You're 
probably better off doing that than trying to parse argv, especially 
since the prefix might be coming from a configuration file anyway.


From ziade.tarek at gmail.com  Thu Nov 19 15:43:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 19 Nov 2009 15:43:32 +0100
Subject: [Distutils] distribute traceback
In-Reply-To: <4222a8490911190538y1e7d602asadc3d909174a01f8@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
	<94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>
	<7088F261-06A5-4F28-A547-2E9016DF0D01@gmail.com>
	<94bdd2610911190526g57668d6fk9d50d2e9cf790a6a@mail.gmail.com>
	<4222a8490911190538y1e7d602asadc3d909174a01f8@mail.gmail.com>
Message-ID: <94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>

On Thu, Nov 19, 2009 at 2:38 PM, Jesse Noller <jnoller at gmail.com> wrote:
[..]
>
> No problem :) I was replying from my phone too.
>
> Yeah, Antoine recently added the looping capability I wanted into
> regrtest.py - I'll see if I can scratch up some time to let it spin -
> the last time I looked at this it happened so rarely as to be nearly
> impossible to debug.

Yes, I've tried to reproduce it in a small test but couldn't yet.
I can't reproduce Neal's problem either (I am on Mac OSX though, maybe atexit
doesn't behave the same way)

Tarek

From jnoller at gmail.com  Thu Nov 19 15:52:12 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 19 Nov 2009 09:52:12 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911182157s3199c41foeabbb2f6e2d65ffc@mail.gmail.com>
	<A3C564C0-B851-43B3-992B-70FB2C00A6F7@gmail.com>
	<94bdd2610911190405j311c62aemb1affd43b204e851@mail.gmail.com>
	<7088F261-06A5-4F28-A547-2E9016DF0D01@gmail.com>
	<94bdd2610911190526g57668d6fk9d50d2e9cf790a6a@mail.gmail.com>
	<4222a8490911190538y1e7d602asadc3d909174a01f8@mail.gmail.com>
	<94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
Message-ID: <4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>

On Thu, Nov 19, 2009 at 9:43 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Nov 19, 2009 at 2:38 PM, Jesse Noller <jnoller at gmail.com> wrote:
> [..]
>>
>> No problem :) I was replying from my phone too.
>>
>> Yeah, Antoine recently added the looping capability I wanted into
>> regrtest.py - I'll see if I can scratch up some time to let it spin -
>> the last time I looked at this it happened so rarely as to be nearly
>> impossible to debug.
>
> Yes, I've tried to reproduce it in a small test but couldn't yet.
> I can't reproduce Neal's problem either (I am on Mac OSX though, maybe atexit
> doesn't behave the same way)
>
> Tarek
>

When I tested it, that exception only got thrown on my machine once
every few thousand runs

From jnoller at gmail.com  Thu Nov 19 16:20:17 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 19 Nov 2009 10:20:17 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <200911191011.22345.ndbecker2@gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
	<4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>
	<200911191011.22345.ndbecker2@gmail.com>
Message-ID: <4222a8490911190720k73083273xa8b2ef2aec969dea@mail.gmail.com>

2009/11/19 Neal Becker <ndbecker2 at gmail.com>:
> On Thursday 19 November 2009, Jesse Noller wrote:
>> On Thu, Nov 19, 2009 at 9:43 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> > On Thu, Nov 19, 2009 at 2:38 PM, Jesse Noller <jnoller at gmail.com> wrote:
>> > [..]
>> >
>> >> No problem :) I was replying from my phone too.
>> >>
>> >> Yeah, Antoine recently added the looping capability I wanted into
>> >> regrtest.py - I'll see if I can scratch up some time to let it spin -
>> >> the last time I looked at this it happened so rarely as to be nearly
>> >> impossible to debug.
>> >
>> > Yes, I've tried to reproduce it in a small test but couldn't yet.
>> > I can't reproduce Neal's problem either (I am on Mac OSX though, maybe
>> > atexit doesn't behave the same way)
>> >
>> > Tarek
>>
>> When I tested it, that exception only got thrown on my machine once
>> every few thousand runs
>>
> Seems 100% repeatable here (fedora 12 x86_64)
>

It's going to take a bit, but I'm downloading/installing FC12 into a
virtual machine.

From ziade.tarek at gmail.com  Thu Nov 19 16:26:10 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 19 Nov 2009 16:26:10 +0100
Subject: [Distutils] distribute traceback
In-Reply-To: <4222a8490911190720k73083273xa8b2ef2aec969dea@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
	<4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>
	<200911191011.22345.ndbecker2@gmail.com>
	<4222a8490911190720k73083273xa8b2ef2aec969dea@mail.gmail.com>
Message-ID: <94bdd2610911190726l5f956294xed504b47a6ebc343@mail.gmail.com>

2009/11/19 Jesse Noller <jnoller at gmail.com>:
[..]
>>>
>>> When I tested it, that exception only got thrown on my machine once
>>> every few thousand runs
>>>
>> Seems 100% repeatable here (fedora 12 x86_64)
>>
>
> It's going to take a bit, but I'm downloading/installing FC12 into a
> virtual machine.
>

Thx for the help, much appreciated

Tarek

From ndbecker2 at gmail.com  Thu Nov 19 16:11:21 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 19 Nov 2009 10:11:21 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
	<4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>
Message-ID: <200911191011.22345.ndbecker2@gmail.com>

On Thursday 19 November 2009, Jesse Noller wrote:
> On Thu, Nov 19, 2009 at 9:43 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> > On Thu, Nov 19, 2009 at 2:38 PM, Jesse Noller <jnoller at gmail.com> wrote:
> > [..]
> >
> >> No problem :) I was replying from my phone too.
> >>
> >> Yeah, Antoine recently added the looping capability I wanted into
> >> regrtest.py - I'll see if I can scratch up some time to let it spin -
> >> the last time I looked at this it happened so rarely as to be nearly
> >> impossible to debug.
> >
> > Yes, I've tried to reproduce it in a small test but couldn't yet.
> > I can't reproduce Neal's problem either (I am on Mac OSX though, maybe
> > atexit doesn't behave the same way)
> >
> > Tarek
> 
> When I tested it, that exception only got thrown on my machine once
> every few thousand runs
> 
Seems 100% repeatable here (fedora 12 x86_64)

From jnoller at gmail.com  Thu Nov 19 16:47:17 2009
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 19 Nov 2009 10:47:17 -0500
Subject: [Distutils] distribute traceback
In-Reply-To: <94bdd2610911190726l5f956294xed504b47a6ebc343@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
	<4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>
	<200911191011.22345.ndbecker2@gmail.com>
	<4222a8490911190720k73083273xa8b2ef2aec969dea@mail.gmail.com>
	<94bdd2610911190726l5f956294xed504b47a6ebc343@mail.gmail.com>
Message-ID: <4222a8490911190747l3e4a9087uad2ecade5efa6f38@mail.gmail.com>

On Thu, Nov 19, 2009 at 10:26 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> 2009/11/19 Jesse Noller <jnoller at gmail.com>:
> [..]
>>>>
>>>> When I tested it, that exception only got thrown on my machine once
>>>> every few thousand runs
>>>>
>>> Seems 100% repeatable here (fedora 12 x86_64)
>>>
>>
>> It's going to take a bit, but I'm downloading/installing FC12 into a
>> virtual machine.
>>
>
> Thx for the help, much appreciated
>
> Tarek
>

I'm not making any promises to be able to get it done today/tomorrow. Work > *

From f.pollastri at inrim.it  Thu Nov 19 16:40:58 2009
From: f.pollastri at inrim.it (Fabrizio Pollastri)
Date: Thu, 19 Nov 2009 16:40:58 +0100
Subject: [Distutils] One package for both python and jython
Message-ID: <4B05670A.6050002@inrim.it>

Hello,
I am trying to upgrade a pure python package distribution based on
distutils in such a way that it will work with both python and jython.

Since my package depends on some part of python library when run by 
python and on some other part of jython library when run by jython,
the source structure is the following.

     my_package_source_dir/
         common_files
         files_for_python
         files_for_jython

It will be a maximum, if distutils let me to reach the following goals.

1. python/jython setup.py sdist command has to build the distribution 
tar my_package.tar.gz containing all files required by python and by jython.

2. python setup.py install command has to install in the python package tree

    my_package_source_dir/
         common_files
         files_for_python

3. jython setup.py install command has to install in the jython package tree

    my_package_source_dir/
         common_files
         files_for_jython

It is possible to implement this behavior with the setup function of
distutils? There are other ways (please, no package split)? Any
suggestion is welcome. Thank you in advance.


Fabrizio Pollastri

From ziade.tarek at gmail.com  Thu Nov 19 17:01:07 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 19 Nov 2009 17:01:07 +0100
Subject: [Distutils] distribute traceback
In-Reply-To: <4222a8490911190747l3e4a9087uad2ecade5efa6f38@mail.gmail.com>
References: <he29ku$ehv$1@ger.gmane.org>
	<94bdd2610911190643q536ceb34i21599a34b6436b5@mail.gmail.com>
	<4222a8490911190652r24753b02s89572ab99c325917@mail.gmail.com>
	<200911191011.22345.ndbecker2@gmail.com>
	<4222a8490911190720k73083273xa8b2ef2aec969dea@mail.gmail.com>
	<94bdd2610911190726l5f956294xed504b47a6ebc343@mail.gmail.com>
	<4222a8490911190747l3e4a9087uad2ecade5efa6f38@mail.gmail.com>
Message-ID: <94bdd2610911190801o6ab66413j9437e4fa67e61a3d@mail.gmail.com>

On Thu, Nov 19, 2009 at 4:47 PM, Jesse Noller <jnoller at gmail.com> wrote:
[..]
>
> I'm not making any promises to be able to get it done today/tomorrow. Work > *

No worries/hurry at all. At least we know that everything gets
installed fine before this TB so it's a minor bug.

From pje at telecommunity.com  Fri Nov 20 03:36:34 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 19 Nov 2009 21:36:34 -0500
Subject: [Distutils] One package for both python and jython
In-Reply-To: <4B05670A.6050002@inrim.it>
References: <4B05670A.6050002@inrim.it>
Message-ID: <20091120023643.CD85B3A40A2@sparrow.telecommunity.com>

At 04:40 PM 11/19/2009 +0100, Fabrizio Pollastri wrote:
>the source structure is the following.
>
>     my_package_source_dir/
>         common_files
>         files_for_python
>         files_for_jython
>
>It will be a maximum, if distutils let me to reach the following goals.
>
>1. python/jython setup.py sdist command has to build the 
>distribution tar my_package.tar.gz containing all files required by 
>python and by jython.
>
>2. python setup.py install command has to install in the python package tree
>
>    my_package_source_dir/
>         common_files
>         files_for_python
>
>3. jython setup.py install command has to install in the jython package tree
>
>    my_package_source_dir/
>         common_files
>         files_for_jython
>
>It is possible to implement this behavior with the setup function of
>distutils?

As long as the Python and Jython files are distinct packages or 
distinct top-level modules, you can simply pass different parameters 
to setup(), based on which platform you're on.  (Specifically, 
changing the 'packages' and 'py_modules' parameters.)

(If you have to have different versions of the *same* packages or 
modules, it's a bit more difficult and the directory layout is more 
complex, but it can still be done.) 


From ziade.tarek at gmail.com  Sun Nov 22 23:52:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 22 Nov 2009 23:52:11 +0100
Subject: [Distutils] PEP 345 - 3 new fields
Message-ID: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>

Hello

As suggested in Catalog-SIG by some people, I would like to propose
the addition of three more fields for 1.2:

"Repository-URL"
    A string containing the URL for the package's repository.
   Example:
        http://svn.python.org/projects/python/trunk/

"Repository-Browse-URL"
     A string containing the URL for the package's browsable repository.
   Example:
      http://svn.python.org/view/python/trunk

"Bug-Tracker-URL"
   A string containing the URL for the package's bug tracker
   Example:
      http://bugs.python.org/


Any thoughts ?

Tarek

From sridharr at activestate.com  Mon Nov 23 01:23:33 2009
From: sridharr at activestate.com (Sridhar Ratnakumar)
Date: Sun, 22 Nov 2009 16:23:33 -0800
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
Message-ID: <op.u3tncwts1q4jlt@whymac>

On Sun, 22 Nov 2009 14:52:11 -0800, Tarek Ziad? <ziade.tarek at gmail.com>  
wrote:

> Hello
> As suggested in Catalog-SIG by some people, I would like to propose
> the addition of three more fields for 1.2:
> "Repository-URL"
>     A string containing the URL for the package's repository.
>    Example:
>         http://svn.python.org/projects/python/trunk/
> "Repository-Browse-URL"
>      A string containing the URL for the package's browsable repository.
>    Example:
>       http://svn.python.org/view/python/trunk
> "Bug-Tracker-URL"
>    A string containing the URL for the package's bug tracker
>    Example:
>       http://bugs.python.org/

What are the possible use-cases (by the tools) of these new fields would  
be? I can imagine Repository-URL to be of use (eg: pip install  
IPython==dev), but have no idea why the other two should be added.

-srid



From ben+python at benfinney.id.au  Mon Nov 23 02:03:01 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 23 Nov 2009 12:03:01 +1100
Subject: [Distutils] PEP 345 - 3 new fields
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<op.u3tncwts1q4jlt@whymac>
Message-ID: <87iqd29hoa.fsf@benfinney.id.au>

"Sridhar Ratnakumar" <sridharr at activestate.com> writes:

> What are the possible use-cases (by the tools) of these new fields
> would be?

One of the important consumers of package metadata is PyPI. I imagine
these fields will be presented to the user along with other data about
the package.

-- 
 \          ?I got an answering machine for my phone. Now when someone |
  `\      calls me up and I'm not home, they get a recording of a busy |
_o__)                                          signal.? ?Steven Wright |
Ben Finney


From floris.bruynooghe at gmail.com  Mon Nov 23 10:20:07 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Mon, 23 Nov 2009 09:20:07 +0000
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <op.u3tncwts1q4jlt@whymac>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<op.u3tncwts1q4jlt@whymac>
Message-ID: <20091123092007.GA18133@laurie.devork>

On Sun, Nov 22, 2009 at 04:23:33PM -0800, Sridhar Ratnakumar wrote:
> On Sun, 22 Nov 2009 14:52:11 -0800, Tarek Ziad?
> <ziade.tarek at gmail.com> wrote:
> 
> >Hello
> >As suggested in Catalog-SIG by some people, I would like to propose
> >the addition of three more fields for 1.2:
> >"Repository-URL"
> >    A string containing the URL for the package's repository.
> >   Example:
> >        http://svn.python.org/projects/python/trunk/
> >"Repository-Browse-URL"
> >     A string containing the URL for the package's browsable repository.
> >   Example:
> >      http://svn.python.org/view/python/trunk
> >"Bug-Tracker-URL"
> >   A string containing the URL for the package's bug tracker
> >   Example:
> >      http://bugs.python.org/
> 
> What are the possible use-cases (by the tools) of these new fields
> would be? I can imagine Repository-URL to be of use (eg: pip install
> IPython==dev), but have no idea why the other two should be added.

Same as for Author and Author-email etc: more information for the
(human) users.  I think this is very valuable when browsing packages
on e.g. PyPI.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ziade.tarek at gmail.com  Mon Nov 23 10:39:42 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 23 Nov 2009 10:39:42 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <20091123092007.GA18133@laurie.devork>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<op.u3tncwts1q4jlt@whymac> <20091123092007.GA18133@laurie.devork>
Message-ID: <94bdd2610911230139p2d3c0cc2i806abdb20c2657c8@mail.gmail.com>

On Mon, Nov 23, 2009 at 10:20 AM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
> On Sun, Nov 22, 2009 at 04:23:33PM -0800, Sridhar Ratnakumar wrote:
>> On Sun, 22 Nov 2009 14:52:11 -0800, Tarek Ziad?
>> <ziade.tarek at gmail.com> wrote:
>>
>> >Hello
>> >As suggested in Catalog-SIG by some people, I would like to propose
>> >the addition of three more fields for 1.2:
>> >"Repository-URL"
>> > ? ?A string containing the URL for the package's repository.
>> > ? Example:
>> > ? ? ? ?http://svn.python.org/projects/python/trunk/
>> >"Repository-Browse-URL"
>> > ? ? A string containing the URL for the package's browsable repository.
>> > ? Example:
>> > ? ? ?http://svn.python.org/view/python/trunk
>> >"Bug-Tracker-URL"
>> > ? A string containing the URL for the package's bug tracker
>> > ? Example:
>> > ? ? ?http://bugs.python.org/
>>
>> What are the possible use-cases (by the tools) of these new fields
>> would be? I can imagine Repository-URL to be of use (eg: pip install
>> IPython==dev), but have no idea why the other two should be added.
>
> Same as for Author and Author-email etc: more information for the
> (human) users. ?I think this is very valuable when browsing packages
> on e.g. PyPI.
>

Yes. What is happening right now is that people are adding sometimes
these info in the long_description,
but there's no way to display these informations at PyPI in a specific
location on the page,
like a portlet for example.

PyPI is extracting urls out of long_description to put them in the
simple index for tools like easy_install to process them, but that's
it.

If you look at plone.org (which is PyPI -compatible) :
http://plone.org/products/jyu.pathkey

Each project can display in the "project resources" some of these
links, but that's currently done manually.

Having them in the metadata will make it easier for PyPI and other
similar servers to get them and display them,
and will encourage the developers to use them from within their setup.py

This proposal was triggered in the discussion about the
rating/commenting system : someone said that
we would have much more value in displaying these links, for people to
find the code and the tracker.


Tarek

From restless.being at gmail.com  Mon Nov 23 12:26:40 2009
From: restless.being at gmail.com (Restless Being)
Date: Mon, 23 Nov 2009 12:26:40 +0100
Subject: [Distutils] Buildout and PIL problem
Message-ID: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com>

Hi all!


I have a virtualenv created with --no-site-packages option and I'm going to
install there Django and some required libs using buildout. The problematic
library is PIL. Here is my buildout.cfg

[buildout]
parts = PIL django

[django]
recipe = djangorecipe
version = 1.1
project = project
eggs =
psycopg2
PIL
south
django-reversion
django-cms
wsgi = true
settings = production

[PIL]
recipe = zc.recipe.egg:custom
egg = PIL==1.1.6
find-links = http://dist.repoze.org

and it is something generally recommended by other users. See:
http://www.stereoplex.com/two-voices/a-django-development-environment-with-zc-buildout

I also tried some other versions of [PIL] like:
[PIL]
recipe = zc.recipe.egg:custom
egg = PIL==1.1.6
find-links = http://dist.repoze.org/PIL-1.1.6.tar.gz
index = http://dist.repoze.org/index


The first problem is that the above recipe installs PIL 1.1.7 instead of
1.1.6. Here is install log:

python bin/buildout -c devel.cfg
Updating django.
Getting distribution for 'PIL'.
WARNING: '' not a valid package name; please use only.-separated package
names in setup.py
In file included from /usr/include/python2.6/Python.h:8,
from libImaging/ImPlatform.h:10,
from libImaging/Imaging.h:14,
from Tk/tkImaging.c:53:
/usr/include/python2.6/pyconfig.h:1028:1: warning: "_POSIX_C_SOURCE"
redefined
In file included from /usr/include/stdio.h:28,
from /usr/include/tcl.h:141,
from /usr/include/tk.h:21,
from Tk/tkImaging.c:51:
/usr/include/features.h:210:1: warning: this is the location of the previous
definition
--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version 1.1.7
platform linux2 2.6.4 (r264:75706, Oct 27 2009, 06:16:59)
[GCC 4.4.1]
--------------------------------------------------------------------
--- TKINTER support available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
--- FREETYPE2 support available
--- LITTLECMS support available
--------------------------------------------------------------------
To check the build, run the selftest.py script.
zip_safe flag not set; analyzing archive contents...
Image: module references __file__
Got PIL 1.1.7.

Second problem is that the above PIL installation doesn't work with Django:

bin/django validate
Error: One or more models did not validate:
picture.picture: "image": To use ImageFields, you need to install the Python
Imaging Library. Get it at http://www.pythonware.com/products/pil/ .


At the same time, everything is ok when I manually install PIL using: pip
install http://effbot.org/downloads/Imaging-1.1.6.tar.gz

Any hints?
Jakub
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091123/acd5b57a/attachment.htm>

From attilaolah at gmail.com  Mon Nov 23 16:29:09 2009
From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=)
Date: Mon, 23 Nov 2009 16:29:09 +0100
Subject: [Distutils] Buildout and PIL problem
In-Reply-To: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com>
References: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com>
Message-ID: <1da06520911230729gefcbd1ci36c4bd7b50459943@mail.gmail.com>

Hi,

On Mon, Nov 23, 2009 at 12:26, Restless Being <restless.being at gmail.com> wrote:
> Hi all!
>
>
> I have a virtualenv created with --no-site-packages option and I'm going to
> install there Django and some required libs using buildout. The problematic
> library is PIL. Here is my buildout.cfg

PIL is known to have issues with buildout. Your problem is that it
puts it's files in <buildout>/eggs/PIL-#.#.#-py#.#-##.egg/, whereas it
should put them in <buildout>/eggs/PIL-#.#.#-py#.#-##.egg/PIL/.

As a workaround, you could run buildout like this (or put this in a
shell script):

./bin/buildout -N && mkdir `find eggs -iname PIL*.egg`/PIL && mv `find
eggs -iname PIL*.egg`/*.* `find eggs -iname PIL*.egg`/PIL

I know it looks rather ugly. I don't know if there's a better solution though.

> [buildout]
> parts = PIL django
>
> [django]
> recipe = djangorecipe
> version = 1.1
> project = project
> eggs =
> psycopg2
> PIL
> south
> django-reversion
> django-cms
> wsgi = true
> settings = production
>
> [PIL]
> recipe = zc.recipe.egg:custom
> egg = PIL==1.1.6
> find-links = http://dist.repoze.org

With my ugly from above, you can simply define PIL as a dependency in
setup.py or or buildout.cfg, no need for a special recipe.

> and it is something generally recommended by other users. See:
> http://www.stereoplex.com/two-voices/a-django-development-environment-with-zc-buildout
>
> I also tried some other versions of [PIL] like:
> [PIL]
> recipe = zc.recipe.egg:custom
> egg = PIL==1.1.6
> find-links = http://dist.repoze.org/PIL-1.1.6.tar.gz
> index = http://dist.repoze.org/index
>
>
> The first problem is that the above recipe installs PIL 1.1.7 instead of
> 1.1.6. Here is install log:

Why would you need 1.1.6? 1.1.7 should be a bugfix release anyway...
- Show quoted text -
It's because the files are not in their directory (PIL), so it's not
recognized as a package. After the movearound everything should work
fine.


Bests,
Attila

From restless.being at gmail.com  Mon Nov 23 19:36:51 2009
From: restless.being at gmail.com (Restless Being)
Date: Mon, 23 Nov 2009 19:36:51 +0100
Subject: [Distutils] Buildout and PIL problem
In-Reply-To: <1da06520911230729gefcbd1ci36c4bd7b50459943@mail.gmail.com>
References: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com>
	<1da06520911230729gefcbd1ci36c4bd7b50459943@mail.gmail.com>
Message-ID: <d5fde36f0911231036q38c4b8bdq7bd6f48a44679807@mail.gmail.com>

2009/11/23 Attila Ol?h <attilaolah at gmail.com>

> PIL is known to have issues with buildout. Your problem is that it
> puts it's files in <buildout>/eggs/PIL-#.#.#-py#.#-##.egg/, whereas it
> should put them in <buildout>/eggs/PIL-#.#.#-py#.#-##.egg/PIL/.
>
> As a workaround, you could run buildout like this (or put this in a
> shell script):
>
> ./bin/buildout -N && mkdir `find eggs -iname PIL*.egg`/PIL && mv `find
> eggs -iname PIL*.egg`/*.* `find eggs -iname PIL*.egg`/PIL
>
> I know it looks rather ugly. I don't know if there's a better solution
> though.
>
It's ugly but the problem is rather that I'd like to use buildout to avoid
such things (it's my first buildout config). But thanks.


>
> Why would you need 1.1.6? 1.1.7 should be a bugfix release anyway...
>
Because 1.1.6 from repoze has correct structure. It has PIL in its egg.

This config installs both PILs 1.1.6 and 1.1.7:

[buildout]
parts = PIL django omlette cms-media

[PIL]
recipe = zc.recipe.egg
eggs = PIL==1.1.6
find-links = http://dist.repoze.org

[django]
recipe = djangorecipe
version = 1.1
project = project
eggs =
PIL
psycopg2
south
django-reversion
django-cms
wsgi = true
settings = production

[omlette]
recipe = collective.recipe.omelette
eggs = ${django:eggs}

[cms-media]
recipe = svetlyak40wt.recipe.symlinks
path = project/media
files =
${buildout:parts-directory}/omlette/cms/media/cms cms


but this work as expected:

[buildout]
parts = PIL django omlette cms-media

[PIL]
recipe = zc.recipe.egg
eggs = PIL==1.1.6
find-links = http://dist.repoze.org

[django]
recipe = djangorecipe
version = 1.1
project = project
eggs =
PIL == 1.1.6
psycopg2
south
django-reversion
django-cms
wsgi = true
settings = production

[omlette]
recipe = collective.recipe.omelette
eggs = ${django:eggs}

[cms-media]
recipe = svetlyak40wt.recipe.symlinks
path = project/media
files =
${buildout:parts-directory}/omlette/cms/media/cms cms

So it's solved. Buildout was installing two different versions of PIL, one
for PIL part (1.1.6) and another one for django part (1.1.7). After I added
specific version to PIL in django it works :D

One more question. Why after I added eggs = psycopg2 just after [buildout]
buildout reported that it is omitting unused eggs in [buildout]? I had to
move psycopg2 to eggs in djangopart.

Jakub
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091123/c51b783a/attachment.htm>

From attilaolah at gmail.com  Mon Nov 23 20:05:03 2009
From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=)
Date: Mon, 23 Nov 2009 20:05:03 +0100
Subject: [Distutils] Buildout and PIL problem
In-Reply-To: <d5fde36f0911231036q38c4b8bdq7bd6f48a44679807@mail.gmail.com>
References: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com> 
	<1da06520911230729gefcbd1ci36c4bd7b50459943@mail.gmail.com> 
	<d5fde36f0911231036q38c4b8bdq7bd6f48a44679807@mail.gmail.com>
Message-ID: <1da06520911231105s8df4769m400136ad816b22ab@mail.gmail.com>

Hi,

On Mon, Nov 23, 2009 at 19:36, Restless Being <restless.being at gmail.com> wrote:
> 2009/11/23 Attila Ol?h <attilaolah at gmail.com>
>>
>> PIL is known to have issues with buildout. Your problem is that it
>> puts it's files in <buildout>/eggs/PIL-#.#.#-py#.#-##.egg/, whereas it
>> should put them in <buildout>/eggs/PIL-#.#.#-py#.#-##.egg/PIL/.
>>
>> As a workaround, you could run buildout like this (or put this in a
>> shell script):
>>
>> ./bin/buildout -N && mkdir `find eggs -iname PIL*.egg`/PIL && mv `find
>> eggs -iname PIL*.egg`/*.* `find eggs -iname PIL*.egg`/PIL
>>
>> I know it looks rather ugly. I don't know if there's a better solution
>> though.
>
> It's ugly but the problem is rather that I'd like to use buildout to avoid
> such things (it's my first buildout config). But thanks.
>
>>
>> Why would you need 1.1.6? 1.1.7 should be a bugfix release anyway...
>
> Because 1.1.6 from repoze has correct structure. It has PIL in its egg.
> This config installs both PILs 1.1.6 and 1.1.7:
> [buildout]
> parts = PIL django omlette cms-media
>
> [PIL]
> recipe = zc.recipe.egg
> eggs = PIL==1.1.6
> find-links = http://dist.repoze.org
>
> [django]
> recipe = djangorecipe
> version = 1.1
> project = project
> eggs =
> PIL
> psycopg2
> south
> django-reversion
> django-cms
> wsgi = true
> settings = production
>
> [omlette]
> recipe = collective.recipe.omelette
> eggs = ${django:eggs}
>
> [cms-media]
> recipe = svetlyak40wt.recipe.symlinks
> path = project/media
> files =
> ${buildout:parts-directory}/omlette/cms/media/cms cms
>
>
> but this work as expected:
> [buildout]
> parts = PIL django omlette cms-media
>
> [PIL]
> recipe = zc.recipe.egg
> eggs = PIL==1.1.6
> find-links = http://dist.repoze.org
>
> [django]
> recipe = djangorecipe
> version = 1.1
> project = project
> eggs =
> PIL == 1.1.6
> psycopg2
> south
> django-reversion
> django-cms
> wsgi = true
> settings = production
>
> [omlette]
> recipe = collective.recipe.omelette
> eggs = ${django:eggs}
>
> [cms-media]
> recipe = svetlyak40wt.recipe.symlinks
> path = project/media
> files =
> ${buildout:parts-directory}/omlette/cms/media/cms cms

Ah, I missed that. Of course, djangorecipe thought it'd be a good idea
to update PIL.

> So it's solved. Buildout was installing two different versions of PIL, one
> for PIL part (1.1.6) and another one for django part (1.1.7). After I added
> specific version to PIL in django it works :D
> One more question. Why after I added eggs = psycopg2 just after [buildout]
> buildout reported that it is omitting unused eggs in [buildout]? I had to
> move psycopg2 to eggs in djangopart.
> Jakub

No idea; psycopg2 used to work for my buildout-based projects, but I
haven't tried it with djangorecipe before, but I've just tried it now:
it is working for me. How does your buildout.cfg look like? Do you
depend on psycopg2 in your setup.py (or is it in an extra)?

Cheers,
Attila

From jim at zope.com  Mon Nov 23 20:46:44 2009
From: jim at zope.com (Jim Fulton)
Date: Mon, 23 Nov 2009 14:46:44 -0500
Subject: [Distutils] Buildout and PIL problem
In-Reply-To: <d5fde36f0911231036q38c4b8bdq7bd6f48a44679807@mail.gmail.com>
References: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com> 
	<1da06520911230729gefcbd1ci36c4bd7b50459943@mail.gmail.com> 
	<d5fde36f0911231036q38c4b8bdq7bd6f48a44679807@mail.gmail.com>
Message-ID: <1099b90b0911231146x64f6775co33bfc0ccca592dd1@mail.gmail.com>

On Mon, Nov 23, 2009 at 1:36 PM, Restless Being
...
> So it's solved. Buildout was installing two different versions of PIL, one
> for PIL part (1.1.6) and another one for django part (1.1.7). After I added
> specific version to PIL in django it works :D

Great! :)

> One more question. Why after I added eggs = psycopg2 just after [buildout]
> buildout reported that it is omitting unused eggs in [buildout]? I had to
> move psycopg2 to eggs in djangopart.

It expect it said "Unused option for buildout: eggs".

The eggs option doesn't mean anything to the buildout section.  This
message was just letting you know that something wasn't right.  The
information you provided was having no effect.

Jim

-- 
Jim Fulton

From restless.being at gmail.com  Mon Nov 23 21:10:33 2009
From: restless.being at gmail.com (Restless Being)
Date: Mon, 23 Nov 2009 21:10:33 +0100
Subject: [Distutils] Buildout and PIL problem
In-Reply-To: <1099b90b0911231146x64f6775co33bfc0ccca592dd1@mail.gmail.com>
References: <d5fde36f0911230326l6aeacd6t5df39a72a518b71f@mail.gmail.com>
	<1da06520911230729gefcbd1ci36c4bd7b50459943@mail.gmail.com>
	<d5fde36f0911231036q38c4b8bdq7bd6f48a44679807@mail.gmail.com>
	<1099b90b0911231146x64f6775co33bfc0ccca592dd1@mail.gmail.com>
Message-ID: <d5fde36f0911231210m44df4483t6495f0021829632@mail.gmail.com>

2009/11/23 Jim Fulton <jim at zope.com>

> It expect it said "Unused option for buildout: eggs".
>

Exactly.


>
> The eggs option doesn't mean anything to the buildout section.  This
> message was just letting you know that something wasn't right.  The
> information you provided was having no effect.
>
I think i didn't understand the following snippet from
http://jacobian.org/writing/django-apps-with-buildout/:

***
[buildout]
parts = python
develop = .
eggs = django-shorturls

[python]
recipe = zc.recipe.egg
interpreter = python
eggs = ${buildout:eggs}
***

Now I see that the "eggs = django-shortuls" isn't there for buildout
section, it is there for other sections which include it with
$(buildout:eggs).

Thanks a lot!
Jakub
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091123/8ab32627/attachment.htm>

From ziade.tarek at gmail.com  Tue Nov 24 10:47:41 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 24 Nov 2009 10:47:41 +0100
Subject: [Distutils] PEP 386 status - last round here ?
Message-ID: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>

Hello,

PEP 386 seem to be ready, and I would like to push if for feedback at
python-dev, just before PEP 345 is pushed.

My only concern is now to make sure the PEP motivations and
explanations are crystal clear.

Anyone see any problem ? or have any concern with this PEP ?

This PEP is the basis for PEP 345 acceptation.

Regards
Tarek

-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From david at ar.media.kyoto-u.ac.jp  Tue Nov 24 10:50:29 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 24 Nov 2009 18:50:29 +0900
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
Message-ID: <4B0BAC65.2040901@ar.media.kyoto-u.ac.jp>

Hi Tarek,

Tarek Ziad? wrote:
> Anyone see any problem ? or have any concern with this PEP ?
>   

Just a question: will it affect how version are checked for tools  (e.g.
get_versions in distutils/cygwinccompiler.py) ?

cheers,

David

From ziade.tarek at gmail.com  Tue Nov 24 11:21:14 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 24 Nov 2009 11:21:14 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0BAC65.2040901@ar.media.kyoto-u.ac.jp>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0BAC65.2040901@ar.media.kyoto-u.ac.jp>
Message-ID: <94bdd2610911240221u6dffe3a9j8076b9dd3817af6c@mail.gmail.com>

On Tue, Nov 24, 2009 at 10:50 AM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
> Hi Tarek,
>
> Tarek Ziad? wrote:
>> Anyone see any problem ? or have any concern with this PEP ?
>>
>
> Just a question: will it affect how version are checked for tools ?(e.g.
> get_versions in distutils/cygwinccompiler.py) ?

This behavior will not be affected.

Although, I havn't looked at gcc, ld and dllwrap version schemes yet.

If they are compatible with RationalVersion these functions will use it.

If not, I will probably keep something similar to LooseVersion, but
dedicated to this usage.
So maybe renamed accordingly so they are specific to gcc/ld/dllwrap schemes

Tarek

>
> cheers,
>
> David
>



-- 
Tarek Ziad? | http://ziade.org |????????????! |????????????

From david at ar.media.kyoto-u.ac.jp  Tue Nov 24 11:34:17 2009
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 24 Nov 2009 19:34:17 +0900
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911240221u6dffe3a9j8076b9dd3817af6c@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	
	<4B0BAC65.2040901@ar.media.kyoto-u.ac.jp>
	<94bdd2610911240221u6dffe3a9j8076b9dd3817af6c@mail.gmail.com>
Message-ID: <4B0BB6A9.4070004@ar.media.kyoto-u.ac.jp>

Tarek Ziad? wrote:
> This behavior will not be affected.
>
>   

Ok.

> Although, I havn't looked at gcc, ld and dllwrap version schemes yet.
>
> If they are compatible with RationalVersion these functions will use it.
>
> If not, I will probably keep something similar to LooseVersion, but
> dedicated to this usage.
> So maybe renamed accordingly so they are specific to gcc/ld/dllwrap schemes
>   

In that case, maybe LooseVersion could be redefined to something which
does not do anything for cygwin case. The current behavior often breaks
cygwin builds because LooseVersion is not capable of parsing upstream
versions, and I don't really understand the rationale for checking the
version in the first place.

thanks,

David

From Ronny.Pfannschmidt at gmx.de  Tue Nov 24 11:57:20 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 24 Nov 2009 11:57:20 +0100
Subject: [Distutils] specifying the shipping of subpackages without their
	superpackages
Message-ID: <1259060240.9155.4.camel@localhost>

Hi,

i recently noticed that shipping a package like 'foo.bar' without also
shipping 'foo' seems to work fine in distutils.

since that would be nice way to ship packages within a namespace i want
to sugest specifying the fact that it works and to clarify how it works.


Regards Ronny


From r.ritz at biologie.hu-berlin.de  Tue Nov 24 12:39:17 2009
From: r.ritz at biologie.hu-berlin.de (Raphael Ritz)
Date: Tue, 24 Nov 2009 12:39:17 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
Message-ID: <heggl4$fn9$1@ger.gmane.org>

Tarek Ziad? wrote:
> Hello
> 
> As suggested in Catalog-SIG by some people, I would like to propose
> the addition of three more fields for 1.2:
> 
> "Repository-URL"
>     A string containing the URL for the package's repository.
>    Example:
>         http://svn.python.org/projects/python/trunk/
> 
> "Repository-Browse-URL"
>      A string containing the URL for the package's browsable repository.
>    Example:
>       http://svn.python.org/view/python/trunk
> 
> "Bug-Tracker-URL"
>    A string containing the URL for the package's bug tracker
>    Example:
>       http://bugs.python.org/
> 
> 
> Any thoughts ?

+1

I miss them all the time when browsing PyPI.

Raphael


> 
> Tarek
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
> 


From chris at simplistix.co.uk  Tue Nov 24 14:03:45 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 24 Nov 2009 13:03:45 +0000
Subject: [Distutils] specifying the shipping of subpackages without
 their superpackages
In-Reply-To: <1259060240.9155.4.camel@localhost>
References: <1259060240.9155.4.camel@localhost>
Message-ID: <4B0BD9B1.5080001@simplistix.co.uk>

Ronny Pfannschmidt wrote:
> i recently noticed that shipping a package like 'foo.bar' without also
> shipping 'foo' seems to work fine in distutils.
> 
> since that would be nice way to ship packages within a namespace i want
> to sugest specifying the fact that it works and to clarify how it works.

There's already for PEP for namespace packages...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From ssteinerx at gmail.com  Tue Nov 24 14:31:34 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 24 Nov 2009 08:31:34 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
Message-ID: <5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>

On Nov 24, 2009, at 4:47 AM, Tarek Ziad? wrote:

> Hello,
> 
> PEP 386 seem to be ready, and I would like to push if for feedback at
> python-dev, just before PEP 345 is pushed.
> 
> My only concern is now to make sure the PEP motivations and explanations are crystal clear.
> 
> Anyone see any problem ? or have any concern with this PEP ?

I think the motivation and explanations are clear.

There are, however, some issues that should be addressed before it's accepted as "final."

1> 	There seems to be a typo on line 29 of verlib.py where it says 'f' < 'b', shouldn't that be 'b' < 'f' ?

2> 	The explanation for "suggest_rational_version" is stubbed out in the README.txt file:
		
		XXX explain here suggest_rational_version
 	
	it should be documented what we expect it to do and that leads into...

3> 	test_suggest_rational_version needs a more comprehensive test suite (like all the version numbers from PyPi).  Right now it only has 11 cases and they're all pretty mild though the doc asserts success rates on PyPi that are much higher.  This should be done so that we can insure that the algorithm stays stable (and hopefully improves) in future versions.  (I will create this expanded test using PyPi data, if requested).

4>	The [#requrires] reference is missing from the README.txt file

5>	There is no cross reference to any other non-python project using this scheme.  There is a note that  "During Pycon, members of the Python, Ubuntu and Fedora community worked on a version standard that would be acceptable for everyone"  but there is no evidence that anyone else has agreed to this standard and, particularly, to this reference implementation.

S


From Ronny.Pfannschmidt at gmx.de  Tue Nov 24 14:45:00 2009
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Tue, 24 Nov 2009 14:45:00 +0100
Subject: [Distutils] specifying the shipping of subpackages without
 their superpackages
In-Reply-To: <4B0BD9B1.5080001@simplistix.co.uk>
References: <1259060240.9155.4.camel@localhost>
	<4B0BD9B1.5080001@simplistix.co.uk>
Message-ID: <1259070300.9155.13.camel@localhost>

On Tue, 2009-11-24 at 13:03 +0000, Chris Withers wrote:
> Ronny Pfannschmidt wrote:
> > i recently noticed that shipping a package like 'foo.bar' without also
> > shipping 'foo' seems to work fine in distutils.
> > 
> > since that would be nice way to ship packages within a namespace i want
> > to sugest specifying the fact that it works and to clarify how it works.
> 
> There's already for PEP for namespace packages...

Yes, but that only deals with package namespacing in terms of the import
system, not a single word about how things get installed within
namespaces.

The current way is to ship dozens of virtually empty __init__.py files
in the superpackages
(i.e. only the call to pkgutil in order to extend __path__)

I consider that unfortunate and undesirable.

I see a clear need to add convient names at the package level and the
current way to deal with namespaces cant handle that.

The ability to install subpackages while ignoring superpackages is a way
to archieve that, cause nothing will overwrite the superpackages
__init__.py.
So it may be subject to usefull additions beyond the mere __path__
extension,

Ronny


From ziade.tarek at gmail.com  Tue Nov 24 21:40:04 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 24 Nov 2009 21:40:04 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
Message-ID: <94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>

On Tue, Nov 24, 2009 at 2:31 PM, ssteinerX at gmail.com
<ssteinerx at gmail.com> wrote:
[..]
>
> There are, however, some issues that should be addressed before it's accepted as "final."
>
> 1> ? ? ?There seems to be a typo on line 29 of verlib.py where it says 'f' < 'b', shouldn't that be 'b' < 'f' ?

Right, fixed. thanks !


>
> 2> ? ? ?The explanation for "suggest_rational_version" is stubbed out in the README.txt file:
>
> ? ? ? ? ? ? ? ?XXX explain here suggest_rational_version
>
> ? ? ? ?it should be documented what we expect it to do and that leads into...
>

This README.txt is a bit outdated. It's an old copy of PEP 386 (which
explains suggest_rational_version)

> 3> ? ? ?test_suggest_rational_version needs a more comprehensive test suite (like all the version numbers from PyPi). ?Right now it only has 11 cases and they're all pretty mild though the doc asserts success rates on PyPi that are much higher. ?This should be done so that we can insure that the algorithm stays stable (and hopefully improves) in future versions. ?(I will create this expanded test using PyPi data, if requested).

That would be great,

>
> 4> ? ? ?The [#requrires] reference is missing from the README.txt file


I'll update it using the latest PEP 376 text

>
> 5> ? ? ?There is no cross reference to any other non-python project using this scheme. ?There is a note that ?"During Pycon, members of the Python, Ubuntu and Fedora community worked on a version standard that would be acceptable for everyone" ?but there is no evidence that anyone else has agreed to this standard and, particularly, to this reference implementation.

I don't think any project uses this scheme yet, besides some Python
projects. What happened is that when we
worked on this scheme, Toshio (Fedora) and Matthias (Ubuntu/Debian)
made sure it would be acceptable for their needs to translate it in
their own schemes automatically.

For instance, RPMs want to avoid date-based versions, and Debian
packages avoids "-" characters.

Tarek

From ssteinerx at gmail.com  Tue Nov 24 23:03:14 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Tue, 24 Nov 2009 17:03:14 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
Message-ID: <1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>


On Nov 24, 2009, at 3:40 PM, Tarek Ziad? wrote:
>> 2>      The explanation for "suggest_rational_version" is stubbed out in the README.txt file:
>> 
>>                XXX explain here suggest_rational_version
>> 
>>        it should be documented what we expect it to do and that leads into...
>> 
> 
> This README.txt is a bit outdated. It's an old copy of PEP 386 (which explains suggest_rational_version)
>> 4>      The [#requrires] reference is missing from the README.txt file
> I'll update it using the latest PEP 376 text


Ok, I'll look for those when I pull later tonight?

> 
>> 3>      test_suggest_rational_version needs a more comprehensive test suite (like all the version numbers from PyPi).  Right now it only has 11 cases and they're all pretty mild though the doc asserts success rates on PyPi that are much higher.  This should be done so that we can insure that the algorithm stays stable (and hopefully improves) in future versions.  (I will create this expanded test using PyPi data, if requested).
> 
> That would be great,

May I have access please?

>> 5>      There is no cross reference to any other non-python project using this scheme.  There is a note that  "During Pycon, members of the Python, Ubuntu and Fedora community worked on a version standard that would be acceptable for everyone"  but there is no evidence that anyone else has agreed to this standard and, particularly, to this reference implementation.
> 
> I don't think any project uses this scheme yet, besides some Python projects. What happened is that when we worked on this scheme, Toshio (Fedora) and Matthias (Ubuntu/Debian) made sure it would be acceptable for their needs to translate it in their own schemes automatically.
> 
> For instance, RPMs want to avoid date-based versions, and Debian packages avoids "-" characters.

Ok, it would be great to have an example of an RPM or deb package showing the direct translation but no big deal.

S


From ziade.tarek at gmail.com  Wed Nov 25 18:04:53 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Nov 2009 18:04:53 +0100
Subject: [Distutils] specifying the shipping of subpackages without
	their superpackages
In-Reply-To: <1259070300.9155.13.camel@localhost>
References: <1259060240.9155.4.camel@localhost>
	<4B0BD9B1.5080001@simplistix.co.uk>
	<1259070300.9155.13.camel@localhost>
Message-ID: <94bdd2610911250904p78dc1ac8i44b9091ab0e81757@mail.gmail.com>

On Tue, Nov 24, 2009 at 2:45 PM, Ronny Pfannschmidt
<Ronny.Pfannschmidt at gmx.de> wrote:
> On Tue, 2009-11-24 at 13:03 +0000, Chris Withers wrote:
>> Ronny Pfannschmidt wrote:
>> > i recently noticed that shipping a package like 'foo.bar' without also
>> > shipping 'foo' seems to work fine in distutils.
>> >
>> > since that would be nice way to ship packages within a namespace i want
>> > to sugest specifying the fact that it works and to clarify how it works.
>>
>> There's already for PEP for namespace packages...
>
> Yes, but that only deals with package namespacing in terms of the import
> system, not a single word about how things get installed within
> namespaces.
>
> The current way is to ship dozens of virtually empty __init__.py files
> in the superpackages
> (i.e. only the call to pkgutil in order to extend __path__)
>
> I consider that unfortunate and undesirable.
>
> I see a clear need to add convient names at the package level and the
> current way to deal with namespaces cant handle that.
>
> The ability to install subpackages while ignoring superpackages is a way
> to archieve that, cause nothing will overwrite the superpackages
> __init__.py.
> So it may be subject to usefull additions beyond the mere __path__
> extension,

I think the namespace PEP lacks of some usage examples, for these use case:

how do I define code in foo.bar.baz, and how do I describe it so
Distutils installs it

I am cc'ing Martin (he's not in distutils-sig IIRC)

Regards
Tarek

From mal at egenix.com  Wed Nov 25 18:29:47 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 25 Nov 2009 18:29:47 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
Message-ID: <4B0D698B.4080708@egenix.com>

Could we extend the pseudo-format of the versions a little to also
include variants which use more than just one character and also
allow hyphens and spaces to be used for additional clarity ?

VERSION_RE = re.compile(r'''
    ^
    (?P<version>\d+\.\d+)          # minimum 'N.N'
    (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
    (?:
        [- .]*
        (?P<prerel>[a-z]+)        # 'a'=alpha, 'b'=beta, 'c'=release candidate
        [- .]*
        (?P<prerelversion>\d+(?:\.\d+)*)
    )?
    (?P<postdev>
        ([- .]*post[- .]*(?P<post>\d+))?
        ([- .]*dev[- .]*(?P<dev>\d+))?
    )?
    $''', re.VERBOSE + re.I)

The pre-release marker would then be interpreted in alphabetical
order, ie. 'alpha' < 'beta' < 'rc'.

This minor change would broaden the scope of the scheme somewhat
and make it more compatible to what's being used outside the
python-dev sphere (esp. with respect to 'c' standing for release
candidate... unless you happen to read it as gamma ;-).

The added optional hypens and spaces make the versions more
readable, IMHO.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Wed Nov 25 18:44:09 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 25 Nov 2009 18:44:09 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0D698B.4080708@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com>
Message-ID: <4B0D6CE9.50306@egenix.com>

M.-A. Lemburg wrote:
> Could we extend the pseudo-format of the versions a little to also
> include variants which use more than just one character and also
> allow hyphens and spaces to be used for additional clarity ?
> 
> VERSION_RE = re.compile(r'''
>     ^
>     (?P<version>\d+\.\d+)          # minimum 'N.N'
>     (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
>     (?:
>         [- .]*
>         (?P<prerel>[a-z]+)        # 'a'=alpha, 'b'=beta, 'c'=release candidate
>         [- .]*
>         (?P<prerelversion>\d+(?:\.\d+)*)
>     )?
>     (?P<postdev>
>         ([- .]*post[- .]*(?P<post>\d+))?
>         ([- .]*dev[- .]*(?P<dev>\d+))?
>     )?
>     $''', re.VERBOSE + re.I)
> 
> The pre-release marker would then be interpreted in alphabetical
> order, ie. 'alpha' < 'beta' < 'rc'.
> 
> This minor change would broaden the scope of the scheme somewhat
> and make it more compatible to what's being used outside the
> python-dev sphere (esp. with respect to 'c' standing for release
> candidate... unless you happen to read it as gamma ;-).

... and even python-dev has switched to "rc" for these:

http://www.python.org/dev/peps/pep-0361/

> The added optional hypens and spaces make the versions more
> readable, IMHO.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Wed Nov 25 19:20:05 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Nov 2009 19:20:05 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0D6CE9.50306@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
Message-ID: <94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>

2009/11/25 M.-A. Lemburg <mal at egenix.com>:
> M.-A. Lemburg wrote:
>> Could we extend the pseudo-format of the versions a little to also
>> include variants which use more than just one character and also
>> allow hyphens and spaces to be used for additional clarity ?
>>
>> VERSION_RE = re.compile(r'''
>> ? ? ^
>> ? ? (?P<version>\d+\.\d+) ? ? ? ? ?# minimum 'N.N'
>> ? ? (?P<extraversion>(?:\.\d+)*) ? # any number of extra '.N' segments
>> ? ? (?:
>> ? ? ? ? [- .]*
>> ? ? ? ? (?P<prerel>[a-z]+) ? ? ? ?# 'a'=alpha, 'b'=beta, 'c'=release candidate
>> ? ? ? ? [- .]*
>> ? ? ? ? (?P<prerelversion>\d+(?:\.\d+)*)
>> ? ? )?
>> ? ? (?P<postdev>
>> ? ? ? ? ([- .]*post[- .]*(?P<post>\d+))?
>> ? ? ? ? ([- .]*dev[- .]*(?P<dev>\d+))?
>> ? ? )?
>> ? ? $''', re.VERBOSE + re.I)
>>
>> The pre-release marker would then be interpreted in alphabetical
>> order, ie. 'alpha' < 'beta' < 'rc'.
>>
>> This minor change would broaden the scope of the scheme somewhat
>> and make it more compatible to what's being used outside the
>> python-dev sphere (esp. with respect to 'c' standing for release
>> candidate... unless you happen to read it as gamma ;-).
>
> ... and even python-dev has switched to "rc" for these:
>
> http://www.python.org/dev/peps/pep-0361/
>
>> The added optional hypens and spaces make the versions more
>> readable, IMHO.

The goal of the scheme is to propose a unique standard, together with
the "suggest_rational_version" API that allow transforming an
almost-rational version to a rational version. IOW, having several
ways of defining separators would
make the version scheme loose.  I have the feeling this would have a
negative impact but I don't have a strong example in mind yet.

Now, suggest_rational_version is meant to transform those close-enough
versions strings into working strings,

Example with "rc" vs "c":

>>> from verlib import suggest_rational_version
>>> suggest_rational_version('2.4rc1')
'2.4c1'

Meaning that a package manager could be permissive and use those
almost-rational form with this API. So if we get
suggest_rational_version work with hypens and spaces as you are
suggesting, that could be a solution.

Now, about the "c" choice. "c" was preferred to "rc" here to avoid
using "r" which has a strong Subversion connotation.

Next, hyphens were removed because they are avoided in some systems
like Debian or Ubuntu,  and have a very specific meaning in
setuptools.

from the setuptools doc:

   The algorithm assumes that strings like "-" and any alpha string that
   alphabetically follows "final"  represents a "patch level".  So, "2.4-1"
   is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
   considered newer than "2.4-1", which in turn is newer than "2.4".



Regards
Tarek

From ssteinerx at gmail.com  Wed Nov 25 19:20:38 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Wed, 25 Nov 2009 13:20:38 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0D6CE9.50306@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
Message-ID: <2E14ABCB-17D9-4C57-96C3-118B012A8086@gmail.com>


On Nov 25, 2009, at 12:44 PM, M.-A. Lemburg wrote:

> M.-A. Lemburg wrote:
>> Could we extend the pseudo-format of the versions a little to also
>> include variants which use more than just one character and also
>> allow hyphens and spaces to be used for additional clarity ?
>> 
>> VERSION_RE = re.compile(r'''
>>    ^
>>    (?P<version>\d+\.\d+)          # minimum 'N.N'
>>    (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
>>    (?:
>>        [- .]*
>>        (?P<prerel>[a-z]+)        # 'a'=alpha, 'b'=beta, 'c'=release candidate
>>        [- .]*
>>        (?P<prerelversion>\d+(?:\.\d+)*)
>>    )?
>>    (?P<postdev>
>>        ([- .]*post[- .]*(?P<post>\d+))?
>>        ([- .]*dev[- .]*(?P<dev>\d+))?
>>    )?
>>    $''', re.VERBOSE + re.I)
>> 
>> The pre-release marker would then be interpreted in alphabetical
>> order, ie. 'alpha' < 'beta' < 'rc'.
>> 
>> This minor change would broaden the scope of the scheme somewhat
>> and make it more compatible to what's being used outside the
>> python-dev sphere (esp. with respect to 'c' standing for release
>> candidate... unless you happen to read it as gamma ;-).
> 
> ... and even python-dev has switched to "rc" for these:

While I like the use of "rc" for release candidate, doesn't this lead to some ambiguity where a,b,c are seen as interim releases with "rc" sorting after all of these?

I don't have an objection to [a-q] OR 'rc' if we're reserving "rc" for release candidate and only "rc" for that purpose.

Otherwise, the sorting becomes ambiguous so if "c" means candidate then "rc" can't also or we don't know, for sure, that blah-rc comes AFTER blah-c; it's ambiguous.

S


From chris at simplistix.co.uk  Wed Nov 25 20:58:48 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 25 Nov 2009 19:58:48 +0000
Subject: [Distutils] new metadata fields to aid PyPI...
In-Reply-To: <94bdd2610911250935v6c315c35i36a842eecf05c5f3@mail.gmail.com>
References: <4B0D427D.5090802@simplistix.co.uk>
	<hejgrl$enh$1@ger.gmane.org>	<4B0D5FAE.2020809@v.loewis.de>
	<hejo2m$aec$1@ger.gmane.org>
	<94bdd2610911250935v6c315c35i36a842eecf05c5f3@mail.gmail.com>
Message-ID: <4B0D8C78.2010001@simplistix.co.uk>

Tarek Ziad? wrote:
> Note that I have introduced three new fields in PEP 345 to help on this (after
> I have read Catalog-SIG threads on the commenting issues -- someone
> suggested them):
> 
> http://www.python.org/dev/peps/pep-0345
> 
> # Repository-URL
> # Repository-Browser-URL
> # Bug-Tracker-URL

I wonder use Repository-Browser-URL but I would like to see:

Documentation-URL
MailingList-URL
ChangeLog-URL

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From mal at egenix.com  Wed Nov 25 21:04:51 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 25 Nov 2009 21:04:51 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>	<4B0D698B.4080708@egenix.com>
	<4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
Message-ID: <4B0D8DE3.1070706@egenix.com>

Tarek Ziad? wrote:
> 2009/11/25 M.-A. Lemburg <mal at egenix.com>:
>> M.-A. Lemburg wrote:
>>> Could we extend the pseudo-format of the versions a little to also
>>> include variants which use more than just one character and also
>>> allow hyphens and spaces to be used for additional clarity ?
>>>
>>> VERSION_RE = re.compile(r'''
>>>     ^
>>>     (?P<version>\d+\.\d+)          # minimum 'N.N'
>>>     (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
>>>     (?:
>>>         [- .]*
>>>         (?P<prerel>[a-z]+)        # 'a'=alpha, 'b'=beta, 'c'=release candidate
>>>         [- .]*
>>>         (?P<prerelversion>\d+(?:\.\d+)*)
>>>     )?
>>>     (?P<postdev>
>>>         ([- .]*post[- .]*(?P<post>\d+))?
>>>         ([- .]*dev[- .]*(?P<dev>\d+))?
>>>     )?
>>>     $''', re.VERBOSE + re.I)
>>>
>>> The pre-release marker would then be interpreted in alphabetical
>>> order, ie. 'alpha' < 'beta' < 'rc'.
>>>
>>> This minor change would broaden the scope of the scheme somewhat
>>> and make it more compatible to what's being used outside the
>>> python-dev sphere (esp. with respect to 'c' standing for release
>>> candidate... unless you happen to read it as gamma ;-).
>>
>> ... and even python-dev has switched to "rc" for these:
>>
>> http://www.python.org/dev/peps/pep-0361/
>>
>>> The added optional hypens and spaces make the versions more
>>> readable, IMHO.
> 
> The goal of the scheme is to propose a unique standard, together with
> the "suggest_rational_version" API that allow transforming an
> almost-rational version to a rational version. IOW, having several
> ways of defining separators would
> make the version scheme loose.  I have the feeling this would have a
> negative impact but I don't have a strong example in mind yet.

So perhaps this is just a misunderstanding on my part regarding
the intended use of the new standard "rational" version format.

If the "rational" format is just used internally by distutils to
compare resource versions, then why do we have to define
the format in a PEP ?

If all external version strings (e.g. from the setup() call
or PyPI) first go through suggest_rational_version(), then
why not define its supported set of version strings in the PEP
instead ?

I was under the impression that developers should be encouraged
to use the new "rational" version format directly in their
package versions - without a helper in between.

IMHO, that intent would go a much longer way, since the user
visible version strings would then follow this standard.

However, since distutils only needs to be able to compare the
versions, without actually knowing what 'alpha', 'beta'
and 'rc' mean, I don't (currently) see a reason not to allow
complete words in the version string.

> Now, suggest_rational_version is meant to transform those close-enough
> versions strings into working strings,
> 
> Example with "rc" vs "c":
> 
>>>> from verlib import suggest_rational_version
>>>> suggest_rational_version('2.4rc1')
> '2.4c1'
> 
> Meaning that a package manager could be permissive and use those
> almost-rational form with this API. So if we get
> suggest_rational_version work with hypens and spaces as you are
> suggesting, that could be a solution.
> 
> Now, about the "c" choice. "c" was preferred to "rc" here to avoid
> using "r" which has a strong Subversion connotation.

I don't believe that any developer who knows that revisions
are marked as "r123" would mistake "rc" for a revision marker ;-)

> Next, hyphens were removed because they are avoided in some systems
> like Debian or Ubuntu,  and have a very specific meaning in
> setuptools.

That's a good point.

RPM doesn't like hyphens in versions either... so forget the hyphen
and spaces idea.

I'd still like to write "1.2beta3" and "2.1rc3", though, and be
standards compliant :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Wed Nov 25 21:09:51 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 25 Nov 2009 21:09:51 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <2E14ABCB-17D9-4C57-96C3-118B012A8086@gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>	<4B0D698B.4080708@egenix.com>
	<4B0D6CE9.50306@egenix.com>
	<2E14ABCB-17D9-4C57-96C3-118B012A8086@gmail.com>
Message-ID: <4B0D8F0F.5040704@egenix.com>

ssteinerX at gmail.com wrote:
> While I like the use of "rc" for release candidate, doesn't this lead to some ambiguity where a,b,c are seen as interim releases with "rc" sorting after all of these?
> 
> I don't have an objection to [a-q] OR 'rc' if we're reserving "rc" for release candidate and only "rc" for that purpose.
> 
> Otherwise, the sorting becomes ambiguous so if "c" means candidate then "rc" can't also or we don't know, for sure, that blah-rc comes AFTER blah-c; it's ambiguous.

The meaning of the version modifiers is not really relevant here.
distutils just needs to be able to compare them and not make
mistakes in those comparisons.

Also note that distutils won't be comparing apples and oranges,
only the version numbers of a single package.

Given that package authors typically don't change their versioning
scheme within a single release cycle, there's nothing much to
worry about, IMO.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Wed Nov 25 21:26:44 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Nov 2009 21:26:44 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0D8DE3.1070706@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
Message-ID: <94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>

On Wed, Nov 25, 2009 at 9:04 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Tarek Ziad? wrote:
>> 2009/11/25 M.-A. Lemburg <mal at egenix.com>:
>>> M.-A. Lemburg wrote:
>>>> Could we extend the pseudo-format of the versions a little to also
>>>> include variants which use more than just one character and also
>>>> allow hyphens and spaces to be used for additional clarity ?
>>>>
>>>> VERSION_RE = re.compile(r'''
>>>> ? ? ^
>>>> ? ? (?P<version>\d+\.\d+) ? ? ? ? ?# minimum 'N.N'
>>>> ? ? (?P<extraversion>(?:\.\d+)*) ? # any number of extra '.N' segments
>>>> ? ? (?:
>>>> ? ? ? ? [- .]*
>>>> ? ? ? ? (?P<prerel>[a-z]+) ? ? ? ?# 'a'=alpha, 'b'=beta, 'c'=release candidate
>>>> ? ? ? ? [- .]*
>>>> ? ? ? ? (?P<prerelversion>\d+(?:\.\d+)*)
>>>> ? ? )?
>>>> ? ? (?P<postdev>
>>>> ? ? ? ? ([- .]*post[- .]*(?P<post>\d+))?
>>>> ? ? ? ? ([- .]*dev[- .]*(?P<dev>\d+))?
>>>> ? ? )?
>>>> ? ? $''', re.VERBOSE + re.I)
>>>>
>>>> The pre-release marker would then be interpreted in alphabetical
>>>> order, ie. 'alpha' < 'beta' < 'rc'.
>>>>
>>>> This minor change would broaden the scope of the scheme somewhat
>>>> and make it more compatible to what's being used outside the
>>>> python-dev sphere (esp. with respect to 'c' standing for release
>>>> candidate... unless you happen to read it as gamma ;-).
>>>
>>> ... and even python-dev has switched to "rc" for these:
>>>
>>> http://www.python.org/dev/peps/pep-0361/
>>>
>>>> The added optional hypens and spaces make the versions more
>>>> readable, IMHO.
>>
>> The goal of the scheme is to propose a unique standard, together with
>> the "suggest_rational_version" API that allow transforming an
>> almost-rational version to a rational version. IOW, having several
>> ways of defining separators would
>> make the version scheme loose. ?I have the feeling this would have a
>> negative impact but I don't have a strong example in mind yet.
>
> So perhaps this is just a misunderstanding on my part regarding
> the intended use of the new standard "rational" version format.
>
> If the "rational" format is just used internally by distutils to
> compare resource versions, then why do we have to define
> the format in a PEP ?
>
> If all external version strings (e.g. from the setup() call
> or PyPI) first go through suggest_rational_version(), then
> why not define its supported set of version strings in the PEP
> instead ?
>
> I was under the impression that developers should be encouraged
> to use the new "rational" version format directly in their
> package versions - without a helper in between.

Yes that's the idea. I was just suggesting that "suggest_rational_version" could
be used (maybe with a warning displayed) to avoid a brutal breakage in
versions supports.

I'll make it clearer in the PEP

[..]
> I'd still like to write "1.2beta3" and "2.1rc3", though, and be
> standards compliant :-)

I don't have any strong opposition into adding "rc", especially since Python
uses it too.

Now, for "alpha", "beta", I am not opposed at them either. I am just
thinking that
we should have only one way to name a pre-release tag, but I guess we can
consider that "b" is an alias for "beta"

So the pre-release tag would be one of those : a, alpha,  b, beta, r,  rc

I don't think we need to support more letters though, e.g. [a-z]

Regards
Tarek

From mal at egenix.com  Wed Nov 25 21:52:45 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 25 Nov 2009 21:52:45 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>	<4B0D698B.4080708@egenix.com>
	<4B0D6CE9.50306@egenix.com>	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
Message-ID: <4B0D991D.3020502@egenix.com>

Tarek Ziad? wrote:
> On Wed, Nov 25, 2009 at 9:04 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> I was under the impression that developers should be encouraged
>> to use the new "rational" version format directly in their
>> package versions - without a helper in between.
> 
> Yes that's the idea. I was just suggesting that "suggest_rational_version" could
> be used (maybe with a warning displayed) to avoid a brutal breakage in
> versions supports.
> 
> I'll make it clearer in the PEP

Ok.

> [..]
>> I'd still like to write "1.2beta3" and "2.1rc3", though, and be
>> standards compliant :-)
> 
> I don't have any strong opposition into adding "rc", especially since Python
> uses it too.
> 
> Now, for "alpha", "beta", I am not opposed at them either. I am just
> thinking that
> we should have only one way to name a pre-release tag, but I guess we can
> consider that "b" is an alias for "beta"
> 
> So the pre-release tag would be one of those : a, alpha,  b, beta, r,  rc
> 
> I don't think we need to support more letters though, e.g. [a-z]

Shouldn't we leave defining the meaning of those tags to the developers ?
IMHO, descriptive names are better than single letter abbreviations.

BTW: How would you name a snapshot using the scheme ?

Say you are working on an upcoming version 3.2.0 of a package and you
use the date as snapshot indicator (as opposed to some revision, which
doesn't have any meaning for an end-user).

I suppose this is the only way:

	3.2.0a0.20091125

... not very obvious :-(

If we'd allow [a-z_] (including the underscore which AFAIK doesn't
cause RPM/deb problems), this could also be written as:

	3.2.0_dev_snapshot.20091125

... much better :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Wed Nov 25 22:33:12 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 25 Nov 2009 22:33:12 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0D991D.3020502@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
Message-ID: <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>

On Wed, Nov 25, 2009 at 9:52 PM, M.-A. Lemburg <mal at egenix.com> wrote:
[..]
> If we'd allow [a-z_] (including the underscore which AFAIK doesn't
> cause RPM/deb problems), this could also be written as:
>
> ? ? ? ?3.2.0_dev_snapshot.20091125
>
> ... much better :-)

How do you sort them in that case ? And can you have snapshots of
pre-release tags ?

e.g., is this right ? :

3.2.0_dev_snapshot.20091125 < 3.2.0a_dev_snapshot.20091125 <
3.2.0rc1_dev_snapshot.20091125 < 3.2.0

From floris.bruynooghe at gmail.com  Wed Nov 25 23:29:19 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Wed, 25 Nov 2009 22:29:19 +0000
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0D991D.3020502@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>
	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>
	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
Message-ID: <20091125222919.GA22623@laurie.devork>

On Wed, Nov 25, 2009 at 09:52:45PM +0100, M.-A. Lemburg wrote:
> BTW: How would you name a snapshot using the scheme ?

> 
> Say you are working on an upcoming version 3.2.0 of a package and you
> use the date as snapshot indicator (as opposed to some revision, which
> doesn't have any meaning for an end-user).
> 
> I suppose this is the only way:
> 
> 	3.2.0a0.20091125
> 
> ... not very obvious :-(

I think it would be this: 3.2.0a0.dev20091125

But given that you came up with something else the non-obvious
argument might be valid.


> If we'd allow [a-z_] (including the underscore which AFAIK doesn't
> cause RPM/deb problems), this could also be written as:
> 
> 	3.2.0_dev_snapshot.20091125
> 
> ... much better :-)

Now this would be 3.2.0.dev20091125


Regards
Floris

From mal at egenix.com  Wed Nov 25 23:36:09 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 25 Nov 2009 23:36:09 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>	<4B0D698B.4080708@egenix.com>
	<4B0D6CE9.50306@egenix.com>	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>	<4B0D8DE3.1070706@egenix.com>	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
Message-ID: <4B0DB159.8010501@egenix.com>

Tarek Ziad? wrote:
> On Wed, Nov 25, 2009 at 9:52 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> [..]
>> If we'd allow [a-z_] (including the underscore which AFAIK doesn't
>> cause RPM/deb problems), this could also be written as:
>>
>>        3.2.0_dev_snapshot.20091125
>>
>> ... much better :-)
> 
> How do you sort them in that case ? And can you have snapshots of
> pre-release tags ?

You sort them based on the tag "_dev_snapshot". The leading underscore
causes the tag to sort before the first release tag (which would
start with a letter, like in "alpha"):

>>> "_dev_snapshot" < "alpha"
True

> e.g., is this right ? :
> 
> 3.2.0_dev_snapshot.20091125 < 3.2.0a_dev_snapshot.20091125 <
> 3.2.0rc1_dev_snapshot.20091125 < 3.2.0

Yes (*), even though, I'd probably drop the "_dev_snapshot" on these and
write:

  3.2.0_dev_snapshot.20091125
< 3.2.0a1
< 3.2.0a1.20091125
< 3.2.0rc1
< 3.2.0rc1.20091125
< 3.2.0

It's clear from the added version component that the version
points to revision after the release revision.

With the PEP format, I could write:

  3.2.0a1.dev20091125
< 3.2.0a1
< 3.2.0a1.post20091125
< 3.2.0rc1
< 3.2.0rc1.post20091125
< 3.2.0

... but I doubt that users would understand the difference
between "dev" and "post".

Furthermore, the "dev" and "post" modifiers don't follow the
same semantics as the pre-release tags.

This version would use the same semantics for both, but
looks even worse:

  3.2.0a1.0a20091125
< 3.2.0a1
< 3.2.0a1.0a20091125
< 3.2.0rc1
< 3.2.0rc1.0a20091125
< 3.2.0

I'd just drop the whole post/dev stuff and allow additional
(\.\d+) components after the pre-release tag part. One
pre-release field per version string is really enough, IMHO.

An alternative based on the same convention we use for Python
would be to use the "alpha 0" release as marker for pre-release
snapshots. The a0 "release" is then never actually released.

Using this convention, I could write:

  3.2.0a0.20091125
< 3.2.0a1
< 3.2.0a1.20091125
< 3.2.0rc1
< 3.2.0rc1.20091125
< 3.2.0

... looks good :-)

(*) with one nit: the third one needs to be
3.2.0rc_dev_snapshot_1.20091125 since the "1" in
"rc1_dev_snapshot" is not allowed.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 25 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Thu Nov 26 13:08:34 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 26 Nov 2009 13:08:34 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0DB159.8010501@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<5F039E36-E882-42C4-9437-576E4C5D33D6@gmail.com>	<94bdd2610911241240x1ce4b6daj88105cb3af81b496@mail.gmail.com>	<1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>	<4B0D698B.4080708@egenix.com>	<4B0D6CE9.50306@egenix.com>	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>	<4B0D8DE3.1070706@egenix.com>	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>	<4B0D991D.3020502@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com>
Message-ID: <4B0E6FC2.5020006@egenix.com>

Here's another take at a minimal change to the format which
includes the things we discussed, adds a few more aliases
for the "post" and "dev" markers and also adds optional
underscores for more readability.

VERSION_RE = re.compile(r'''
    ^
    (?P<version>\d+\.\d+)          # minimum 'N.N'
    (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
    (?:                            # pre-release tag
        _?
        (?P<prerel>(a|alpha|b|beta|c|rc))
        _?
        (?P<prerelversion>\d+(?:\.\d+)*)
    )?
    (?P<postdev>
        (\.(post|fix|sp)_?(?P<post>\d+))?
        (\.(dev|pre|build|nightly)_?(?P<dev>\d+))?
    )?
    $''', re.VERBOSE + re.I)

Examples:

  3.2.0a0.20091125
< 3.2.0a1
= 3.2.0_alpha_1
< 3.2.0a1.20091125
< 3.2.0rc1
= 3.2.0c1
< 3.2.0rc1.20091125
= 3.2.0_rc1.20091125
< 3.2.0
< 3.2.0.sp1
= 3.2.0.fix1
= 3.2.0.post1

One nit I have with the order of the N.N.devN version is that it is regarded
"more" than any of the pre-release tags, but less than the release itself:

  1.0a1
< 1.0rc1
< 1.0.dev456
< 1.0

IMHO, the order should be:

  1.0.dev456
< 1.0a1.dev456
< 1.0a1
< 1.0rc1.dev456
< 1.0rc1
< 1.0

since the .dev versions are really only snapshots leading up to
some release, i.e. 1.0.dev456 is a snapshot leading up to the
first pre-release of the 1.0 :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 26 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ziade.tarek at gmail.com  Thu Nov 26 18:36:35 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 26 Nov 2009 18:36:35 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0E6FC2.5020006@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
Message-ID: <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>

On Thu, Nov 26, 2009 at 1:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Here's another take at a minimal change to the format which
> includes the things we discussed, adds a few more aliases
> for the "post" and "dev" markers and also adds optional
> underscores for more readability.
>
> VERSION_RE = re.compile(r'''
> ? ?^
> ? ?(?P<version>\d+\.\d+) ? ? ? ? ?# minimum 'N.N'
> ? ?(?P<extraversion>(?:\.\d+)*) ? # any number of extra '.N' segments
> ? ?(?: ? ? ? ? ? ? ? ? ? ? ? ? ? ?# pre-release tag
> ? ? ? ?_?
> ? ? ? ?(?P<prerel>(a|alpha|b|beta|c|rc))
> ? ? ? ?_?
> ? ? ? ?(?P<prerelversion>\d+(?:\.\d+)*)
> ? ?)?
> ? ?(?P<postdev>
> ? ? ? ?(\.(post|fix|sp)_?(?P<post>\d+))?
> ? ? ? ?(\.(dev|pre|build|nightly)_?(?P<dev>\d+))?
> ? ?)?
> ? ?$''', re.VERBOSE + re.I)
>
> Examples:
>
> ?3.2.0a0.20091125
> < 3.2.0a1
> = 3.2.0_alpha_1
> < 3.2.0a1.20091125
> < 3.2.0rc1
> = 3.2.0c1
> < 3.2.0rc1.20091125
> = 3.2.0_rc1.20091125
> < 3.2.0
> < 3.2.0.sp1
> = 3.2.0.fix1
> = 3.2.0.post1
>
> One nit I have with the order of the N.N.devN version is that it is regarded
> "more" than any of the pre-release tags, but less than the release itself:
>
> ?1.0a1
> < 1.0rc1
> < 1.0.dev456
> < 1.0
>
> IMHO, the order should be:
>
> ?1.0.dev456
> < 1.0a1.dev456
> < 1.0a1
> < 1.0rc1.dev456
> < 1.0rc1
> < 1.0
>
> since the .dev versions are really only snapshots leading up to
> some release, i.e. 1.0.dev456 is a snapshot leading up to the
> first pre-release of the 1.0 :-)

That's right, that's a bug in the PEP and/or verlib.py

The changes look good to me.

If you made some changes in verlib, would you mind pushing them back at

http://bitbucket.org/tarek/distutilsversion ?

I'll update the PEP then, and if there's no more feedback here, I'll
propose it at python-dev

Regards
Tarek

From floris.bruynooghe at gmail.com  Thu Nov 26 20:55:49 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Thu, 26 Nov 2009 19:55:49 +0000
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B0E6FC2.5020006@egenix.com>
References: <1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
Message-ID: <20091126195549.GA29498@laurie.devork>

On Thu, Nov 26, 2009 at 01:08:34PM +0100, M.-A. Lemburg wrote:
> Examples:
> 
>   3.2.0a0.20091125
> < 3.2.0a1
> = 3.2.0_alpha_1

Frankly I find this confusing.  I'm fine with 'alpha' being a synonym
for 'a' but the underscores just confuse things IMHO.

> < 3.2.0a1.20091125
> < 3.2.0rc1
> = 3.2.0c1
> < 3.2.0rc1.20091125
> = 3.2.0_rc1.20091125
> < 3.2.0
> < 3.2.0.sp1
> = 3.2.0.fix1
> = 3.2.0.post1

I also object to the alternatives for the 'dev' and 'post' markers as
they make it more confusing for me.  While someone might prefer one
word over the other their meaning does not change to decide their
ordering, that just seems like uneeded complexity (there should be
only one obvious way to do a thing?).

When I now see the versioning number of a project I need to go and
look up the pep to know if it's compliant but just using one of the
alternatives that I'm not used to.  If there's only one choice it's a
lot easier.

(This same argument goes for 'a' == 'alpha' 'b' == 'beta' and 'c' ==
'rc' but those at least are mnemonic so easier to remember)


> One nit I have with the order of the N.N.devN version is that it is regarded
> "more" than any of the pre-release tags, but less than the release itself:
> 
>   1.0a1
> < 1.0rc1
> < 1.0.dev456
> < 1.0
> 
> IMHO, the order should be:
> 
>   1.0.dev456
> < 1.0a1.dev456
> < 1.0a1
> < 1.0rc1.dev456
> < 1.0rc1
> < 1.0
> 
> since the .dev versions are really only snapshots leading up to
> some release, i.e. 1.0.dev456 is a snapshot leading up to the
> first pre-release of the 1.0 :-)

But in this case if I want to make a pre-release of 1.0 but after the
last rc then I can't, I can only make a post-release of the last rc.
That's almost more un-intuitive that forcing your first pre-release to
be '1.0a0.dev456' instead of just '1.0.dev456'.



Regards
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From ziade.tarek at gmail.com  Thu Nov 26 21:45:48 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 26 Nov 2009 21:45:48 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091126195549.GA29498@laurie.devork>
References: <1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
Message-ID: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>

On Thu, Nov 26, 2009 at 8:55 PM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
[..]
>> since the .dev versions are really only snapshots leading up to
>> some release, i.e. 1.0.dev456 is a snapshot leading up to the
>> first pre-release of the 1.0 :-)
>
> But in this case if I want to make a pre-release of 1.0 but after the
> last rc then I can't, I can only make a post-release of the last rc.
> That's almost more un-intuitive that forcing your first pre-release to
> be '1.0a0.dev456' instead of just '1.0.dev456'.

It seems to me that the number of development versions of rc releases
is very low compared to the number of development snapshots done for
1.0, before the pre-release cycle starts.

(I don't think I have ever needed a dev snapshot of a rc version)

I am +1 for keeping the intuitive writing for the pre-release cycle.

e.g.

 1.0.dev456
 < 1.0a1.dev456
 < 1.0a1
 < 1.0rc1.dev456
 < 1.0rc1
 < 1.0rc1.post123
 < 1.0

Regards,
Tarek

From ziade.tarek at gmail.com  Thu Nov 26 21:46:30 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 26 Nov 2009 21:46:30 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
References: <1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
Message-ID: <94bdd2610911261246s78276f07qbb7490d41e24e4d0@mail.gmail.com>

2009/11/26 Tarek Ziad? <ziade.tarek at gmail.com>:

> I am +1 for keeping the intuitive writing for the pre-release cycle.

s/for the pre-release cycle./before the pre-release cycle starts/

From alerades at gmail.com  Thu Nov 26 22:58:35 2009
From: alerades at gmail.com (Alex Rades)
Date: Thu, 26 Nov 2009 16:58:35 -0500
Subject: [Distutils] zc.buildout usage in the real world
Message-ID: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>

Hi,

I'm currently working on a project based on django, together with a
dozen of external libraries. Often I am in one of these scenarios:

- I discover bugs in an external libraries, and i fix them, but the
upstream author is busy and I need to deploy the fix ASAP. So I
produce a patch.
- I have to install a particular revision of an external library from git/svn/hg

The approach I've used so far has been setting up a private
pypi-compatible repository, and recreate packages which fit my needs.
But it really feels like killing an ant with a sledgehammer. I don't
understand if there is a better way to:

- Patch packages before installing
- Fetch packages via git/svn/hg or via direct http link

I'm currently using the *great* minitage.recipe.scripts recipe to do
all of this, and infact i can do something like:
----------------------------
[complex-deps]
interpreter = python
recipe = minitage.recipe.scripts
urls =
  http://media.djangoproject.com/releases/1.1.1/Django-1.1.1.tar.gz
  http://github.com/robhudson/django-debug-toolbar/commit/f36b72655349cf3b8e9105bed3be5cd065162476

Django-1.1.1-patches = ${buildout:directory}/patches/django-admin-fixes.diff
debug_toolbar-patches =
${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
-----------------------------

Which is very very handy, it handles all my use case in a compact and
nice way. The minitage.recipe.scripts is nice but is not widely used,
so I'm a bit worried about its future and possible breakages. You know
- when you work on a large project you have to think hard about your
dependencies. It's like getting married.

So the question is: Are there alternative solutions? Am I the only one
with such needs?

Thank you

From chris at simplistix.co.uk  Thu Nov 26 23:36:55 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 26 Nov 2009 22:36:55 +0000
Subject: [Distutils] zc.buildout usage in the real world
In-Reply-To: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
References: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
Message-ID: <4B0F0307.9000009@simplistix.co.uk>

Alex Rades wrote:
> The approach I've used so far has been setting up a private
> pypi-compatible repository, and recreate packages which fit my needs.
> But it really feels like killing an ant with a sledgehammer. I don't
> understand if there is a better way to:
> 
> - Patch packages before installing
> - Fetch packages via git/svn/hg or via direct http link

What I do is check out the offending package, tweak the version in their 
setup.py to include the identifier of the revision I checked out, then 
do "python setup.py sdist" and put that sdist into my "egg server".

If I needed to apply patches that the packager won't accept (hasn't 
happened yet!), I'd do an export of the upstream package into my source 
control system, apply the patches and then do the process I've already 
described.

My "egg server" is just an authentication-required folder on an https 
web server, I use lovely.buildouthttp and a find-links to make this work.

This is actually even quicker and simpler than it looks.

> [complex-deps]
> interpreter = python
> recipe = minitage.recipe.scripts
> urls =
>   http://media.djangoproject.com/releases/1.1.1/Django-1.1.1.tar.gz
>   http://github.com/robhudson/django-debug-toolbar/commit/f36b72655349cf3b8e9105bed3be5cd065162476
> 
> Django-1.1.1-patches = ${buildout:directory}/patches/django-admin-fixes.diff
> debug_toolbar-patches =
> ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff

This kind of patching makes me nervous...

> Which is very very handy, it handles all my use case in a compact and
> nice way. The minitage.recipe.scripts is nice but is not widely used,

...but then minitage as a whole makes me nervous, I just don't trust it, 
but I'm afraid I couldn't tell you why. Gut feeling...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From kevin at bud.ca  Thu Nov 26 23:52:36 2009
From: kevin at bud.ca (Kevin Teague)
Date: Thu, 26 Nov 2009 14:52:36 -0800 (PST)
Subject: [Distutils] zc.buildout usage in the real world
In-Reply-To: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
References: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
Message-ID: <b7b56232-c4b9-41d0-b4cf-173b3fc1ab56@j9g2000prh.googlegroups.com>

mr.developer is a good recipe that could help with this, check it out:

http://pypi.python.org/pypi/mr.developer

The buildout below will auto-checkout django-debug-toolbar from your
fork when buildout is run (replace 'myapp' with your apps name of
course, and have django-debug-toolbar in the install_requires field):

[buildout]
parts = app
develop = .
extensions = mr.developer
auto-checkout = django-debug-toolbar

[sources]
django-debug-toolbar = git git://github.com/robhudson/django-debug-toolbar.git

[app]
recipe = zc.recipe.egg
eggs = myapp


From pje at telecommunity.com  Fri Nov 27 02:41:04 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Thu, 26 Nov 2009 20:41:04 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091126195549.GA29498@laurie.devork>
References: <1316C3F4-F5C5-42F8-A18D-F2C4EE79EB2B@gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
Message-ID: <20091127014114.1A1B23A40A2@sparrow.telecommunity.com>

At 07:55 PM 11/26/2009 +0000, Floris Bruynooghe wrote:
>On Thu, Nov 26, 2009 at 01:08:34PM +0100, M.-A. Lemburg wrote:
> > Examples:
> >
> >   3.2.0a0.20091125
> > < 3.2.0a1
> > = 3.2.0_alpha_1
>
>Frankly I find this confusing.  I'm fine with 'alpha' being a synonym
>for 'a' but the underscores just confuse things IMHO.

They're also not backwards-compatible with the current schemes being 
used by setuptools and its derivatives (Distribute, pip, buildout, 
etc.), which interprets underscores differently.

(Technically, an underscore should never be in a version, since 
underscores are used in filenames as an escaped version of a dash.)


From kiorky at cryptelium.net  Fri Nov 27 08:55:59 2009
From: kiorky at cryptelium.net (kiorky)
Date: Fri, 27 Nov 2009 08:55:59 +0100
Subject: [Distutils] zc.buildout usage in the real world
In-Reply-To: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
References: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
Message-ID: <4B0F860F.809@cryptelium.net>



Alex Rades a ?crit :
> Hi,
> 

> 
> Which is very very handy, it handles all my use case in a compact and
> nice way. The minitage.recipe.scripts is nice but is not widely used,

But supported by a company which will do not switch for another recipes.
We deploy on various systems, that's why we created minitage both as sysadmins
and developers efforts.

> so I'm a bit worried about its future and possible breakages. 

We have a tracker, mails & irc and we don't like to have spending bugs for a
long time ;)

You know
> - when you work on a large project you have to think hard about your
> dependencies. It's like getting married.
> 

If minitage is dead, all our current deployments (lot of projects) there will be
in bad shape. Just a word to say that you ll have years to see it dead.

> So the question is: Are there alternative solutions? Am I the only one
> with such needs?
This is not only the way to do, and i told you the alternatives like doing with
mr.developer.
That depend on your feelings and habits and use cases.

I like very much both my recipes and mr.developer,
When i need to apply a small change, mostly for upstream approval, i go for
minitage.recipe.scripts. Or for a single patch without hacking, (ZODDB +
RelStorage, for example), i go also for minitage.recipe.scripts.
But for more deeper hacking, i'd go for mr.developer.*

After that, there are much more things in minitage.recipe.* that zc.equivalent,
that i can't do without forking them and that's that i ve done that, some years
ago. Now their code is too much different, and it was not a primary goal too, to
merge back. Specially for minitage integration, even if you are not tied with
minitage by using only the recipes, i don't think the code integrating it
belongs to any zc.recipe.*.

> 
> Thank you
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig

-- 
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091127/295d5d05/attachment.pgp>

From kiorky at cryptelium.net  Fri Nov 27 08:59:58 2009
From: kiorky at cryptelium.net (kiorky)
Date: Fri, 27 Nov 2009 08:59:58 +0100
Subject: [Distutils] zc.buildout usage in the real world
In-Reply-To: <4B0F0307.9000009@simplistix.co.uk>
References: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
	<4B0F0307.9000009@simplistix.co.uk>
Message-ID: <4B0F86FE.3020001@cryptelium.net>

Chris Withers a ?crit :
> Alex Rades wrote:
> 
> This is actually even quicker and simpler than it looks.
> 
>> [complex-deps]
>> interpreter = python
>> recipe = minitage.recipe.scripts
>> urls =
>>   http://media.djangoproject.com/releases/1.1.1/Django-1.1.1.tar.gz
>>   http://github.com/robhudson/django-debug-toolbar/commit/f36b72655349cf3b8e9105bed3be5cd065162476
>> Django-1.1.1-patches =
>> ${buildout:directory}/patches/django-admin-fixes.diff
>> debug_toolbar-patches =
>> ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
> 

> This kind of patching makes me nervous...
why ? patches are a common way to 'patch' code.

> 
>> Which is very very handy, it handles all my use case in a compact and
>> nice way. The minitage.recipe.scripts is nice but is not widely used,
> 
> ...but then minitage as a whole makes me nervous, I just don't trust it,
> but I'm afraid I couldn't tell you why. Gut feeling...

Do not spread your personal fud, Chris. Moreover without arguments.

> 
> Chris
> 

-- 
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091127/9d37a6c4/attachment.pgp>

From chris at simplistix.co.uk  Fri Nov 27 09:22:41 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 27 Nov 2009 08:22:41 +0000
Subject: [Distutils] zc.buildout usage in the real world
In-Reply-To: <4B0F86FE.3020001@cryptelium.net>
References: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
	<4B0F0307.9000009@simplistix.co.uk>
	<4B0F86FE.3020001@cryptelium.net>
Message-ID: <4B0F8C51.3020401@simplistix.co.uk>

kiorky wrote:
>>> ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
> 
>> This kind of patching makes me nervous...
> why ? patches are a common way to 'patch' code.

Maybe 10 years ago. With a DVCS, or even a non-d VCS for that matter, 
why would you need to apply patches like this, which would need to be 
re-done with every release of the software you're patching, when you can 
let the (D)VCS do the hard work and cut your own sdist that has any 
patching done already the way you want it?

>> ...but then minitage as a whole makes me nervous, I just don't trust it,
>> but I'm afraid I couldn't tell you why. Gut feeling...
> 
> Do not spread your personal fud, Chris. Moreover without arguments.

*shrugs* Everyone's entitled to their own opinion...

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From kiorky at cryptelium.net  Fri Nov 27 09:34:53 2009
From: kiorky at cryptelium.net (kiorky)
Date: Fri, 27 Nov 2009 09:34:53 +0100
Subject: [Distutils] zc.buildout usage in the real world
In-Reply-To: <4B0F8C51.3020401@simplistix.co.uk>
References: <ea7358da0911261358q3961161ub424d52a66eb76d2@mail.gmail.com>
	<4B0F0307.9000009@simplistix.co.uk>
	<4B0F86FE.3020001@cryptelium.net>
	<4B0F8C51.3020401@simplistix.co.uk>
Message-ID: <4B0F8F2D.8050306@cryptelium.net>



Chris Withers a ?crit :
> kiorky wrote:
>>>> ${buildout:directory}/patches/debug_toolbar_sqlpanel.diff
>>
>>> This kind of patching makes me nervous...
>> why ? patches are a common way to 'patch' code.
> 
> Maybe 10 years ago. With a DVCS, or even a non-d VCS for that matter,
> why would you need to apply patches like this, which would need to be
> re-done with every release of the software you're patching, when you can
> let the (D)VCS do the hard work and cut your own sdist that has any
> patching done already the way you want it?

Just let everybody do what he want like he want.
I prefer to use forks, but sometime, it is just too much for a single patch.
More over if the patch is already distributed as-is an i just need to use it
without modifications.

[part]
Foo-patch=/patch

is quickier that

SCM clone /
sed "s/version/specialversion/g" -i setup.py
patch < patch
SCM ci -m azer
SCM push /somewhere
python setup.py sdist
upload /somewhere

verify the url
make it indexed somehow

[part]
eggs=myegg


And making special dists everywhere make deployments tedious and add to you the
job to keep your house clean and to know what is where. With a lot of packages,
it is not that simple as integrating a single patch applied to a buildout which
will generate under the hood an egg with a special name which will be never used
even if you drop it in a shared cache unless you use the same patch in another
project (minitage.recipe generates an egg with a special version, when you
decide to install it with a patch, that s allow to use both the official egg and
patched eggs in the same sharedcache).

> 
>>> ...but then minitage as a whole makes me nervous, I just don't trust it,
>>> but I'm afraid I couldn't tell you why. Gut feeling...
>>
>> Do not spread your personal fud, Chris. Moreover without arguments.
> 
> *shrugs* Everyone's entitled to their own opinion...

The fact that you are not knowing or understanding  what minitage is or what it
does or allow is not a reason to say something may not be a good choice to think on.

More over minitage "are":
	- paster templates (minitage.paste.*)
	- collection of recipes which also integrate system dependencies (making CFLAGS
and LDFLAGS pointing to your dependencies, for example.) (minitage.recipe.*).
	- a meta-package-manager with goal to isolate dependencies and projects from
the system and install them in a well know layout (minitage.core).

And there is http://minitage.org  + Pypi pages as a documentation.



> 
> Chris
> 

-- 
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091127/087d748b/attachment-0001.pgp>

From ozarow at gmail.com  Fri Nov 27 11:39:58 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Fri, 27 Nov 2009 11:39:58 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
Message-ID: <20091127103958.GU2966@piotro.eu>

[Tarek Ziad?, 2009-11-26]
> On Thu, Nov 26, 2009 at 8:55 PM, Floris Bruynooghe
> <floris.bruynooghe at gmail.com> wrote:
> [..]
> >> since the .dev versions are really only snapshots leading up to
> >> some release, i.e. 1.0.dev456 is a snapshot leading up to the
> >> first pre-release of the 1.0 :-)
> >
> > But in this case if I want to make a pre-release of 1.0 but after the
> > last rc then I can't, I can only make a post-release of the last rc.
> > That's almost more un-intuitive that forcing your first pre-release to
> > be '1.0a0.dev456' instead of just '1.0.dev456'.
> 
> It seems to me that the number of development versions of rc releases
> is very low compared to the number of development snapshots done for
> 1.0, before the pre-release cycle starts.
> 
> (I don't think I have ever needed a dev snapshot of a rc version)
> 
> I am +1 for keeping the intuitive writing for the pre-release cycle.
> 
> e.g.
> 
>  1.0.dev456
>  < 1.0a1.dev456
>  < 1.0a1
>  < 1.0rc1.dev456
>  < 1.0rc1
>  < 1.0rc1.post123
>  < 1.0

why not simply use "-" and "+" where "-" is before zero-length string
and "+" is after any other string... and then sort the rest
alphabetically? f.e.

  1.0-a1-dev456
< 1.0-a1
< 1.0-a1+dev456
< 1.0-dev456
< 1.0-rc1-dev456
< 1.0-rc1
< 1.0-rc1+post123
< 1.0
< 1.0+post123

don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
and "+" right now[0]). I'm not sure about rpm, but I bet it has
something similar and it will be much easier for us to simply handle two
characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...

[0] dpkg --compare-versions '1.0~a1~dev456' '<=' '1.0~a1+dev456' && echo true || echo false
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From chris at simplistix.co.uk  Fri Nov 27 13:05:28 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Fri, 27 Nov 2009 12:05:28 +0000
Subject: [Distutils] The "incompatible versions" problem
Message-ID: <4B0FC088.50008@simplistix.co.uk>

Hi All,

Do you guys on this lists have any ideas how packages can indicate to 
package managers like setuptools could indicate that if they are 
required by another package, that requirement must be for greater than a 
certain version?

This stems from the "backwards compatibility" discussion for 
zope.interface here:

https://mail.zope.org/pipermail/zope-dev/2009-November/038417.html

Off the top of my head, zope.component could spell it's requirement as:

def setup(
   ...
   incompatible_with='<4.0'
   )

Whereby, setuptools would raise an exception if the following packages 
were installed:

def setup(
   ...
   install_requires=['zope.component']
   )

def setup(
   ...
   install_requires=['zope.component >= 3.4.1']
   )

...while it wouldn't for the following package:

def setup(
   ...
   install_requires=['zope.component >= 4']
   )

Ideas?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From ziade.tarek at gmail.com  Fri Nov 27 13:24:14 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 27 Nov 2009 13:24:14 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091127103958.GU2966@piotro.eu>
References: <4B0D6CE9.50306@egenix.com> <4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
Message-ID: <94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>

On Fri, Nov 27, 2009 at 11:39 AM, Piotr Ozarowski <ozarow at gmail.com> wrote:
> [Tarek Ziad?, 2009-11-26]
>> On Thu, Nov 26, 2009 at 8:55 PM, Floris Bruynooghe
>> <floris.bruynooghe at gmail.com> wrote:
>> [..]
>> >> since the .dev versions are really only snapshots leading up to
>> >> some release, i.e. 1.0.dev456 is a snapshot leading up to the
>> >> first pre-release of the 1.0 :-)
>> >
>> > But in this case if I want to make a pre-release of 1.0 but after the
>> > last rc then I can't, I can only make a post-release of the last rc.
>> > That's almost more un-intuitive that forcing your first pre-release to
>> > be '1.0a0.dev456' instead of just '1.0.dev456'.
>>
>> It seems to me that the number of development versions of rc releases
>> is very low compared to the number of development snapshots done for
>> 1.0, before the pre-release cycle starts.
>>
>> (I don't think I have ever needed a dev snapshot of a rc version)
>>
>> I am +1 for keeping the intuitive writing for the pre-release cycle.
>>
>> e.g.
>>
>> ?1.0.dev456
>> ?< 1.0a1.dev456
>> ?< 1.0a1
>> ?< 1.0rc1.dev456
>> ?< 1.0rc1
>> ?< 1.0rc1.post123
>> ?< 1.0
>
> why not simply use "-" and "+" where "-" is before zero-length string
> and "+" is after any other string... and then sort the rest
> alphabetically? f.e.
>
> ?1.0-a1-dev456
> < 1.0-a1
> < 1.0-a1+dev456
> < 1.0-dev456
> < 1.0-rc1-dev456
> < 1.0-rc1
> < 1.0-rc1+post123
> < 1.0
> < 1.0+post123
>
> don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
> and "+" right now[0]). I'm not sure about rpm, but I bet it has
> something similar and it will be much easier for us to simply handle two
> characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...

It's different from RPMs, since they use a strcmp(), segment  by segment,
so I think they have to extract the dev/post suffixes and to put them in front
as an epoch marker maybe ? (ccing Toshio)

Meaning that they would need to split on "-", and replace "+" by "z",
and use the segments in reverse order before comparing the first
Major.Minor.Micro part,

# rmpdev-vercmp "1.0-a1-dev456"   "1.0"
0:1.0-a1-dev456 is newer

# rmpdev-vercmp "1.0~a1"   "1.0-a1"
0:1.0~a1-None is newer

# rmpdev-vercmp "1.0-a1-2"   "1.0-a1~2"
These are equals

>From rmp5.org:
"""
The algorithm that RPM uses to determine the version ordering of
packages is simple and developers are encouraged not to rely on the
details of its working. Developers should keep their numbering scheme
simple so any reasonable ordering algorithm would work. The version
comparison algorithm is in the routine rpmvercmp() and it is just a
segmented strcmp(3). First, the boundaries of the segments are found
using isdigit(3)/isalpha(3). Each segment is then compared in order
with the right most segment being the least significant. The
alphabetical portions are compared using a lexical graphical ascii
ordering, the digit segments strip leading zeroes and compare the
strlen before doing a strcmp. If both numerical strings are equal, the
longer string is larger. Notice that the algorithm has no knowledge of
decimal fractions, and perl-5.6 is "older" than perl-5.00503 because
the number 6 is less than the number 503.
"""

Tarek

From ziade.tarek at gmail.com  Fri Nov 27 13:31:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 27 Nov 2009 13:31:32 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
Message-ID: <94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>

On Fri, Nov 27, 2009 at 1:24 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
[..]
>> don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
>> and "+" right now[0]). I'm not sure about rpm, but I bet it has
>> something similar and it will be much easier for us to simply handle two
>> characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...
>
> It's different from RPMs, since they use a strcmp(), segment ?by segment,
> so I think they have to extract the dev/post suffixes and to put them in front
> as an epoch marker maybe ? (ccing Toshio)
>

That makes me think that a nice add-on to the lib and the PEP would be
to provide APIs
to translate a Python PEP 386 version to a Debian/Ubuntu or RPM ones -
and any major
packaging system out there. (whatever scheme we pick)

From ben+python at benfinney.id.au  Fri Nov 27 13:52:47 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 27 Nov 2009 23:52:47 +1100
Subject: [Distutils] Common version-comparison semantics for peace love and
	harmony (was: PEP 386 status - last round here ?)
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
Message-ID: <873a40xh7k.fsf_-_@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> That makes me think that a nice add-on to the lib and the PEP would be
> to provide APIs to translate a Python PEP 386 version to a
> Debian/Ubuntu or RPM ones - and any major packaging system out there.
> (whatever scheme we pick)

I'd like to register, once again, the point that this would not *be* a
problem if PEP 386 described a version comparison scheme that simply
works without special keywords. Have each segment compared
alphanumerically, and it will not *need* translation to work with other
packaging systems.

Special keywords are not, I maintain, special enough to break the normal
version-comparison semantics.

-- 
 \      ?I would rather be exposed to the inconveniences attending too |
  `\      much liberty than those attending too small a degree of it.? |
_o__)                                                ?Thomas Jefferson |
Ben Finney


From ozarow at gmail.com  Fri Nov 27 14:00:57 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Fri, 27 Nov 2009 14:00:57 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
References: <94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
Message-ID: <20091127130057.GV2966@piotro.eu>

[Tarek Ziad?, 2009-11-27]
> On Fri, Nov 27, 2009 at 1:24 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> [..]
> >> don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
> >> and "+" right now[0]). I'm not sure about rpm, but I bet it has
> >> something similar and it will be much easier for us to simply handle two
> >> characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...
> >
> > It's different from RPMs, since they use a strcmp(), segment ?by segment,
> > so I think they have to extract the dev/post suffixes and to put them in front
> > as an epoch marker maybe ? (ccing Toshio)

We use that as well. Epoch cannot be used here, though (IMO). Unless you're
talking about segment's epoch, not the whole version one, but then, "-"
and "+" will be still much easier for rpm. Let's wait for Toshio's
reply...

> That makes me think that a nice add-on to the lib and the PEP would be
> to provide APIs
> to translate a Python PEP 386 version to a Debian/Ubuntu or RPM ones -
> and any major
> packaging system out there. (whatever scheme we pick)

that would be nice, but we still have to implement it on our side, f.e. in
tools like uscan (or in its configuration files: debian/watch). uscan
scans upstream repositories (like PyPi) for new tarballs, then uses
regular expressions to change the versions, compares them and
downloads/renames/notifies maintainer. My point is: it's easier to
feed uscan with s/-/~/g instead of s/rc/~rc/;s/\<c/~rc/g;s/\<a/~a/g;...
but it's not a very big problem - we handle much uglier upstream version
shemas ;-). If all pypi packages would use the same rules, it would
allow us to handle it in uscan instead of in per package basis
(debian/watch files), though.
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From ziade.tarek at gmail.com  Fri Nov 27 14:30:12 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 27 Nov 2009 14:30:12 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony (was: PEP 386 status - last round here ?)
In-Reply-To: <873a40xh7k.fsf_-_@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
Message-ID: <94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>

On Fri, Nov 27, 2009 at 1:52 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Tarek Ziad? <ziade.tarek at gmail.com> writes:
>
>> That makes me think that a nice add-on to the lib and the PEP would be
>> to provide APIs to translate a Python PEP 386 version to a
>> Debian/Ubuntu or RPM ones - and any major packaging system out there.
>> (whatever scheme we pick)
>
> I'd like to register, once again, the point that this would not *be* a
> problem if PEP 386 described a version comparison scheme that simply
> works without special keywords. Have each segment compared
> alphanumerically, and it will not *need* translation to work with other
> packaging systems.
>
> Special keywords are not, I maintain, special enough to break the normal
> version-comparison semantics.

Not sure what you mean by special keywords. If it's about 'post' and
'dev', the problem we have is to be able to sort those post and pre
releases with other versions.  And this is not possible with a simple
alphanumerical
comparison.

And this is not a special use case, and has to be part imho of our
version-comparison semantics.
otherwise, our scheme would be useless for developers that work as a
daily basis with snapshots,
dev versions etc.

Tarek

From ssteinerx at gmail.com  Fri Nov 27 15:31:59 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Fri, 27 Nov 2009 09:31:59 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
Message-ID: <9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>


On Nov 27, 2009, at 7:31 AM, Tarek Ziad? wrote:

> On Fri, Nov 27, 2009 at 1:24 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> [..]
>>> don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
>>> and "+" right now[0]). I'm not sure about rpm, but I bet it has
>>> something similar and it will be much easier for us to simply handle two
>>> characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...
>> 
>> It's different from RPMs, since they use a strcmp(), segment  by segment,
>> so I think they have to extract the dev/post suffixes and to put them in front
>> as an epoch marker maybe ? (ccing Toshio)
>> 
> That makes me think that a nice add-on to the lib and the PEP would be
> to provide APIs to translate a Python PEP 386 version to a Debian/Ubuntu or RPM ones -
> and any major packaging system out there. (whatever scheme we pick)

Wouldn't it be cool if the package that goes along with this PEP became the standard version checker used by ALL of these distributions?

It's not like they don't have Python interpreters around...

What would that API have to look like?  Maybe start a doc in the repo to hold that spec as we develop it?

I'm still very interested in the increment_version functionality we talked about earlier so that we could have our build process automatically up our release version numbers so we have a standard way of maintaining incremental versioning.

S




From ziade.tarek at gmail.com  Fri Nov 27 15:41:02 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 27 Nov 2009 15:41:02 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>
Message-ID: <94bdd2610911270641w35930f39mc0ee06a1cf5256f3@mail.gmail.com>

On Fri, Nov 27, 2009 at 3:31 PM, ssteinerX at gmail.com
<ssteinerx at gmail.com> wrote:
>
> On Nov 27, 2009, at 7:31 AM, Tarek Ziad? wrote:
>
>> On Fri, Nov 27, 2009 at 1:24 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>> [..]
>>>> don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
>>>> and "+" right now[0]). I'm not sure about rpm, but I bet it has
>>>> something similar and it will be much easier for us to simply handle two
>>>> characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...
>>>
>>> It's different from RPMs, since they use a strcmp(), segment ?by segment,
>>> so I think they have to extract the dev/post suffixes and to put them in front
>>> as an epoch marker maybe ? (ccing Toshio)
>>>
>> That makes me think that a nice add-on to the lib and the PEP would be
>> to provide APIs to translate a Python PEP 386 version to a Debian/Ubuntu or RPM ones -
>> and any major packaging system out there. (whatever scheme we pick)
>
> Wouldn't it be cool if the package that goes along with this PEP became the standard version checker used by ALL of these distributions?
>
> It's not like they don't have Python interpreters around...

Piotr makes the point that every packaging system has already its
recipes and own tools, that are not Python-based for some of them. But
it doesn't hurt to provide something on our side too it I think.

>
> What would that API have to look like? ?Maybe start a doc in the repo to hold that spec as we develop it?

The PEP 386 verlib API is ending in Distutils anyways, so third party
package managers will be able to
use verlib in their tools if they want.

So once the version scheme is decided, we can talk about extending the
lib to add more functions besides "suggest_rational_version". The
question will be to decide if it's stable enough to have it Distutils.

But that won't be a PEP discussion anymore, e.g. just candy on the top
of the version scheme.

>
> I'm still very interested in the increment_version functionality we talked about earlier so that we could have our build process automatically up our release version numbers so we have a standard way of maintaining incremental versioning.

That would be an interesting feature

Tarek

From ozarow at gmail.com  Fri Nov 27 15:58:08 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Fri, 27 Nov 2009 15:58:08 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>
References: <4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>
Message-ID: <20091127145808.GW2966@piotro.eu>

[ssteinerX at gmail.com, 2009-11-27]
> > That makes me think that a nice add-on to the lib and the PEP would be
> > to provide APIs to translate a Python PEP 386 version to a Debian/Ubuntu or RPM ones -
> > and any major packaging system out there. (whatever scheme we pick)
> 
> Wouldn't it be cool if the package that goes along with this PEP
> became the standard version checker used by ALL of these
> distributions?

used where? In Debian we don't touch upstream code in order to adjust it
to our version schema, so although package version is 4:0.1~alpha1+r23-5
module.__version__ still returns 0.1alpha1 or 0.1a.svn23. It only
matters for tools like uscan that look for new upstream releases and
prepare tarball names for us.

Please also note that such translations can only be a hint, you cannot
rely on it. Imagine situation where upstream author changes version
schema from YYYYMMDD to X.Y.Z, we simply bump epoch so after 20091127-1
comes 1:5.6.7-1, how would you know that epoch bump was needed without
knowing the history of the package?

We cannot really trust versions in requires.txt/install_requires/etc.
either (so there's no need to translate them) as we have to double check
these versions in most modules anyway (they're usually too low, too
high or simply missing).
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

From ssteinerx at gmail.com  Fri Nov 27 16:03:23 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Fri, 27 Nov 2009 10:03:23 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911270641w35930f39mc0ee06a1cf5256f3@mail.gmail.com>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>
	<94bdd2610911270641w35930f39mc0ee06a1cf5256f3@mail.gmail.com>
Message-ID: <DC7B1033-6B76-48FD-A517-57E7865E00F3@gmail.com>


On Nov 27, 2009, at 9:41 AM, Tarek Ziad? wrote:
>> I'm still very interested in the increment_version functionality we talked about earlier so that we could have our build process automatically up our release version numbers so we have a standard way of maintaining incremental versioning.
> 
> That would be an interesting feature

I think I'll start it a separate project, calling into the API, that can just be put into any build process.

If it works well enough stand-alone, maybe we'll incorporate it later.

S


From ssteinerx at gmail.com  Fri Nov 27 16:16:17 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Fri, 27 Nov 2009 10:16:17 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091127145808.GW2966@piotro.eu>
References: <4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<9B98CAD7-68FE-41F7-B4C8-DCD1CCD2DAAC@gmail.com>
	<20091127145808.GW2966@piotro.eu>
Message-ID: <6256CBFC-1D93-450F-BECC-7195A42F9D46@gmail.com>


On Nov 27, 2009, at 9:58 AM, Piotr Ozarowski wrote:

> [ssteinerX at gmail.com, 2009-11-27]
>>> That makes me think that a nice add-on to the lib and the PEP would be
>>> to provide APIs to translate a Python PEP 386 version to a Debian/Ubuntu or RPM ones -
>>> and any major packaging system out there. (whatever scheme we pick)
>> 
>> Wouldn't it be cool if the package that goes along with this PEP
>> became the standard version checker used by ALL of these
>> distributions?
> 
> used where?

I'm thinking of a tool, callable within *any* build process, for verifying that the version conforms to the "RationalVersion" scheme, for example.  

Especially, I'm interested in a tool that would automatically increment version numbers for e.g. in process development versions where each build needs a new version number (maybe in several places e.g. setup.py, documentation, history file, etc.).

S


From a.badger at gmail.com  Fri Nov 27 20:55:08 2009
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Fri, 27 Nov 2009 11:55:08 -0800
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
References: <4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
Message-ID: <20091127195508.GE9447@clingman.lan>

Many many things in this thread.... Unfortunately, there's so many ways to
do versioning that it's almost a bikeshed topic and so there's a lot of
different ideas of what could be right.  Let me start by saying what I think
is "right" and then the rest of my message will be devoted to what things
seem like good compromises and what don't :-)

I'm in agreement with Ben Finney's idea::
"""
I'd like to register, once again, the point that this would not *be* a
problem if PEP 386 described a version comparison scheme that simply
works without special keywords. Have each segment compared
alphanumerically, and it will not *need* translation to work with other
packaging systems.

Special keywords are not, I maintain, special enough to break the normal
version-comparison semantics.
"""

However, at PyCon, this was only popular among the people who have to deal
with packaging of python projects.  Everyone else wants the ability to sort
some things as coming before a release -- for instance, alphas, betas, release
candidates.  (Note that snapshots don't need to be pre-releases although
this PEP makes them that way).

On Fri, Nov 27, 2009 at 01:24:14PM +0100, Tarek Ziad? wrote:
> On Fri, Nov 27, 2009 at 11:39 AM, Piotr Ozarowski <ozarow at gmail.com> wrote:
> > [Tarek Ziad?, 2009-11-26]
> >> On Thu, Nov 26, 2009 at 8:55 PM, Floris Bruynooghe
> >> <floris.bruynooghe at gmail.com> wrote:
> >> [..]
> >> >> since the .dev versions are really only snapshots leading up to
> >> >> some release, i.e. 1.0.dev456 is a snapshot leading up to the
> >> >> first pre-release of the 1.0 :-)
> >> >
> >> > But in this case if I want to make a pre-release of 1.0 but after the
> >> > last rc then I can't, I can only make a post-release of the last rc.
> >> > That's almost more un-intuitive that forcing your first pre-release to
> >> > be '1.0a0.dev456' instead of just '1.0.dev456'.
> >>
> >> It seems to me that the number of development versions of rc releases
> >> is very low compared to the number of development snapshots done for
> >> 1.0, before the pre-release cycle starts.
> >>
> >> (I don't think I have ever needed a dev snapshot of a rc version)
> >>
> >> I am +1 for keeping the intuitive writing for the pre-release cycle.
> >>
> >> e.g.
> >>
> >> ?1.0.dev456
> >> ?< 1.0a1.dev456
> >> ?< 1.0a1
> >> ?< 1.0rc1.dev456
> >> ?< 1.0rc1
> >> ?< 1.0rc1.post123
> >> ?< 1.0

Note, what I "intuitivly" see from the given version numbers is more
like this::
  0.9
  < 0.9.dev456
  < 1.0rc1
  < 1.0rc1.dev456
  < 1.0rc1.post123
  < 1.0
  < 1.0.dev456
  < 1.0a1
  < 1.0a1.dev456

I've been trained by many projects to see "rc" as "release candidate" but the
definition of "a" and "b" differs from project to project.  Many projects
use them as "patchlevels" for minor post-releases instead of abbreviations
for alpha and beta.  Also, if you have a directory of revisions being served
up by apache it's going to put them in strcmp() order, so 1.0 is going to sort
after the rc's, the .dev's, and the a1's.  Much confusion to overcome here.

Is this a bikeshed?  In some ways it is because some people will
"intuitively" see prereleases and others will see postreleases. However,
versioning impacts end users so we should strive to make it as easy as
possible for people to see the real meaning of our versions.  At pycon I
suggested that everything should be treated as a postrelease except for
certain specific words like "alpha", "beta", and "rc" and that we should strive to
keep that list of specific words as small as possible to avoid confusion.

This might be a good place to say that I agree with part of what
Marc-Andre Lemburg says:
"""
The pre-release marker would then be interpreted in alphabetical
order, ie. 'alpha' < 'beta' < 'rc'.

This minor change would broaden the scope of the scheme somewhat
and make it more compatible to what's being used outside the
python-dev sphere (esp. with respect to 'c' standing for release
candidate... unless you happen to read it as gamma ;-).
"""

Like Marc-Andre I think alpha, beta, rc contain the information to remove
confusion while "a", "b", and "c" do not.  However, I do not like having
aliases (ie: "a"== "alpha").  Aliases have several problems:

1) How do you sort: 1.0a1 and 1.0alpha1 ?  Using an alias these are the same
release but since the developer of the package did this intentionally, they
obviously meant them to mean something different...  we just don't know
what that something is.

2) Eventually, someone will think that "a", "b", and "c" are for
patchlevels.  Then we'll have someone releasing code that they expect to
work like this::
  1.0-alpha1
  1.0
  1.0-a1

The version checking code will okay it since each one is correct by itself.
It's the meaning that the author attributes to them that will be wrong.


Floris Bruynooghe points these issues out although he uses "dev" and "post"
which have distinct meanings in the PEP:

"""
I also object to the alternatives for the 'dev' and 'post' markers as
they make it more confusing for me.  While someone might prefer one
word over the other their meaning does not change to decide their
ordering, that just seems like uneeded complexity (there should be
only one obvious way to do a thing?).

When I now see the versioning number of a project I need to go and
look up the pep to know if it's compliant but just using one of the
alternatives that I'm not used to.  If there's only one choice it's a
lot easier.

(This same argument goes for 'a' == 'alpha' 'b' == 'beta' and 'c' ==
'rc' but those at least are mnemonic so easier to remember)
"""

By pointing these out, I don't want to stop progress since other people seem
very attached to using "a", "b", "c" instead of fully spelled out alpha, beta
and rc but I defintely see that as a wart.  (Emphasis on "instead of", not
in addition to.)


> > why not simply use "-" and "+" where "-" is before zero-length string
> > and "+" is after any other string... and then sort the rest
> > alphabetically? f.e.
> >
> > ?1.0-a1-dev456
> > < 1.0-a1
> > < 1.0-a1+dev456
> > < 1.0-dev456
> > < 1.0-rc1-dev456
> > < 1.0-rc1
> > < 1.0-rc1+post123
> > < 1.0
> > < 1.0+post123
> >
> > don't worry about Debian, we'll simply replace "-" with "~" (we use "~"
> > and "+" right now[0]). I'm not sure about rpm, but I bet it has
> > something similar and it will be much easier for us to simply handle two
> > characters instead of recognizing that dev1 < a1 < b1 < c1 == rc1 ...
> 
> It's different from RPMs, since they use a strcmp(), segment  by segment,
> so I think they have to extract the dev/post suffixes and to put them in front
> as an epoch marker maybe ? (ccing Toshio)
> 
There's a few things to think about here.  First, there's the underlying tool.
In Debian, that's dpkg and in Fedora, it's rpm (Note: www.rpm.org, not
www.rpm5.org, the latter is a fork.)  The tool enforces things like "-" is a
separator between fields, "." as a separator inside fields, and ordering of
packages based on what's contained in the fields.  In rpm we have this to
work with::

:Name: Name of a package.  This makes "python" different from
   "python-docs" different from "gcc"
:Epoch: This is the trump card of the version sorting.  We seldom use this
    as it doesn't show up in our filenames or other end-user visible
    interfaces.  It is used for correcting mistakes in versioning or times
    when we decide that we absolutely have to revert a package rather than
    continuing to package the current release.
:Version: This is generally the upstream release.  However, upstream will
    frequently make releases that won't order correctly (for instance,
    1.0alpha1 => 1.0;  strcmp() will order these as 1.0 => 1.0alpha1)
    All distributions have to transform the version in some way to make
    ordering work correctly with the upstream's versions but they have
    different methods.  Additionally, we want the upstream version to be
    apparent from the version and release fields since end users will see
    these two fields in the filenames of the packages and we want them to
    know they're getting, for instance,  the upstream 2.6.1 release of
    python.
:Release: The release of the package within the distribution.  If upstream
    version ordering exactly fit the rpm algorithm, this would just be an
    integer that incremented everytime we built the same upstream version of
    the package.  It's the least significant of the fields on ordering.

Each distribution has its own rules for dealing with the upstream version so
that rpm will order things correctly.  In Fedora, they involve taking the
initial version string that consists of [0-9.] and leaving that in version.
Anything beyond that initial portion is put into the release after our
portion of the release.  For instance if upstream makes the following
releases:

Upstream            Version  Release
--------            -------  -------
libjpeg-2.0alpha1   2.0      0.1.alpha1
libjpeg-2.0         2.0      1
libjpeg-2.0a        2.0      2.a

Gory details:
https://fedoraproject.org/wiki/Packaging:NamingGuidelines#Package_Version
(Version and Release section)

Here's how we would map each of the examples in Fedora.  Note that at the
moment this is a manual process as we can't depend on upstreams to only
follow this.  I'll separate our version and release with a "-" as that's how
rpm will show it to our users:

  1.0.dev456            1.0-0.1.dev456
  < 1.0a1.dev456        1.0-0.2.a1.dev456
  < 1.0a1               1.0-0.3.a1
  < 1.0rc1.dev456       1.0-0.4.rc1.dev456
  < 1.0rc1              1.0-0.5.rc1
  < 1.0rc1.post123      1.0-0.6.rc1.post123
  < 1.0                 1.0-1

============

  1.0-a1-dev456         1.0-0.1.a1.dev456
  < 1.0-a1              1.0-0.2.a1
  < 1.0-a1+dev456       1.0-0.3.a1+dev456
  < 1.0-dev456          1.0-0.4.dev456
  < 1.0-rc1-dev456      1.0-0.5.rc1.dev456
  < 1.0-rc1             1.0-0.6.rc1
  < 1.0-rc1+post123     1.0-0.7.rc1+post123
  < 1.0                 1.0-1
  < 1.0+post123         1.0-2.post123

We can work with whatever is given to us... just the number of hoops and
special cases that we have to remeber what the position is when we do so.
That said, I don't see the reason for this change.

1) current post is similar to (+post and +dev)
   current dev is similar to (-post and -dev)
   Why duplicate functionality that's already in the PEP?

2) If we remove the "post" version string, then you are substituting a nice,
spelled out word ("post") with a cryptic symbol ('+').

3) The position of the -dev456 in your list satisfies neither set of people
who want to define the meaning of dev.  It's showing up between a1 and
rc1 whereas one camp wants it to sort before a1 and the other camp wants it
to sort just before 1.0.

?) Where does 1.0-post123 fall in your list?

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091127/a30fec55/attachment-0001.pgp>

From ben+python at benfinney.id.au  Sat Nov 28 00:32:58 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 28 Nov 2009 10:32:58 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
Message-ID: <87ocmnwnkl.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> On Fri, Nov 27, 2009 at 1:52 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > I'd like to register, once again, the point that [translating PEP
> > 386 version comparison semantics to existing semantics] would not
> > *be* a problem if PEP 386 described a version comparison scheme that
> > simply works without special keywords. Have each segment compared
> > alphanumerically, and it will not *need* translation to work with
> > other packaging systems.
>
> Not sure what you mean by special keywords. If it's about 'post' and
> 'dev'

Yes, and others that have been proposed to be exceptions to alphanumeric
ordering.

> the problem we have is to be able to sort those post and pre releases
> with other versions. And this is not possible with a simple
> alphanumerical comparison.

It is, if the version strings are chosen to fit within such a comparison
semantic.

> And this is not a special use case, and has to be part imho of our
> version-comparison semantics.
> otherwise, our scheme would be useless for developers that work as a
> daily basis with snapshots,
> dev versions etc.

This is a red herring, AFAICT. It's been discussed already that workflow
is orthogonal to version comparison semantics. That is, nothing about a
workflow involving snapshots or dev versions etc. implies that
exceptional version keywords need to be accommodated. Exactly the same
workflow can take place by choosing different version strings that work
with existing version-comparison semantics.

Moreover, AIUI there is no injunction that all projects must follow
exactly the semantics of PEP 386, right? So why not have a *simple*
standard (all version string components compared alphanumerically) in
PEP 386, that is known to work unmodified with existing package systems,
without Python-specific translation layers for version comparison?

-- 
 \         ?Alternative explanations are always welcome in science, if |
  `\   they are better and explain more. Alternative explanations that |
_o__) explain nothing are not welcome.? ?Victor J. Stenger, 2001-11-05 |
Ben Finney


From eric at trueblade.com  Sat Nov 28 01:12:36 2009
From: eric at trueblade.com (Eric Smith)
Date: Fri, 27 Nov 2009 19:12:36 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <87ocmnwnkl.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<20091126195549.GA29498@laurie.devork>	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>	<20091127103958.GU2966@piotro.eu>	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>	<873a40xh7k.fsf_-_@benfinney.id.au>	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
Message-ID: <4B106AF4.401@trueblade.com>

Ben Finney wrote:
> Yes, and others that have been proposed to be exceptions to alphanumeric
> ordering.
> 
>> the problem we have is to be able to sort those post and pre releases
>> with other versions. And this is not possible with a simple
>> alphanumerical comparison.
> 
> It is, if the version strings are chosen to fit within such a comparison
> semantic.
> 
>> And this is not a special use case, and has to be part imho of our
>> version-comparison semantics.
>> otherwise, our scheme would be useless for developers that work as a
>> daily basis with snapshots,
>> dev versions etc.
> 
> This is a red herring, AFAICT. It's been discussed already that workflow
> is orthogonal to version comparison semantics. That is, nothing about a
> workflow involving snapshots or dev versions etc. implies that
> exceptional version keywords need to be accommodated. Exactly the same
> workflow can take place by choosing different version strings that work
> with existing version-comparison semantics.

How would you propose Python itself (a project that doesn't have 
especially wacky release numbers) specify 3.1.1rc1 in such a scheme?



From ben+python at benfinney.id.au  Sat Nov 28 01:49:02 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 28 Nov 2009 11:49:02 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au> <4B106AF4.401@trueblade.com>
Message-ID: <877htbwk1t.fsf@benfinney.id.au>

Eric Smith <eric at trueblade.com> writes:

> Ben Finney wrote:
> >> the problem we have is to be able to sort those post and pre
> >> releases with other versions. And this is not possible with a
> >> simple alphanumerical comparison.
> >
> > It is, if the version strings are chosen to fit within such a
> > comparison semantic.
[?]

> How would you propose Python itself (a project that doesn't have
> especially wacky release numbers) specify 3.1.1rc1 in such a scheme?

Since the specification is about version *comparison* semantics, I'd
need to know example version strings that should compare ?prior? and
?subsequent? for the version string I'm to propose. Can you give
examples?

-- 
 \        ?All opinions are not equal. Some are a very great deal more |
  `\    robust, sophisticated and well supported in logic and argument |
_o__)                                     than others.? ?Douglas Adams |
Ben Finney


From pje at telecommunity.com  Sat Nov 28 02:02:52 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 27 Nov 2009 20:02:52 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <87ocmnwnkl.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
Message-ID: <20091128010302.35F143A40A8@sparrow.telecommunity.com>

At 10:32 AM 11/28/2009 +1100, Ben Finney wrote:
>This is a red herring, AFAICT. It's been discussed already that 
>workflow is orthogonal to version comparison semantics. That is, 
>nothing about a workflow involving snapshots or dev versions etc. 
>implies that exceptional version keywords need to be accommodated. 
>Exactly the same workflow can take place by choosing different 
>version strings that work with existing version-comparison 
>semantics. Moreover, AIUI there is no injunction that all projects 
>must follow exactly the semantics of PEP 386, right? So why not have 
>a *simple* standard (all version string components compared 
>alphanumerically) in PEP 386, that is known to work unmodified with 
>existing package systems, without Python-specific translation layers 
>for version comparison?

For the simple reason that it involves asking one group of people to 
change how they work -- with no immediate benefit -- so that another 
group can avoid having to change how they work.

Why should developers change, and packagers not?

It's simply not an equitable request, which is why the proposal is unrealistic.


From ben+python at benfinney.id.au  Sat Nov 28 02:26:58 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 28 Nov 2009 12:26:58 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
Message-ID: <87y6lrv3q5.fsf@benfinney.id.au>

"P.J. Eby" <pje at telecommunity.com> writes:

> At 10:32 AM 11/28/2009 +1100, Ben Finney wrote:
> > Moreover, AIUI there is no injunction that all projects must follow
> > exactly the semantics of PEP 386, right? So why not have a *simple*
> > standard (all version string components compared alphanumerically)
> > in PEP 386, that is known to work unmodified with existing package
> > systems, without Python-specific translation layers for version
> > comparison?
>
> For the simple reason that it involves asking one group of people to
> change how they work -- with no immediate benefit -- so that another
> group can avoid having to change how they work.

My understanding of PEP 386 is that it *isn't* about asking Python
developers to change how they work. Is that not right?

> Why should developers change, and packagers not?

Packagers *do* change how they work; Piotr has been explaining at length
how packagers put in efforts to meet version-comparison semantics half
way. I'm asking that the standardisation effort at least not make this
effort more difficult that it needs to be.

Moreover, what is a standard for if not for encouraging people to follow
it for the benefit of many?

> It's simply not an equitable request, which is why the proposal is
> unrealistic.

I think you're reading a proposal that I didn't write.

-- 
 \     ?Our task must be to free ourselves from our prison by widening |
  `\    our circle of compassion to embrace all humanity and the whole |
_o__)                       of nature in its beauty.? ?Albert Einstein |
Ben Finney


From eric at trueblade.com  Sat Nov 28 03:49:08 2009
From: eric at trueblade.com (Eric Smith)
Date: Fri, 27 Nov 2009 21:49:08 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <877htbwk1t.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<20091126195549.GA29498@laurie.devork>	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>	<20091127103958.GU2966@piotro.eu>	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>	<873a40xh7k.fsf_-_@benfinney.id.au>	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>	<87ocmnwnkl.fsf@benfinney.id.au>
	<4B106AF4.401@trueblade.com> <877htbwk1t.fsf@benfinney.id.au>
Message-ID: <4B108FA4.6080601@trueblade.com>

Ben Finney wrote:
> Eric Smith <eric at trueblade.com> writes:
> 
>> Ben Finney wrote:
>>>> the problem we have is to be able to sort those post and pre
>>>> releases with other versions. And this is not possible with a
>>>> simple alphanumerical comparison.
>>> It is, if the version strings are chosen to fit within such a
>>> comparison semantic.
> [?]
> 
>> How would you propose Python itself (a project that doesn't have
>> especially wacky release numbers) specify 3.1.1rc1 in such a scheme?
> 
> Since the specification is about version *comparison* semantics, I'd
> need to know example version strings that should compare ?prior? and
> ?subsequent? for the version string I'm to propose. Can you give
> examples?

I want to install the latest version of Python. My choices are 3.3.1 
(the released version) and 3.3.1rc1.

From pje at telecommunity.com  Sat Nov 28 04:00:22 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Fri, 27 Nov 2009 22:00:22 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <87y6lrv3q5.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
Message-ID: <20091128030031.D10583A40A8@sparrow.telecommunity.com>

At 12:26 PM 11/28/2009 +1100, Ben Finney wrote:
>I think you're reading a proposal that I didn't write.

Are you not the person who's proposed using simple alphanumeric 
strings for version comparison?

>My understanding of PEP 386 is that it *isn't* about asking Python 
>developers to change how they work. Is that not right?

Asking them to generate alternative versioning schemes, without the 
ability to use pre-release versions (e.g. alpha, beta, etc.) would 
most certainly be a change from the status quo.

>  Moreover, what is a standard for if not for encouraging people to 
> follow it for the benefit of many?

Usually, they get something in trade.  Supporting WSGI means you get 
to use WSGI apps, or run on a WSGI server, for example.  So, cost and 
benefit are balanced.

In contrast, a popular Python library or app will get packaged if 
somebody wants it, and not if they don't.  There isn't a lot of 
incentive for a developer to do something different, solely for the 
sake of the packager...  which is probably why things are the way they are now.

If, on the other hand, there's some positive benefit being offered 
(e.g. a new tool that makes life easier for them or perhaps their 
users), then there might be some incentive to learn a new way of 
coding versions.  But without such an incentive, who wants to learn 
some new way to encode prerelease versions?

This is why I've argued for keeping a scheme in 386 that can 
mechanically translate most existing versioning schemes found in the 
wild: it means that most people won't have to do a thing, as tool 
builders can just use suggest_version().


From lac at openend.se  Sat Nov 28 07:31:15 2009
From: lac at openend.se (Laura Creighton)
Date: Sat, 28 Nov 2009 07:31:15 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: Message from "P.J. Eby" <pje@telecommunity.com> of "Fri,
	27 Nov 2009 22:00:22 EST."
	<20091128030031.D10583A40A8@sparrow.telecommunity.com> 
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com> 
Message-ID: <200911280631.nAS6VFUi013532@theraft.openend.se>

It occurs to me that this problem would go away if we had a way to 
ask, for any given version number, 'what was your creation date' and
the sorting 'earlier' and 'later' by that date.  Can somebody explain
why we aren't doing this?

Laura


From ben+python at benfinney.id.au  Sat Nov 28 08:53:28 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 28 Nov 2009 18:53:28 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au> <4B106AF4.401@trueblade.com>
	<877htbwk1t.fsf@benfinney.id.au> <4B108FA4.6080601@trueblade.com>
Message-ID: <87fx7zultz.fsf@benfinney.id.au>

Eric Smith <eric at trueblade.com> writes:

> Ben Finney wrote:
> > Since the specification is about version *comparison* semantics, I'd
> > need to know example version strings that should compare ?prior? and
> > ?subsequent? for the version string I'm to propose. Can you give
> > examples?
>
> I want to install the latest version of Python. My choices are 3.3.1
> (the released version) and 3.3.1rc1.

And what version string should this compare prior to?

That is, you want me to propose an example version string X. I'm asking
for you to tell me corresponding examples of version strings W and Y
where:

    W < X < Y

since the comparison semantics are what we're discussing here.

-- 
 \     ?There will be a Moscow Exhibition of the Arts by 15,000 Soviet |
  `\     Republic painters and sculptors. These were executed over the |
_o__)               past two years.? ?newspaper article, Soviet Weekly |
Ben Finney


From ben+python at benfinney.id.au  Sat Nov 28 08:55:19 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 28 Nov 2009 18:55:19 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
Message-ID: <87bpinulqw.fsf@benfinney.id.au>

"P.J. Eby" <pje at telecommunity.com> writes:

> At 12:26 PM 11/28/2009 +1100, Ben Finney wrote:
> > My understanding of PEP 386 is that it *isn't* about asking Python
> > developers to change how they work. Is that not right?
>
> Asking them to generate alternative versioning schemes, without the
> ability to use pre-release versions (e.g. alpha, beta, etc.) would
> most certainly be a change from the status quo.

We seem to be talking entirely past each other; I can't see how to
relate what I wrote to your reply.

-- 
 \              ?Programs must be written for people to read, and only |
  `\        incidentally for machines to execute.? ?Abelson & Sussman, |
_o__)              _Structure and Interpretation of Computer Programs_ |
Ben Finney


From regebro at gmail.com  Sat Nov 28 09:03:34 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Sat, 28 Nov 2009 09:03:34 +0100
Subject: [Distutils] The "incompatible versions" problem
In-Reply-To: <4B0FC088.50008@simplistix.co.uk>
References: <4B0FC088.50008@simplistix.co.uk>
Message-ID: <319e029f0911280003q2cb2ba19w58f8600d17fd0ca2@mail.gmail.com>

On Fri, Nov 27, 2009 at 13:05, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> Do you guys on this lists have any ideas how packages can indicate to
> package managers like setuptools could indicate that if they are required by
> another package, that requirement must be for greater than a certain
> version?
>
> This stems from the "backwards compatibility" discussion for zope.interface
> here:
>
> https://mail.zope.org/pipermail/zope-dev/2009-November/038417.html
>
> Off the top of my head, zope.component could spell it's requirement as:
>
> def setup(
> ?...
> ?incompatible_with='<4.0'
> ?)
>
> Whereby, setuptools would raise an exception if the following packages were
> installed:
>
> def setup(
> ?...
> ?install_requires=['zope.component']
> ?)

I think this is the problem. I understand what you mean, but it's not
obvious to me at all that this should raise an error in this
situation. But yet, for your suggestion to work, it has to. In
practice, that means that you, to use zope.component >= 4.0, must give
a version number. But even when an API changes, it may be possible to
support both versions. And that would then be impossible to do in this
case.

I suspect it's an unsolvable problem. It's too open ended.
-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From ziade.tarek at gmail.com  Sat Nov 28 09:30:35 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 09:30:35 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <87ocmnwnkl.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
Message-ID: <94bdd2610911280030n373e017dxcd86e28374353694@mail.gmail.com>

On Sat, Nov 28, 2009 at 12:32 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
[..]
>> the problem we have is to be able to sort those post and pre releases
>> with other versions. And this is not possible with a simple
>> alphanumerical comparison.
>
> It is, if the version strings are chosen to fit within such a comparison
> semantic.

So how do you sort development version and post-release versions
in a "simple" alphanumeric order ?

>
>> And this is not a special use case, and has to be part imho of our
>> version-comparison semantics.
>> otherwise, our scheme would be useless for developers that work as a
>> daily basis with snapshots,
>> dev versions etc.
>
> This is a red herring, AFAICT. It's been discussed already that workflow
> is orthogonal to version comparison semantics. That is, nothing about a
> workflow involving snapshots or dev versions etc. implies that
> exceptional version keywords need to be accommodated. Exactly the same
> workflow can take place by choosing different version strings that work
> with existing version-comparison semantics.

I don't understand what you mean by workflow. When you need to compare versions,
wether they are development versions, or pre and post-release
versions, or final version,
you need to sort them.

>
> Moreover, AIUI there is no injunction that all projects must follow
> exactly the semantics of PEP 386, right? So why not have a *simple*
> standard (all version string components compared alphanumerically) in
> PEP 386, that is known to work unmodified with existing package systems,
> without Python-specific translation layers for version comparison?

A project process is not a private thing. If you use third party
libraries, you hit the problem
of having to install development versions or pre-release versions.

Tarek.

From ziade.tarek at gmail.com  Sat Nov 28 09:41:06 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 09:41:06 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <20091128010302.35F143A40A8@sparrow.telecommunity.com>
References: <4B0D6CE9.50306@egenix.com> <20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
Message-ID: <94bdd2610911280041u5dee8db8s98fcc9f1f8883c11@mail.gmail.com>

On Sat, Nov 28, 2009 at 2:02 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 10:32 AM 11/28/2009 +1100, Ben Finney wrote:
>>
>> This is a red herring, AFAICT. It's been discussed already that workflow
>> is orthogonal to version comparison semantics. That is, nothing about a
>> workflow involving snapshots or dev versions etc. implies that exceptional
>> version keywords need to be accommodated. Exactly the same workflow can take
>> place by choosing different version strings that work with existing
>> version-comparison semantics. Moreover, AIUI there is no injunction that all
>> projects must follow exactly the semantics of PEP 386, right? So why not
>> have a *simple* standard (all version string components compared
>> alphanumerically) in PEP 386, that is known to work unmodified with existing
>> package systems, without Python-specific translation layers for version
>> comparison?
>
> For the simple reason that it involves asking one group of people to change
> how they work -- with no immediate benefit -- so that another group can
> avoid having to change how they work.
>
> Why should developers change, and packagers not?
>
> It's simply not an equitable request, which is why the proposal is
> unrealistic.

That's completely wrong, the proposal is a benefit for all of us,
because it standardizes something that is already being done.

PEP 386 propose a scheme to be adopted by developers or tools, but if some
people want to stick with their own internal version scheme for
development versions or post/pre release versions, they can do it
without any problem. And they don't have to follow
any PEP 386 convention for their internal work.

And when they release their projects, if they stick with
MAJOR.MINOR.MICRO, it will work.

But as soon as they publish development versions, or post or pre
releases, they have two choice:

- they use a standard that is described in PEP 386 - so their package
is installable by installers that follow it

- they provide documentation about their own internal standard

Today, if they release their package with their own custom version
scheme, setuptools will
just install those package, and screw the ordering if it doesn't
follow its heuristic version sorter.

So PEP 386 is just about describing a standard for something that is
already being done:
installing versioned distributions in someone's computer.

Tarek

From ziade.tarek at gmail.com  Sat Nov 28 09:49:04 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 09:49:04 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <87y6lrv3q5.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
Message-ID: <94bdd2610911280049g2b9ac5f6i1f368e331b8fd6c3@mail.gmail.com>

On Sat, Nov 28, 2009 at 2:26 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
[..]
>
>> Why should developers change, and packagers not?
>
> Packagers *do* change how they work; Piotr has been explaining at length
> how packagers put in efforts to meet version-comparison semantics half
> way. I'm asking that the standardisation effort at least not make this
> effort more difficult that it needs to be.

Piotr, and all os packagers will always have to transform our versions
into theirs.

Once we have chosen it, it will not make their life more difficult.
It's just one regular
expression doing the work. Its size may vary that's all.

Moreover, they will be able to do a better work because they will stop guessing
the versions of Python packages, they will just stick with a
*documented* standard.

>
> Moreover, what is a standard for if not for encouraging people to follow
> it for the benefit of many?
>
>> It's simply not an equitable request, which is why the proposal is
>> unrealistic.
>
> I think you're reading a proposal that I didn't write.

It's a realistic standard. It's an equitable request. You can do
whatever you want to
do with your versions internally. But as soon as you publish them, they HAVE to
be ordered by other tools that deal with your package, like installers.

So you have two choices:
- an implicit, heuristic ordering (that's what is happening today)
- a explicit, documented ordering. that's the goal of PEP 386.

I don't see why this is not a benefit for all of us.

Tarek

From ziade.tarek at gmail.com  Sat Nov 28 09:52:57 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 09:52:57 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <20091128030031.D10583A40A8@sparrow.telecommunity.com>
References: <4B0D6CE9.50306@egenix.com> <20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
Message-ID: <94bdd2610911280052n79aa657dq86736899360181d3@mail.gmail.com>

On Sat, Nov 28, 2009 at 4:00 AM, P.J. Eby <pje at telecommunity.com> wrote:
> At 12:26 PM 11/28/2009 +1100, Ben Finney wrote:
>>
>> I think you're reading a proposal that I didn't write.
>
> Are you not the person who's proposed using simple alphanumeric strings for
> version comparison?
>
>> My understanding of PEP 386 is that it *isn't* about asking Python
>> developers to change how they work. Is that not right?
>
> Asking them to generate alternative versioning schemes, without the ability
> to use pre-release versions (e.g. alpha, beta, etc.) would most certainly be
> a change from the status quo.
>
>> ?Moreover, what is a standard for if not for encouraging people to follow
>> it for the benefit of many?
>
> Usually, they get something in trade. ?Supporting WSGI means you get to use
> WSGI apps, or run on a WSGI server, for example. ?So, cost and benefit are
> balanced.
>
> In contrast, a popular Python library or app will get packaged if somebody
> wants it, and not if they don't. ?There isn't a lot of incentive for a
> developer to do something different, solely for the sake of the packager...
> ?which is probably why things are the way they are now.
>
> If, on the other hand, there's some positive benefit being offered (e.g. a
> new tool that makes life easier for them or perhaps their users), then there
> might be some incentive to learn a new way of coding versions. ?But without
> such an incentive, who wants to learn some new way to encode prerelease
> versions?
>
> This is why I've argued for keeping a scheme in 386 that can mechanically
> translate most existing versioning schemes found in the wild: it means that
> most people won't have to do a thing, as tool builders can just use
> suggest_version().

Yes, this function can be used internally,  and that was a good idea
to have that function.

But as soon as it is used inside a public package manager, like pip or
setuptools or distribute, I expect this function to raise some kind of
warning so the developers knows its version
doesn't meet the public, documented standard.

From ziade.tarek at gmail.com  Sat Nov 28 10:27:14 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 10:27:14 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <200911280631.nAS6VFUi013532@theraft.openend.se>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
Message-ID: <94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>

On Sat, Nov 28, 2009 at 7:31 AM, Laura Creighton <lac at openend.se> wrote:
> It occurs to me that this problem would go away if we had a way to
> ask, for any given version number, 'what was your creation date' and
> the sorting 'earlier' and 'later' by that date. ?Can somebody explain
> why we aren't doing this?

You mean like a timestamp before or after the version ?

I might be wrong but I think that would be similar to what RPM calls
an Epoch. A number that can be used to compare two packages when their
versions number don't follow the standard scheme anymore. But that's
just a fallback.
But for the sake of simplicity and standardization, this extra number
is avoided.

Meaning that it would be better to define and use a standard for the
released packages, than introducing a timestamp and say: do whatever
you want with your version numbers.

At some point, we all agree that MAJOR.MINOR.MICRO is an accepted
standard and we are arguing about pre/post/dev releases.

My point is that we are able to define a public scheme for those too,
so all package installers share the same conventions.

This doesn't imply that developers have to use it internally. It just
implies that, if the developer releases his package at some point to
PyPI, he will know *how* his version number will be sorted by package
managers, because it's documented in a PEP. e.g. a shared convention.


Regards
Tarek

From lac at openend.se  Sat Nov 28 10:32:33 2009
From: lac at openend.se (Laura Creighton)
Date: Sat, 28 Nov 2009 10:32:33 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: Message from Toshio Kuratomi <a.badger@gmail.com> of "Fri,
	27 Nov 2009 11:55:08 PST." <20091127195508.GE9447@clingman.lan> 
References: <4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<20091127195508.GE9447@clingman.lan> 
Message-ID: <200911280932.nAS9WY6u024685@theraft.openend.se>


Let us say a package forks.  Or there are two packages that provide
much similar functionality.  I want to say that my package depends
on having one or the other of the packages.  How would I do that?
A concrete example would be:

requires nose 0.11.1 or greater OR py.test 1.1.1 or greater
(because what I have written is a plugin that works with either,
 but requires a py.test or a nose that is recent enough to have
 the plugin architecture)


Laura

From lac at openend.se  Sat Nov 28 10:50:36 2009
From: lac at openend.se (Laura Creighton)
Date: Sat, 28 Nov 2009 10:50:36 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: Message from =?ISO-8859-1?Q?Tarek_Ziad=E9?=
	<ziade.tarek@gmail.com> of "Sat, 28 Nov 2009 10:27:14 +0100."
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com> 
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com> 
Message-ID: <200911280950.nAS9oa2F025917@theraft.openend.se>

In a message of Sat, 28 Nov 2009 10:27:14 +0100, Tarek Ziad? writes:
>On Sat, Nov 28, 2009 at 7:31 AM, Laura Creighton <lac at openend.se> wrote:
>> It occurs to me that this problem would go away if we had a way to
>> ask, for any given version number, 'what was your creation date' and
>> the sorting 'earlier' and 'later' by that date. ?Can somebody explain
>> why we aren't doing this?
>
>You mean like a timestamp before or after the version ?
>
>I might be wrong but I think that would be similar to what RPM calls
>an Epoch. A number that can be used to compare two packages when their
>versions number don't follow the standard scheme anymore. But that's
>just a fallback.
>But for the sake of simplicity and standardization, this extra number
>is avoided.
>
>Meaning that it would be better to define and use a standard for the
>released packages, than introducing a timestamp and say: do whatever
>you want with your version numbers.

But I think that it is the other way around ... what we want is a
timestamp.  The algorithm is for guessing which version is ealier
in the absence of a timestamp.

>At some point, we all agree that MAJOR.MINOR.MICRO is an accepted
>standard and we are arguing about pre/post/dev releases.

We have no way to enforce this on the world.  The arguments here may
have gone away, but out in the wild, I think we need to accept that
people are going to name their packages whatever they like.  We
can suggest in the strongest possible terms that they use this
numbering convention, but it will still just be a suggestion.

>My point is that we are able to define a public scheme for those too,
>so all package installers share the same conventions.
>
>This doesn't imply that developers have to use it internally. It just
>implies that, if the developer releases his package at some point to
>PyPI, he will know *how* his version number will be sorted by package
>managers, because it's documented in a PEP. e.g. a shared convention.

It's not that I am 'anti-convention', it is just that I think that
if we include a timestamp, and use that _first_ when trying to find
out if a package is earlier than another package, then we will be
able to interoperate with those people who out of ignorance or some
business reason cannot respect the convention.

Laura

>Regards
>Tarek

From cournape at gmail.com  Sat Nov 28 11:11:07 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 28 Nov 2009 19:11:07 +0900
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <200911280950.nAS9oa2F025917@theraft.openend.se>
References: <4B0D6CE9.50306@egenix.com> <87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<ziade.tarek@gmail.com>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
Message-ID: <5b8d13220911280211r5177643byfe343846cf732e3a@mail.gmail.com>

2009/11/28 Laura Creighton <lac at openend.se>:
> In a message of Sat, 28 Nov 2009 10:27:14 +0100, Tarek Ziad? writes:
>>On Sat, Nov 28, 2009 at 7:31 AM, Laura Creighton <lac at openend.se> wrote:
>>> It occurs to me that this problem would go away if we had a way to
>>> ask, for any given version number, 'what was your creation date' and
>>> the sorting 'earlier' and 'later' by that date. ?Can somebody explain
>>> why we aren't doing this?
>>
>>You mean like a timestamp before or after the version ?
>>
>>I might be wrong but I think that would be similar to what RPM calls
>>an Epoch. A number that can be used to compare two packages when their
>>versions number don't follow the standard scheme anymore. But that's
>>just a fallback.
>>But for the sake of simplicity and standardization, this extra number
>>is avoided.
>>
>>Meaning that it would be better to define and use a standard for the
>>released packages, than introducing a timestamp and say: do whatever
>>you want with your version numbers.
>
> But I think that it is the other way around ... what we want is a
> timestamp. ?The algorithm is for guessing which version is ealier
> in the absence of a timestamp.
>
>>At some point, we all agree that MAJOR.MINOR.MICRO is an accepted
>>standard and we are arguing about pre/post/dev releases.
>
> We have no way to enforce this on the world.

Actually, there is: refuse any packages on pypi which does not follow
the standard. In exchange of using the standard (whatever it ends up
being), you can use pypi.

David

From ozarow at gmail.com  Sat Nov 28 11:44:04 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Sat, 28 Nov 2009 11:44:04 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <200911280950.nAS9oa2F025917@theraft.openend.se>
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
Message-ID: <20091128104404.GQ6281@piotro.eu>

[Laura Creighton, 2009-11-28]
> what we want is a
> timestamp.  The algorithm is for guessing which version is ealier
> in the absence of a timestamp.

do you want to use timestamps only to sort versions?

Consider this example:

 1.0 timestamp=20090323
 2.0 timestamp=20090618
 1.1 timestamp=20091128 (fix in 1.x branch)

how would you sort it?
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091128/af186d66/attachment.pgp>

From floris.bruynooghe at gmail.com  Sat Nov 28 12:01:45 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 28 Nov 2009 11:01:45 +0000
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <200911280950.nAS9oa2F025917@theraft.openend.se>
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
Message-ID: <20091128110145.GA25937@laurie.devork>

On Sat, Nov 28, 2009 at 10:50:36AM +0100, Laura Creighton wrote:
> But I think that it is the other way around ... what we want is a
> timestamp.  The algorithm is for guessing which version is ealier
> in the absence of a timestamp.

Does this work when you have a two maintenance branches releasing in
parallel?  Consider this example:

2009-01-01: 1.0
2009-02-01: 1.1
2009-02-20: 1.0.1    (1.0 maint branch)
2009-02-21: 1.1.1    (1.1 maint branch)


Regards
Floris

From eric at trueblade.com  Sat Nov 28 12:05:53 2009
From: eric at trueblade.com (Eric Smith)
Date: Sat, 28 Nov 2009 06:05:53 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <87fx7zultz.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<20091126195549.GA29498@laurie.devork>	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>	<20091127103958.GU2966@piotro.eu>	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>	<873a40xh7k.fsf_-_@benfinney.id.au>	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>	<87ocmnwnkl.fsf@benfinney.id.au>
	<4B106AF4.401@trueblade.com>	<877htbwk1t.fsf@benfinney.id.au>
	<4B108FA4.6080601@trueblade.com> <87fx7zultz.fsf@benfinney.id.au>
Message-ID: <4B110411.1040904@trueblade.com>

Ben Finney wrote:
> Eric Smith <eric at trueblade.com> writes:
> 
>> Ben Finney wrote:
>>> Since the specification is about version *comparison* semantics, I'd
>>> need to know example version strings that should compare ?prior? and
>>> ?subsequent? for the version string I'm to propose. Can you give
>>> examples?
>> I want to install the latest version of Python. My choices are 3.3.1
>> (the released version) and 3.3.1rc1.
> 
> And what version string should this compare prior to?
> 
> That is, you want me to propose an example version string X. I'm asking
> for you to tell me corresponding examples of version strings W and Y
> where:
> 
>     W < X < Y
> 
> since the comparison semantics are what we're discussing here.
> 

W = 3.3.0
Y = 3.3.1

If you're going to suggest 3.3.0.99 or some such, I think that won't 
work. It's important that code sees that as 3.3.1.<something>. I've seen 
code that fails when the pre-release versions have a different 
MAJOR.MINOR.MICRO than the final version. I really think we need a 
"subtractive" versioning scheme.

Eric.


From floris.bruynooghe at gmail.com  Sat Nov 28 12:28:38 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 28 Nov 2009 11:28:38 +0000
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <20091128030031.D10583A40A8@sparrow.telecommunity.com>
References: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
Message-ID: <20091128112838.GB25937@laurie.devork>

On Fri, Nov 27, 2009 at 10:00:22PM -0500, P.J. Eby wrote:
> This is why I've argued for keeping a scheme in 386 that can
> mechanically translate most existing versioning schemes found in the
> wild: it means that most people won't have to do a thing, as tool
> builders can just use suggest_version().


This is an important point.  I tought the aim of PEP 386 was to create
a version scheme that can represent every version number developers
want to express, not to create one that allows everyones favourite
syntax.

And it seems most of the discussion has now ventured back into the
syntax (flamewars/bikeshedding?) instead of whether everyone can
express the versions they need to.


Regards
Floris

From ben+python at benfinney.id.au  Sat Nov 28 12:53:26 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 28 Nov 2009 22:53:26 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au> <4B106AF4.401@trueblade.com>
	<877htbwk1t.fsf@benfinney.id.au> <4B108FA4.6080601@trueblade.com>
	<87fx7zultz.fsf@benfinney.id.au> <4B110411.1040904@trueblade.com>
Message-ID: <87pr72uaq1.fsf@benfinney.id.au>

Eric Smith <eric at trueblade.com> writes:

> Ben Finney wrote:
> > That is, you want me to propose an example version string X. I'm
> > asking for you to tell me corresponding examples of version strings
> > W and Y where:
> >
> >     W < X < Y
> >
> > since the comparison semantics are what we're discussing here.
> >
>
> W = 3.3.0
> Y = 3.3.1
>
> If you're going to suggest 3.3.0.99 or some such

Yes. Or any version string that would obviously fall between the two.

> I think that won't work. It's important that code sees that as
> 3.3.1.<something>. I've seen code that fails when the pre-release
> versions have a different MAJOR.MINOR.MICRO than the final version.

It's exactly that kind of code that I don't feel needs to be
accommodated. It's clearly causing ambiguities and contortions, and now
we talk about needing Python-specific version comparison algorithms in
order to automate the comparison.

Why is this worth the trouble, when the standard could simply describe
obvious-to-everyone version comparison semantics that are far easier for
everyone to get right?

-- 
 \        ?Visitors are expected to complain at the office between the |
  `\                     hours of 9 and 11 a.m. daily.? ?hotel, Athens |
_o__)                                                                  |
Ben Finney


From lac at openend.se  Sat Nov 28 13:22:50 2009
From: lac at openend.se (Laura Creighton)
Date: Sat, 28 Nov 2009 13:22:50 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: Message from Floris Bruynooghe <floris.bruynooghe@gmail.com> 
	of "Sat,
	28 Nov 2009 11:01:45 GMT." <20091128110145.GA25937@laurie.devork> 
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
	<20091128110145.GA25937@laurie.devork> 
Message-ID: <200911281222.nASCMocd003469@theraft.openend.se>

In a message of Sat, 28 Nov 2009 11:01:45 GMT, Floris Bruynooghe writes:
>On Sat, Nov 28, 2009 at 10:50:36AM +0100, Laura Creighton wrote:
>> But I think that it is the other way around ... what we want is a
>> timestamp.  The algorithm is for guessing which version is ealier
>> in the absence of a timestamp.
>
>Does this work when you have a two maintenance branches releasing in
>parallel?  Consider this example:
>
>2009-01-01: 1.0
>2009-02-01: 1.1
>2009-02-20: 1.0.1    (1.0 maint branch)
>2009-02-21: 1.1.1    (1.1 maint branch)
>
>
>Regards
>Floris

Not if the only thing you sort on is the date -- which by the way should
have hours and minutes as well -- because, as you say that is not enough
to tell the 1.1 branch from the 1.0 branch.  But once you have a name for
a series of releases, then a timestamp can sort the series.

Let us say that you now find a horrible bug and make:

2009-02-22-22:01: 1.0.2
2009-02-22-22:04: 1.1.2

Now I want to say 'requires this bugfix'.  Right now I think that if
I say requires 1.0.2 or later, then people with 1.1 will expect that
they are ok, when they are not.  Or am I misunderstanding?

Laura



From cournape at gmail.com  Sat Nov 28 13:34:00 2009
From: cournape at gmail.com (David Cournapeau)
Date: Sat, 28 Nov 2009 21:34:00 +0900
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <200911281222.nASCMocd003469@theraft.openend.se>
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
	<floris.bruynooghe@gmail.com> <20091128110145.GA25937@laurie.devork>
	<200911281222.nASCMocd003469@theraft.openend.se>
Message-ID: <5b8d13220911280434v1a5b301t2da14189867557e6@mail.gmail.com>

On Sat, Nov 28, 2009 at 9:22 PM, Laura Creighton <lac at openend.se> wrote:
> In a message of Sat, 28 Nov 2009 11:01:45 GMT, Floris Bruynooghe writes:
>>On Sat, Nov 28, 2009 at 10:50:36AM +0100, Laura Creighton wrote:
>>> But I think that it is the other way around ... what we want is a
>>> timestamp. ?The algorithm is for guessing which version is ealier
>>> in the absence of a timestamp.
>>
>>Does this work when you have a two maintenance branches releasing in
>>parallel? ?Consider this example:
>>
>>2009-01-01: 1.0
>>2009-02-01: 1.1
>>2009-02-20: 1.0.1 ? ?(1.0 maint branch)
>>2009-02-21: 1.1.1 ? ?(1.1 maint branch)
>>
>>
>>Regards
>>Floris
>
> Not if the only thing you sort on is the date -- which by the way should
> have hours and minutes as well -- because, as you say that is not enough
> to tell the 1.1 branch from the 1.0 branch. ?But once you have a name for
> a series of releases, then a timestamp can sort the series.
>
> Let us say that you now find a horrible bug and make:
>
> 2009-02-22-22:01: 1.0.2
> 2009-02-22-22:04: 1.1.2
>
> Now I want to say 'requires this bugfix'. ?Right now I think that if
> I say requires 1.0.2 or later, then people with 1.1 will expect that
> they are ok, when they are not. ?Or am I misunderstanding?

This is a slippery road. You will always find cases where any version
scheme will fail if you start  caring about those issues through
version schemes only. In your example, that would be if different
micro releases are not synchronized, which happens quite often in my
experience.

cheers,

David

From lac at openend.se  Sat Nov 28 13:52:39 2009
From: lac at openend.se (Laura Creighton)
Date: Sat, 28 Nov 2009 13:52:39 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: Message from David Cournapeau <cournape@gmail.com> of "Sat,
	28 Nov 2009 21:34:00 +0900."
	<5b8d13220911280434v1a5b301t2da14189867557e6@mail.gmail.com> 
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
	<floris.bruynooghe@gmail.com>
	<20091128110145.GA25937@laurie.devork>
	<200911281222.nASCMocd003469@theraft.openend.se>
	<5b8d13220911280434v1a5b301t2da14189867557e6@mail.gmail.com> 
Message-ID: <200911281252.nASCqdtJ004936@theraft.openend.se>

In a message of Sat, 28 Nov 2009 21:34:00 +0900, David Cournapeau writes:
>On Sat, Nov 28, 2009 at 9:22 PM, Laura Creighton <lac at openend.se> wrote:
>> In a message of Sat, 28 Nov 2009 11:01:45 GMT, Floris Bruynooghe writes
>:
>>>On Sat, Nov 28, 2009 at 10:50:36AM +0100, Laura Creighton wrote:
>>>> But I think that it is the other way around ... what we want is a
>>>> timestamp. ??The algorithm is for guessing which version is ealier
>>>> in the absence of a timestamp.
>>>
>>>Does this work when you have a two maintenance branches releasing in
>>>parallel? ??Consider this example:
>>>
>>>2009-01-01: 1.0
>>>2009-02-01: 1.1
>>>2009-02-20: 1.0.1 ?? ??(1.0 maint branch)
>>>2009-02-21: 1.1.1 ?? ??(1.1 maint branch)
>>>
>>>
>>>Regards
>>>Floris
>>
>> Not if the only thing you sort on is the date -- which by the way shoul
>d
>> have hours and minutes as well -- because, as you say that is not enoug
>h
>> to tell the 1.1 branch from the 1.0 branch. ??But once you have a name fo
>r
>> a series of releases, then a timestamp can sort the series.
>>
>> Let us say that you now find a horrible bug and make:
>>
>> 2009-02-22-22:01: 1.0.2
>> 2009-02-22-22:04: 1.1.2
>>
>> Now I want to say 'requires this bugfix'. Right now I think that if
>> I say requires 1.0.2 or later, then people with 1.1 will expect that
>> they are ok, when they are not. ??Or am I misunderstanding?
>
>This is a slippery road. You will always find cases where any version
>scheme will fail if you start  caring about those issues through
>version schemes only. In your example, that would be if different
>micro releases are not synchronized, which happens quite often in my
>experience.
>
>cheers,
>
>David

Yes, this is why I am coming to the conclusion that we are asking
too much of a versioning scheme.  

Thus I think we need, instead, a way to specify two things.

First of all  Requires-Dist:  needs to be able to specify multiple
branches.  And second of all we need to get a unique name for a
branch.  But once we have that, we can work on timestamps to find
'later' -- and we won't have to worry if people need to make
3.0.gamma or 3.0.c4 in the future because their beta went really
badly, or something.

And, all the while, there will be the strict naming convention for
those of us who like such things, and I am one of them.  I just want
to be able to interoperate with people who don't.

Laura

From ssteinerx at gmail.com  Sat Nov 28 14:57:10 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Sat, 28 Nov 2009 08:57:10 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <200911280950.nAS9oa2F025917@theraft.openend.se>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
Message-ID: <B36B0155-5A68-4E54-AC59-0F0E7D70E2E3@gmail.com>


On Nov 28, 2009, at 4:50 AM, Laura Creighton wrote:
>> At some point, we all agree that MAJOR.MINOR.MICRO is an accepted
>> standard and we are arguing about pre/post/dev releases.
> 
> We have no way to enforce this on the world.  The arguments here may
> have gone away, but out in the wild, I think we need to accept that
> people are going to name their packages whatever they like.  We
> can suggest in the strongest possible terms that they use this
> numbering convention, but it will still just be a suggestion.

	I think the PEP should lay out a standard scheme, the Python standard library (distutils) should have the tools to support it, and either conform to it or not at your product and your user's peril.  

Other than providing tools to suggest a PEP compliant version for you that is as close to your current version as possible, I don't think we should support ANY other versioning scheme but the rational one laid out in the PEP, period.  Once the bikeshed is built and painted, throw away all the other exotic materials and paint colors.

How about if we make it a suggestion like this:

	If you want your tools to be installable in a reliable way, using standard Python installation tools,  so that your users always know that they are getting the exact version that they want, and so they can install things from PyPi, the Python standard code repository, we suggest that you follow the PEP 386 versioning scheme.

	The PEP 386 scheme will be supported by a broad range of tools including Python's distutils, the Distribute installation system, and any other tools using the Python standard library routines for versioning.

	Otherwise, we suggest that you devote a large amount of time preparing an extremely detailed  explanation about why you chose not to follow a community standard and also either provide your own comprehensive installation tools, or explain exactly how to use standard Python installation tools with your non-standard versioning scheme.

	Also, please write a FAQ about exactly how to determine exactly which version of your product is installed and how to determine what the "next" and "previous" versions are since the users can't determine that using the standard Python tools or reading the Python documentation.

	Ok?  It's just a suggestion.

	Feel free to make it as difficult for yourself and your users by inventing whatever scheme seems best to you and best of luck with that.

S


From ziade.tarek at gmail.com  Sat Nov 28 16:27:04 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 16:27:04 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091127195508.GE9447@clingman.lan>
References: <4B0D8DE3.1070706@egenix.com> <4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<20091127195508.GE9447@clingman.lan>
Message-ID: <94bdd2610911280727m3d501f47p5422ada50c0e8a9b@mail.gmail.com>

On Fri, Nov 27, 2009 at 8:55 PM, Toshio Kuratomi <a.badger at gmail.com> wrote:
[lots of explanation]

Thanks for these explanations Toshio. I am starting to think that
whatever we use on Python side will be fine for you guys, (and for
Ubuntu/Fedora guys), as long as it is
described in the PEP.

So I think that using '.' as we stated before "works" for everyone,

Now about the usage of "a1" as alpha for us, and a post release patch for you,
this is something that won't be changed because it's deeply used in
the Python community, (by Python itself, and most framworks) so we
need to keep a status quo.


>
> ?) Where does 1.0-post123 fall in your list?

post would always be prefixed by "+" in Piotr release IIRC

Regards
Tarek

From ziade.tarek at gmail.com  Sat Nov 28 16:36:59 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 16:36:59 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <200911281252.nASCqdtJ004936@theraft.openend.se>
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
	<floris.bruynooghe@gmail.com> <20091128110145.GA25937@laurie.devork>
	<200911281222.nASCMocd003469@theraft.openend.se> <cournape@gmail.com>
	<5b8d13220911280434v1a5b301t2da14189867557e6@mail.gmail.com>
	<200911281252.nASCqdtJ004936@theraft.openend.se>
Message-ID: <94bdd2610911280736g4b17175bi55b663653b18f4c5@mail.gmail.com>

On Sat, Nov 28, 2009 at 1:52 PM, Laura Creighton <lac at openend.se> wrote:
[...]
>
> Yes, this is why I am coming to the conclusion that we are asking
> too much of a versioning scheme.
>
> Thus I think we need, instead, a way to specify two things.
>
> First of all ?Requires-Dist: ?needs to be able to specify multiple
> branches. ?And second of all we need to get a unique name for a
> branch. ?But once we have that, we can work on timestamps to find
> 'later' -- and we won't have to worry if people need to make
> 3.0.gamma or 3.0.c4 in the future because their beta went really
> badly, or something.

Do you mean that people will have to provide a unique name for each branch
of their project ?

I don't think that moving this complexity to the Requires-Dist field
is going to help,
but maybe, you could provide some examples of defining some versions
dependencies, for several branches, with development versions,
pre-release versions etc ?

Regards,
Tarek

From ziade.tarek at gmail.com  Sat Nov 28 17:07:01 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 28 Nov 2009 17:07:01 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <20091128112838.GB25937@laurie.devork>
References: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<20091128112838.GB25937@laurie.devork>
Message-ID: <94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>

On Sat, Nov 28, 2009 at 12:28 PM, Floris Bruynooghe
<floris.bruynooghe at gmail.com> wrote:
> On Fri, Nov 27, 2009 at 10:00:22PM -0500, P.J. Eby wrote:
>> This is why I've argued for keeping a scheme in 386 that can
>> mechanically translate most existing versioning schemes found in the
>> wild: it means that most people won't have to do a thing, as tool
>> builders can just use suggest_version().
>
>
> This is an important point. ?I tought the aim of PEP 386 was to create
> a version scheme that can represent every version number developers
> want to express, not to create one that allows everyones favourite
> syntax.
>

Absolutely,

"suggest_rational_version" is just here to allow package managers
softwares to handle existing versions, and eventually raise a warning
when they get those.


> And it seems most of the discussion has now ventured back into the
> syntax (flamewars/bikeshedding?) instead of whether everyone can
> express the versions they need to.

Agreed. We jumped again in the black hole :)  I don't see much
progress in the thread.

Fact 0: This PEP is important for interoperability

Fact 1:
We do need to express development versions, post/pre release versions
and we do want to have a standard to be shared among public python
package installers.

Fact 2:
there are many ways to express those markers, but the PEP will
document *one* way and promote its usage, so public package installers
are all *compatible* (distutils, pip, etc..)

Fact3:
suggest_rational_ version will offer a tool for installers to deal
with more version schemes, and it is suggested that it warns the
developer that the scheme is not
PEP 386 compliant.

So If the current proposal works for all cases (e.g. people can
translate their schemes
into PEP 386 one), I am proposing to:

1- reject the "+", " ", "-" proposal, and stick with "." so we have
only one way to express the segments. (ala Python itself)

2 - keep the aliases (alpha/beta/rc) because they are not controversial

3 - stick with "post" "dev" for the post and pre-release tags because
*we need them to sort development versions and post versions* in
installers, and because *they don't hurt* for people that are not
publishing such versions. They will be able to use their own dev
markers internally if they want.

Next, once the PEP is edited, I am proposing to move this discussion
in python-dev,
for another round, and eventually have Guido accept it or reject it
and move forward with PEP 345.

Because as far as I am concerned, even if we change the syntax in PEP
386 a million times, some people will not like it at the end.

regards
Tarek

From ozarow at gmail.com  Sat Nov 28 17:17:34 2009
From: ozarow at gmail.com (Piotr Ozarowski)
Date: Sat, 28 Nov 2009 17:17:34 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911280727m3d501f47p5422ada50c0e8a9b@mail.gmail.com>
References: <4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<20091127195508.GE9447@clingman.lan>
	<94bdd2610911280727m3d501f47p5422ada50c0e8a9b@mail.gmail.com>
Message-ID: <20091128161734.GS6281@piotro.eu>

[Tarek Ziad?, 2009-11-28]
>                                       I am starting to think that
> whatever we use on Python side will be fine for you guys, (and for
> Ubuntu/Fedora guys), as long as it is
> described in the PEP.

yup
-- 
Piotr O?arowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091128/791150ac/attachment-0001.pgp>

From a.badger at gmail.com  Sat Nov 28 18:01:26 2009
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 28 Nov 2009 09:01:26 -0800
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911280727m3d501f47p5422ada50c0e8a9b@mail.gmail.com>
References: <4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<20091127195508.GE9447@clingman.lan>
	<94bdd2610911280727m3d501f47p5422ada50c0e8a9b@mail.gmail.com>
Message-ID: <20091128170126.GD22261@clingman.lan>

On Sat, Nov 28, 2009 at 04:27:04PM +0100, Tarek Ziad? wrote:
> On Fri, Nov 27, 2009 at 8:55 PM, Toshio Kuratomi <a.badger at gmail.com> wrote:
> [lots of explanation]
> 
> Thanks for these explanations Toshio. I am starting to think that
> whatever we use on Python side will be fine for you guys, (and for
> Ubuntu/Fedora guys), as long as it is
> described in the PEP.
> 
Pretty much.

> So I think that using '.' as we stated before "works" for everyone,
> 
> Now about the usage of "a1" as alpha for us, and a post release patch for you,
> this is something that won't be changed because it's deeply used in
> the Python community, (by Python itself, and most framworks) so we
> need to keep a status quo.
>
>
<nod> -- The only caveat to that is that aliases are a bad idea.

So please don't add "alpha" as an alias for "a".  If you must, add "alpha"
and "a" separately so there is a predictable sorting between "alpha1" and
"a1"... although this is still a way to make confusion rather than eliminate
it.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091128/1bd6bdeb/attachment.pgp>

From a.badger at gmail.com  Sat Nov 28 18:07:25 2009
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Sat, 28 Nov 2009 09:07:25 -0800
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>
References: <94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<20091128112838.GB25937@laurie.devork>
	<94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>
Message-ID: <20091128170725.GE22261@clingman.lan>

On Sat, Nov 28, 2009 at 05:07:01PM +0100, Tarek Ziad? wrote:
> 
> So If the current proposal works for all cases (e.g. people can
> translate their schemes
> into PEP 386 one), I am proposing to:
> 
> 1- reject the "+", " ", "-" proposal, and stick with "." so we have
> only one way to express the segments. (ala Python itself)
> 
+1

> 2 - keep the aliases (alpha/beta/rc) because they are not controversial
> 
Rather than aliases, I'd like to see a sort order worked out.  Someone will
make both foo-1.0alpha1 and foo-1.0a1 because they don't understand that
they're just supposed to be aliases.  Better to document explicitly what
happens in that case.

> 3 - stick with "post" "dev" for the post and pre-release tags because
> *we need them to sort development versions and post versions* in
> installers, and because *they don't hurt* for people that are not
> publishing such versions. They will be able to use their own dev
> markers internally if they want.
> 
+1

> Next, once the PEP is edited, I am proposing to move this discussion
> in python-dev,
> for another round, and eventually have Guido accept it or reject it
> and move forward with PEP 345.
> 
> Because as far as I am concerned, even if we change the syntax in PEP
> 386 a million times, some people will not like it at the end.
> 
Agreed :-)

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091128/540db78a/attachment.pgp>

From floris.bruynooghe at gmail.com  Sat Nov 28 20:02:05 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Sat, 28 Nov 2009 19:02:05 +0000
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>
References: <94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<20091128112838.GB25937@laurie.devork>
	<94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>
Message-ID: <20091128190205.GA29410@laurie.devork>

On Sat, Nov 28, 2009 at 05:07:01PM +0100, Tarek Ziad? wrote:
> On Sat, Nov 28, 2009 at 12:28 PM, Floris Bruynooghe
> <floris.bruynooghe at gmail.com> wrote:
> > This is an important point. ?I tought the aim of PEP 386 was to create
> > a version scheme that can represent every version number developers
> > want to express, not to create one that allows everyones favourite
> > syntax.
> >
> 
> Absolutely,

Maybe worth explaining this in the abstract of the motivation of the
PEP to focus discussion?  Otherwise I reckon chances are pretty high
that the syntax flameware will flare up again when this gets proposed
on python-dev (or here if there's need for another round).


> So If the current proposal works for all cases (e.g. people can
> translate their schemes
> into PEP 386 one), I am proposing to:
[...]

+1, I think that's a good approach
(+0 on the aliases part though)


Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From mal at egenix.com  Sat Nov 28 20:11:20 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Sat, 28 Nov 2009 20:11:20 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<4B0D698B.4080708@egenix.com>
	<4B0D6CE9.50306@egenix.com>	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>	<4B0D8DE3.1070706@egenix.com>	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>	<4B0D991D.3020502@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
Message-ID: <4B1175D8.4090107@egenix.com>

Tarek Ziad? wrote:
> On Thu, Nov 26, 2009 at 1:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Here's another take at a minimal change to the format which
>> includes the things we discussed, adds a few more aliases
>> for the "post" and "dev" markers and also adds optional
>> underscores for more readability.
>>
>> VERSION_RE = re.compile(r'''
>>    ^
>>    (?P<version>\d+\.\d+)          # minimum 'N.N'
>>    (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
>>    (?:                            # pre-release tag
>>        _?
>>        (?P<prerel>(a|alpha|b|beta|c|rc))
>>        _?
>>        (?P<prerelversion>\d+(?:\.\d+)*)
>>    )?
>>    (?P<postdev>
>>        (\.(post|fix|sp)_?(?P<post>\d+))?
>>        (\.(dev|pre|build|nightly)_?(?P<dev>\d+))?
>>    )?
>>    $''', re.VERBOSE + re.I)
>>
>> Examples:
>>
>>  3.2.0a0.20091125
>> < 3.2.0a1
>> = 3.2.0_alpha_1
>> < 3.2.0a1.20091125
>> < 3.2.0rc1
>> = 3.2.0c1
>> < 3.2.0rc1.20091125
>> = 3.2.0_rc1.20091125
>> < 3.2.0
>> < 3.2.0.sp1
>> = 3.2.0.fix1
>> = 3.2.0.post1
>>
>> One nit I have with the order of the N.N.devN version is that it is regarded
>> "more" than any of the pre-release tags, but less than the release itself:
>>
>>  1.0a1
>> < 1.0rc1
>> < 1.0.dev456
>> < 1.0
>>
>> IMHO, the order should be:
>>
>>  1.0.dev456
>> < 1.0a1.dev456
>> < 1.0a1
>> < 1.0rc1.dev456
>> < 1.0rc1
>> < 1.0
>>
>> since the .dev versions are really only snapshots leading up to
>> some release, i.e. 1.0.dev456 is a snapshot leading up to the
>> first pre-release of the 1.0 :-)
> 
> That's right, that's a bug in the PEP and/or verlib.py
> 
> The changes look good to me.
> 
> If you made some changes in verlib, would you mind pushing them back at
> 
> http://bitbucket.org/tarek/distutilsversion ?

I haven't made any changes to verlib, only to the RE.

Here's the test for it:

import re

VERSION_RE = re.compile(r'''
    ^
    (?P<version>\d+\.\d+)          # minimum 'N.N'
    (?P<extraversion>(?:\.\d+)*)   # any number of extra '.N' segments
    (?:                            # pre-release tag
        _?
        (?P<prerel>(a|alpha|b|beta|c|rc))
        _?
        (?P<prerelversion>\d+(?:\.\d+)*)
    )?
    (?P<postdev>
        (\.(post|fix|sp)_?(?P<post>\d+))?
        (\.(dev|pre|build|nightly)_?(?P<dev>\d+))?
    )?
    $''', re.VERBOSE + re.I)

test = """\
 3.2.0a0.20091125
 < 3.2.0a1
 = 3.2.0_alpha_1
 < 3.2.0a1.20091125
 < 3.2.0rc1
 = 3.2.0c1
 < 3.2.0rc1.20091125
 = 3.2.0_rc1.20091125
 < 3.2.0
 < 3.2.0.sp1
 = 3.2.0.fix1
 = 3.2.0.post1
"""

testcases = [x.strip() for x in test.split()]

comp = None
version = None
for x in testcases:
    if x in '<=':
        continue
    assert VERSION_RE.match(x), x


-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 28 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From pje at telecommunity.com  Sat Nov 28 22:00:46 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 28 Nov 2009 16:00:46 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <200911280950.nAS9oa2F025917@theraft.openend.se>
References: <4B0D6CE9.50306@egenix.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
Message-ID: <20091128210050.98E553A4115@sparrow.telecommunity.com>

At 10:50 AM 11/28/2009 +0100, Laura Creighton wrote:
>But I think that it is the other way around ... what we want is a
>timestamp.  The algorithm is for guessing which version is ealier
>in the absence of a timestamp.

You're assuming a process with no branches, where e.g. version 1.2 is 
being updated while 2.0 is being developed.  Timestamps are not a 
general substitute for versions.


From pje at telecommunity.com  Sat Nov 28 22:05:15 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 28 Nov 2009 16:05:15 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <94bdd2610911280041u5dee8db8s98fcc9f1f8883c11@mail.gmail.co
 m>
References: <4B0D6CE9.50306@egenix.com> <20091126195549.GA29498@laurie.devork>
	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<94bdd2610911280041u5dee8db8s98fcc9f1f8883c11@mail.gmail.com>
Message-ID: <20091128210518.62A0C3A4116@sparrow.telecommunity.com>

At 09:41 AM 11/28/2009 +0100, Tarek Ziad? wrote:
>That's completely wrong, the proposal is a benefit for all of us,
>because it standardizes something that is already being done.

You seem to have misunderstood me; I'm objecting to Ben Finney's 
"simple alphanumeric sort", not to PEP 386 in general.


>PEP 386 propose a scheme to be adopted by developers or tools, but if some
>people want to stick with their own internal version scheme for
>development versions or post/pre release versions, they can do it
>without any problem. And they don't have to follow
>any PEP 386 convention for their internal work.

This is not actually true, since developers working in a team 
situation can be sharing and building binary releases of these packages.


>So you have two choices:
>- an implicit, heuristic ordering (that's what is happening today)
>- a explicit, documented ordering. that's the goal of PEP 386.

Setuptools' version scheme *is* explicit and documented -- as you 
should know, since I pointed you to those docs when you were writing PEP 386.


From pje at telecommunity.com  Sat Nov 28 22:10:02 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sat, 28 Nov 2009 16:10:02 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B1175D8.4090107@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
Message-ID: <20091128211007.311B93A4119@sparrow.telecommunity.com>

At 08:11 PM 11/28/2009 +0100, M.-A. Lemburg wrote:
>Tarek Ziad? wrote:
> > On Thu, Nov 26, 2009 at 1:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> >> Here's another take at a minimal change to the format which
> >> includes the things we discussed, adds a few more aliases
> >> for the "post" and "dev" markers and also adds optional
> >> underscores for more readability.

Please don't add underscores to the syntax -- they will cause 
problems with the filename escaping and parsing used today by 
setuptools and compatible tools, and will produce inconsistent 
comparisons between "rational" versions and the version schemes 
supported by setuptools.

Ideally, it would be best to keep PEP 386 versions a strict subset of 
setuptools-supported versions, to minimize migration difficulties.


From ben+python at benfinney.id.au  Sat Nov 28 22:53:50 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 29 Nov 2009 08:53:50 +1100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
References: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<20091128112838.GB25937@laurie.devork>
	<94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>
Message-ID: <87d432tixd.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> 3 - stick with "post" "dev" for the post and pre-release tags because
> *we need them to sort development versions and post versions* in
> installers, and because *they don't hurt* for people that are not
> publishing such versions.

I disagree on both counts ? these keywords are not needed, and are not
harmless ? and have demonstrated otherwise, so I am convinced the above
conclusion is a mistake.

However, I don't intend to repeat the discussion further (rejoice!). Do
with the information what you will.

-- 
 \        ?I washed a sock. Then I put it in the dryer. When I took it |
  `\                                 out, it was gone.? ?Steven Wright |
_o__)                                                                  |
Ben Finney


From g.brandl at gmx.net  Sat Nov 28 22:58:19 2009
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 28 Nov 2009 22:58:19 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <87pr72uaq1.fsf@benfinney.id.au>
References: <4B0D6CE9.50306@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<20091126195549.GA29498@laurie.devork>	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>	<20091127103958.GU2966@piotro.eu>	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>	<873a40xh7k.fsf_-_@benfinney.id.au>	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>	<87ocmnwnkl.fsf@benfinney.id.au>
	<4B106AF4.401@trueblade.com>	<877htbwk1t.fsf@benfinney.id.au>
	<4B108FA4.6080601@trueblade.com>	<87fx7zultz.fsf@benfinney.id.au>
	<4B110411.1040904@trueblade.com> <87pr72uaq1.fsf@benfinney.id.au>
Message-ID: <hes6ei$isn$1@ger.gmane.org>

Ben Finney schrieb:
> Eric Smith <eric at trueblade.com> writes:
> 
>> Ben Finney wrote:
>> > That is, you want me to propose an example version string X. I'm
>> > asking for you to tell me corresponding examples of version strings
>> > W and Y where:
>> >
>> >     W < X < Y
>> >
>> > since the comparison semantics are what we're discussing here.
>> >
>>
>> W = 3.3.0
>> Y = 3.3.1
>>
>> If you're going to suggest 3.3.0.99 or some such
> 
> Yes. Or any version string that would obviously fall between the two.

A specific version comparison scheme will never be obvious to all people
at the same time.  However we can provide one that is obvious to *most*,
and it is not the one you're pushing.

>> I think that won't work. It's important that code sees that as
>> 3.3.1.<something>. I've seen code that fails when the pre-release
>> versions have a different MAJOR.MINOR.MICRO than the final version.
> 
> It's exactly that kind of code that I don't feel needs to be
> accommodated. It's clearly causing ambiguities and contortions, and now
> we talk about needing Python-specific version comparison algorithms in
> order to automate the comparison.

The need for pre/post-release versions is hardly Python-specific.  It is
used by a large fraction of developers in some way or other.

> Why is this worth the trouble, when the standard could simply describe
> obvious-to-everyone version comparison semantics that are far easier for
> everyone to get right?

Very true.  Surely, it will be obvious to everyone that 3.1alpha1 is a
version greater than 3.1.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From chris at simplistix.co.uk  Sat Nov 28 23:33:49 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Sat, 28 Nov 2009 22:33:49 +0000
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
Message-ID: <4B11A54D.1020706@simplistix.co.uk>

Tarek Ziad? wrote:
> Hello
> 
> As suggested in Catalog-SIG by some people, I would like to propose
> the addition of three more fields for 1.2:
> 
> "Repository-URL"
>     A string containing the URL for the package's repository.
>    Example:
>         http://svn.python.org/projects/python/trunk/
> 
> "Repository-Browse-URL"
>      A string containing the URL for the package's browsable repository.
>    Example:
>       http://svn.python.org/view/python/trunk
> 
> "Bug-Tracker-URL"
>    A string containing the URL for the package's bug tracker
>    Example:
>       http://bugs.python.org/
> 
> 
> Any thoughts ?

Hmmm, I thought I replied to this but don't see my mail in the archives:

Yes to all of the above, could we also please have:

Mailing-List-URL

Change-Log-URL

Documentation-URL

I see the primary consumer of all 6 of these being PyPI, to give 
potential package users more direct information about where to go in a 
standard format.

thanks,

Chris


From chris at simplistix.co.uk  Sat Nov 28 23:45:33 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Sat, 28 Nov 2009 22:45:33 +0000
Subject: [Distutils] The "incompatible versions" problem
In-Reply-To: <319e029f0911280003q2cb2ba19w58f8600d17fd0ca2@mail.gmail.com>
References: <4B0FC088.50008@simplistix.co.uk>
	<319e029f0911280003q2cb2ba19w58f8600d17fd0ca2@mail.gmail.com>
Message-ID: <4B11A80D.2090407@simplistix.co.uk>

Lennart Regebro wrote:
>> Whereby, setuptools would raise an exception if the following packages were
>> installed:
>>
>> def setup(
>>  ...
>>  install_requires=['zope.component']
>>  )
> 
> I think this is the problem. I understand what you mean, but it's not
> obvious to me at all that this should raise an error in this
> situation.

...I'm not even suggesting it should ;-) I was just presenting the 
problem, and typing a possibility as I was writing.

> But yet, for your suggestion to work, it has to.

Well, per library. The sketch I put down allowed zope.component 4.0 to 
complain if a library requests zope.component, didn't specify 4.0 and 
newer, but yet 4.0+ ended up installed.

> In
> practice, that means that you, to use zope.component >= 4.0, must give
> a version number. But even when an API changes, it may be possible to
> support both versions. And that would then be impossible to do in this
> case.

Well, that's why I put the hint in zope.component's setup function call 
in the example... It would only be used for incompatible API versions, 
otherwise it could be left out, and any newer version would be fine...

Chris


From ziade.tarek at gmail.com  Sun Nov 29 00:04:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 00:04:11 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <87d432tixd.fsf@benfinney.id.au>
References: <94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<20091128112838.GB25937@laurie.devork>
	<94bdd2610911280807r6b3764f8q1e5f30c203b75bfa@mail.gmail.com>
	<87d432tixd.fsf@benfinney.id.au>
Message-ID: <94bdd2610911281504s32e5e60bu5fdafe929a2a200c@mail.gmail.com>

On Sat, Nov 28, 2009 at 10:53 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Tarek Ziad? <ziade.tarek at gmail.com> writes:
>
>> 3 - stick with "post" "dev" for the post and pre-release tags because
>> *we need them to sort development versions and post versions* in
>> installers, and because *they don't hurt* for people that are not
>> publishing such versions.
>
> I disagree on both counts ? these keywords are not needed, and are not
> harmless ? and have demonstrated otherwise, so I am convinced the above
> conclusion is a mistake.

Sorry I failed to understand your demonstration. I fail to see how the
version scheme
is useful and can be used by developers without dev and post versions,
and how these keywords are not needed.

Regards,
Tarek

From ziade.tarek at gmail.com  Sun Nov 29 00:19:43 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 00:19:43 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <4B11A54D.1020706@simplistix.co.uk>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
Message-ID: <94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>

On Sat, Nov 28, 2009 at 11:33 PM, Chris Withers <chris at simplistix.co.uk> wrote:
[..]
>
> Yes to all of the above, could we also please have:
>
> Mailing-List-URL

+1

>
> Change-Log-URL

I don't see a lot of projects with a changelog file available online, rather a
changelog file in the release in CHANGELOG and sometimes a changelog
displayed in the long_description field, so it's shown at PyPI.

Can you detail on this one ?

>
> Documentation-URL

How would this one be different from Home-page ?

Regards
Tarek

From greg.ewing at canterbury.ac.nz  Sun Nov 29 01:02:45 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 29 Nov 2009 13:02:45 +1300
Subject: [Distutils] Common version-comparison semantics for peace love
 and harmony
In-Reply-To: <200911281222.nASCMocd003469@theraft.openend.se>
References: <873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<87y6lrv3q5.fsf@benfinney.id.au> <pje@telecommunity.com>
	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
	<94bdd2610911280127u63f67a04j2743bc3a05c1a8b6@mail.gmail.com>
	<200911280950.nAS9oa2F025917@theraft.openend.se>
	<20091128110145.GA25937@laurie.devork>
	<200911281222.nASCMocd003469@theraft.openend.se>
Message-ID: <4B11BA25.4000106@canterbury.ac.nz>

Laura Creighton wrote:

> Now I want to say 'requires this bugfix'.  Right now I think that if
> I say requires 1.0.2 or later, then people with 1.1 will expect that
> they are ok, when they are not.  Or am I misunderstanding?

In cases like that, I don't think any scheme that relies on
comparing with a single version number will be sufficient.
You really need to specify a more complex requirement,
such as

   1.0.2 <= x < 1.1 or x >= 1.1.2

Even if you have a notion of grouping releases into series,
it can still get complicated. E.g. suppose the bug was only
fixed in the 1.1.3 release of the 1.1 branch -- then you
need to compare with different micro release numbers in
each branch.

-- 
Greg


From ben+python at benfinney.id.au  Sun Nov 29 01:24:18 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 29 Nov 2009 11:24:18 +1100
Subject: [Distutils] PEP 345 - 3 new fields
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
Message-ID: <871vjitbyl.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> On Sat, Nov 28, 2009 at 11:33 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> > Change-Log-URL
>
> I don't see a lot of projects with a changelog file available online,
> rather a changelog file in the release in CHANGELOG and sometimes a
> changelog displayed in the long_description field, so it's shown at
> PyPI.

I think that's a poor practice; the long description shouldn't need to
change every release and shouldn't include a changelog.

> Can you detail on this one ?

I would like to see more projects making their changelog a separate
document, so it could be referenced simply and separately from other
information about the project. This field would encourage that by making
it more visible.

> > Documentation-URL
>
> How would this one be different from Home-page ?

The home page is a ?first impressions? page. It usually doesn't jump
straight into a user's guide and technical documentation and interface
specification.

Many projects have a separate index page specifically for documentation;
the URL of that page would be the value for this field.

-- 
 \           ?Value your freedom or you will lose it, teaches history. |
  `\     ?Don't bother us with politics,? respond those who don't want |
_o__)                               to learn.? ?Richard Stallman, 2002 |
Ben Finney


From tseaver at palladion.com  Sun Nov 29 04:07:38 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Sat, 28 Nov 2009 22:07:38 -0500
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <200911280631.nAS6VFUi013532@theraft.openend.se>
References: <4B0D6CE9.50306@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<20091126195549.GA29498@laurie.devork>	<94bdd2610911261245r5c91a9dl4e6c48d4976b5efb@mail.gmail.com>	<20091127103958.GU2966@piotro.eu>	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>	<873a40xh7k.fsf_-_@benfinney.id.au>	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>	<87ocmnwnkl.fsf@benfinney.id.au>	<20091128010302.35F143A40A8@sparrow.telecommunity.com>	<87y6lrv3q5.fsf@benfinney.id.au>	<20091128030031.D10583A40A8@sparrow.telecommunity.com>
	<200911280631.nAS6VFUi013532@theraft.openend.se>
Message-ID: <hesohm$l8o$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Laura Creighton wrote:
> It occurs to me that this problem would go away if we had a way to 
> ask, for any given version number, 'what was your creation date' and
> the sorting 'earlier' and 'later' by that date.  Can somebody explain
> why we aren't doing this?

Branches / tags for "stable" / "old" release lines are often created
after tags for the "current" / "testing" / "trunk" lines:  creation time
is not a relevant comparator there.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAksR5XoACgkQ+gerLs4ltQ4AFwCgjAsVzCTHLAq/MvqISL+yumRa
3LsAoI0C1dEjDLZvY0JiKtRvxyEQ+Zys
=Lla1
-----END PGP SIGNATURE-----


From ziade.tarek at gmail.com  Sun Nov 29 10:08:22 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 10:08:22 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <871vjitbyl.fsf@benfinney.id.au>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
Message-ID: <94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>

On Sun, Nov 29, 2009 at 1:24 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
[..]
>> Can you detail on this one ?
>
> I would like to see more projects making their changelog a separate
> document, so it could be referenced simply and separately from other
> information about the project. This field would encourage that by making
> it more visible.

How about making it "Change-Log" in that case, and have the content of
the changelog
directly added in the metadata ?  Rather than an url.

>
>> > Documentation-URL
>>
>> How would this one be different from Home-page ?
>
> The home page is a ?first impressions? page. It usually doesn't jump
> straight into a user's guide and technical documentation and interface
> specification.
>
> Many projects have a separate index page specifically for documentation;
> the URL of that page would be the value for this field.

I am 0+ for this one, as I expect the documentation page to be
reachable very easily
from a project's home page.

Regards
Tarek

From ziade.tarek at gmail.com  Sun Nov 29 10:26:56 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 10:26:56 +0100
Subject: [Distutils] Common version-comparison semantics for peace love
	and harmony
In-Reply-To: <20091128210518.62A0C3A4116@sparrow.telecommunity.com>
References: <4B0D6CE9.50306@egenix.com> <20091127103958.GU2966@piotro.eu>
	<94bdd2610911270424jd9fdb2dg103aebfe3555e998@mail.gmail.com>
	<94bdd2610911270431i3f7d2281na94941054b72225b@mail.gmail.com>
	<873a40xh7k.fsf_-_@benfinney.id.au>
	<94bdd2610911270530o5d7e86dbnccafab59c1455da9@mail.gmail.com>
	<87ocmnwnkl.fsf@benfinney.id.au>
	<20091128010302.35F143A40A8@sparrow.telecommunity.com>
	<94bdd2610911280041u5dee8db8s98fcc9f1f8883c11@mail.gmail.com>
	<20091128210518.62A0C3A4116@sparrow.telecommunity.com>
Message-ID: <94bdd2610911290126x19876425u87df1f937704688@mail.gmail.com>

On Sat, Nov 28, 2009 at 10:05 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 09:41 AM 11/28/2009 +0100, Tarek Ziad? wrote:
>>
>> That's completely wrong, the proposal is a benefit for all of us,
>> because it standardizes something that is already being done.
>
> You seem to have misunderstood me; I'm objecting to Ben Finney's "simple
> alphanumeric sort", not to PEP 386 in general.

Ok,

>
>
>> PEP 386 propose a scheme to be adopted by developers or tools, but if some
>> people want to stick with their own internal version scheme for
>> development versions or post/pre release versions, they can do it
>> without any problem. And they don't have to follow
>> any PEP 386 convention for their internal work.
>
> This is not actually true, since developers working in a team situation can
> be sharing and building binary releases of these packages.

As long as they are using and sharing their own tool that respect
their specific version scheme,
where they have a classical scheme for final releases, and a specific
scheme for development releases, how would that be a problem if they
don't publish at PyPI or to others those specific versions ?


>> So you have two choices:
>> - an implicit, heuristic ordering (that's what is happening today)
>> - a explicit, documented ordering. that's the goal of PEP 386.
>
> Setuptools' version scheme *is* explicit and documented -- as you should
> know, since I pointed you to those docs when you were writing PEP 386.

Sorry I was unclear. Implicit was the wrong word. What I wanted to say is that
setuptools will order any version string a project provides without complaining,

IOW, 'Foo' and 'Bar' are valid version numbers with it.

So you have two choices:
 - an loose version scheme, where any string can be a version and will
be compared
   (that's what is happening today)
 - a stricter version scheme, based on a segmented ordering. that's
the goal of PEP 386.


Tarek

From ben+python at benfinney.id.au  Sun Nov 29 14:19:27 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 30 Nov 2009 00:19:27 +1100
Subject: [Distutils] PEP 345 - 3 new fields
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
Message-ID: <87skbxsc2o.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> How about making it "Change-Log" in that case, and have the content of
> the changelog
> directly added in the metadata ?  Rather than an url.

That might be good, if the changelog was in a separate file. It's quite
commonly a very long text file, as it retains the change history of the
code base indefinitely.

-- 
 \      ?Shepherds ? look after their sheep so they can, first, fleece |
  `\   them and second, turn them into meat. That's much more like the |
_o__)      priesthood as I know it.? ?Christopher Hitchens, 2008-10-29 |
Ben Finney


From ziade.tarek at gmail.com  Sun Nov 29 14:26:11 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 14:26:11 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <87skbxsc2o.fsf@benfinney.id.au>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
Message-ID: <94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>

On Sun, Nov 29, 2009 at 2:19 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Tarek Ziad? <ziade.tarek at gmail.com> writes:
>
>> How about making it "Change-Log" in that case, and have the content of
>> the changelog
>> directly added in the metadata ? ?Rather than an url.
>
> That might be good, if the changelog was in a separate file. It's quite
> commonly a very long text file, as it retains the change history of the
> code base indefinitely.
>

Yes, and if it's reStructuredText based, we will be able to do what people
do actually, by integrating a reST-based CHANGELOG file at the end of
long_description:

http://pypi.python.org/pypi/zc.buildout#change-history

From mal at egenix.com  Sun Nov 29 14:34:10 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun, 29 Nov 2009 14:34:10 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091128211007.311B93A4119@sparrow.telecommunity.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<4B0D698B.4080708@egenix.com>
	<4B0D6CE9.50306@egenix.com>	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>	<4B0D8DE3.1070706@egenix.com>	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>	<4B0D991D.3020502@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
Message-ID: <4B127852.7010908@egenix.com>

P.J. Eby wrote:
> At 08:11 PM 11/28/2009 +0100, M.-A. Lemburg wrote:
>> Tarek Ziad? wrote:
>> > On Thu, Nov 26, 2009 at 1:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> >> Here's another take at a minimal change to the format which
>> >> includes the things we discussed, adds a few more aliases
>> >> for the "post" and "dev" markers and also adds optional
>> >> underscores for more readability.
> 
> Please don't add underscores to the syntax -- they will cause problems
> with the filename escaping and parsing used today by setuptools and
> compatible tools, and will produce inconsistent comparisons between
> "rational" versions and the version schemes supported by setuptools.
> 
> Ideally, it would be best to keep PEP 386 versions a strict subset of
> setuptools-supported versions, to minimize migration difficulties.

Filename parsing is a bad idea to begin with. The meta data
incorporated into file names should be read from a meta data file
or database instead, with the filename just being another parameter
in the set of meta data parameters.

Frankly, we are defining a standard for *distutils* and new packages
here - so I'm not sure in what way setuptools compatibility can be
used as argument for limiting the readability of a new standard.
OTOH, I'm sure that by the time distribute can take over the role
of setuptools, such limitations will have been lifted.

Besides, setuptools users can, of course, continue to use version
strings without embedded underscores.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 29 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ben+python at benfinney.id.au  Sun Nov 29 15:05:26 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 30 Nov 2009 01:05:26 +1100
Subject: [Distutils] PEP 345 - 3 new fields
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
Message-ID: <87k4x9s9y1.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> On Sun, Nov 29, 2009 at 2:19 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > [having the changelog in the package metadata] might be good, if the
> > changelog was in a separate file. It's quite commonly a very long
> > text file, as it retains the change history of the code base
> > indefinitely.
>
> Yes, and if it's reStructuredText based, we will be able to do what
> people do actually, by integrating a reST-based CHANGELOG file at the
> end of long_description:

Let's not get ahead of ourselves :-) Asking developers to declare where
their changelog file is will be much easier to get than asking them to
change the format of that file.

Stick to a simple standard of ?this file contains the changelog?, which
will let people simply move their existing changelog of whatever format
into that file. Having it in a predictable location would be a big step
forward, and an easy one to enforce.

-- 
 \          ?Our products just aren't engineered for security.? ?Brian |
  `\             Valentine, senior vice-president of Microsoft Windows |
_o__)                                                development, 2002 |
Ben Finney


From ziade.tarek at gmail.com  Sun Nov 29 15:13:31 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 15:13:31 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B127852.7010908@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
	<4B127852.7010908@egenix.com>
Message-ID: <94bdd2610911290613w19d533a9q9b285156d9d235d@mail.gmail.com>

2009/11/29 M.-A. Lemburg <mal at egenix.com>:
[..]
>>
>> Please don't add underscores to the syntax -- they will cause problems
>> with the filename escaping and parsing used today by setuptools and
>> compatible tools, and will produce inconsistent comparisons between
>> "rational" versions and the version schemes supported by setuptools.
>>
>> Ideally, it would be best to keep PEP 386 versions a strict subset of
>> setuptools-supported versions, to minimize migration difficulties.
>
> Filename parsing is a bad idea to begin with. The meta data
> incorporated into file names should be read from a meta data file
> or database instead, with the filename just being another parameter
> in the set of meta data parameters.

Notice that the metadada files added during installation are the ones
that will be used
to query installed distributions (see PEP 376)

Their syntax is currently :  name + '-' + version + '.egg-info' , with
all occurences of "-" in the name being escaped by "_".

So I don't see any technical problem in using "-" in the version, as
long as the first "-" in the egg-info
name can be used to be able to extract the name and version of the project.

*But*, on a second thought,  I don't see any problem in completely
removing the version from
the egg-info filename in PEP 376, since looping on *.egg-info files
suffices for our APIs, and since
we said that we didn't want to support multiple versions.

>
> Frankly, we are defining a standard for *distutils* and new packages
> here - so I'm not sure in what way setuptools compatibility can be
> used as argument for limiting the readability of a new standard.
> OTOH, I'm sure that by the time distribute can take over the role
> of setuptools, such limitations will have been lifted.
>

Yes, Distribute will follow all PEPs as soon as they are accepted,

Last, as I said in a previous mail, I tend to agree with the people
who said that we should stick with only one way to write the version
scheme for the sake of clarity. e.g. dropping aliases and picking
*one* way to write the markers after major.minor.micro.

I would tend to pick the same scheme than Python for the pre-releases
(and c + rc):

    N.N[.N][(a|b|c|rc)N]

And, for the post/dev markers I think dots are OK for readability,

Tarek

From ziade.tarek at gmail.com  Sun Nov 29 15:16:40 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 15:16:40 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <87k4x9s9y1.fsf@benfinney.id.au>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
	<87k4x9s9y1.fsf@benfinney.id.au>
Message-ID: <94bdd2610911290616q6368f39fq7bbd6965b4bbfd63@mail.gmail.com>

On Sun, Nov 29, 2009 at 3:05 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Tarek Ziad? <ziade.tarek at gmail.com> writes:
>
>> On Sun, Nov 29, 2009 at 2:19 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
>> > [having the changelog in the package metadata] might be good, if the
>> > changelog was in a separate file. It's quite commonly a very long
>> > text file, as it retains the change history of the code base
>> > indefinitely.
>>
>> Yes, and if it's reStructuredText based, we will be able to do what
>> people do actually, by integrating a reST-based CHANGELOG file at the
>> end of long_description:
>
> Let's not get ahead of ourselves :-) Asking developers to declare where
> their changelog file is will be much easier to get than asking them to
> change the format of that file.
>
> Stick to a simple standard of ?this file contains the changelog?, which
> will let people simply move their existing changelog of whatever format
> into that file. Having it in a predictable location would be a big step
> forward, and an easy one to enforce.
>

That doesn't differ from what we have today with long_description:

"""
Description (optional)

    A longer description of the package that can run to several
paragraphs. Software that deals with metadata should not assume any
maximum size for this field, though people shouldn't include their
instruction manual as the description.

    The contents of this field can be written using reStructuredText
markup [1]. For programs that work with the metadata, supporting
markup is optional; programs can also display the contents of the
field as-is. This means that authors should be conservative in the
markup they use.
""""

IOW, Changelog would be a similar field in the metada, meaning that a
simple text works too

Regards,
Tarek

From pje at telecommunity.com  Sun Nov 29 19:50:01 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Sun, 29 Nov 2009 13:50:01 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B127852.7010908@egenix.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0D698B.4080708@egenix.com> <4B0D6CE9.50306@egenix.com>
	<94bdd2610911251020hf1acfa1q4a22c777259e6446@mail.gmail.com>
	<4B0D8DE3.1070706@egenix.com>
	<94bdd2610911251226s1b1244e3ua8d198fc4448b331@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
	<4B127852.7010908@egenix.com>
Message-ID: <20091129185007.80F2C3A40A8@sparrow.telecommunity.com>

At 02:34 PM 11/29/2009 +0100, M.-A. Lemburg wrote:
>P.J. Eby wrote:
> > At 08:11 PM 11/28/2009 +0100, M.-A. Lemburg wrote:
> >> Tarek Ziad? wrote:
> >> > On Thu, Nov 26, 2009 at 1:08 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> >> >> Here's another take at a minimal change to the format which
> >> >> includes the things we discussed, adds a few more aliases
> >> >> for the "post" and "dev" markers and also adds optional
> >> >> underscores for more readability.
> >
> > Please don't add underscores to the syntax -- they will cause problems
> > with the filename escaping and parsing used today by setuptools and
> > compatible tools, and will produce inconsistent comparisons between
> > "rational" versions and the version schemes supported by setuptools.
> >
> > Ideally, it would be best to keep PEP 386 versions a strict subset of
> > setuptools-supported versions, to minimize migration difficulties.
>
>Filename parsing is a bad idea to begin with.

Practicality beats purity.

>Frankly, we are defining a standard for *distutils* and new packages
>here - so I'm not sure in what way setuptools compatibility can be
>used as argument for limiting the readability of a new standard.
>OTOH, I'm sure that by the time distribute can take over the role
>of setuptools, such limitations will have been lifted.

Unless we're talking about strictly Python 3 here (in which case all 
bets are off and compatibility isn't a prime concern) Distribute is 
limited by the need to be able to consume setuptools-based packages, 
or it loses a key user incentive (i.e., its "just works"-ness).


>Besides, setuptools users can, of course, continue to use version
>strings without embedded underscores.

Setuptools users also consume other people's packages, which means 
they will wind up with bad version comparisons in that case.  And 
Distribute can't work around this unless it escapes underscores as 
some other character when it makes files for consumption by those 
tools, since it will otherwise lose the ability to parse those 
versions when they come back in.

If you want mass uptake of a standard, you need to take these sort of 
things into consideration; incentive engineering is not optional when 
you want grass-roots support for an independently-created standard.

WSGI and setuptools have been widely adopted in spite of their 
technical and ideological flaws, because they had good incentive engineering.

Or, in other words, because practicality beats purity, every single time.


From ziade.tarek at gmail.com  Sun Nov 29 21:15:05 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 29 Nov 2009 21:15:05 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091129185007.80F2C3A40A8@sparrow.telecommunity.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
	<4B127852.7010908@egenix.com>
	<20091129185007.80F2C3A40A8@sparrow.telecommunity.com>
Message-ID: <94bdd2610911291215u21724e59lddd2b9a0ac78857a@mail.gmail.com>

2009/11/29 P.J. Eby <pje at telecommunity.com>:
[..]
>
> WSGI and setuptools have been widely adopted in spite of their technical and
> ideological flaws, because they had good incentive engineering.
>
> Or, in other words, because practicality beats purity, every single time.

Do you mean here that this independently-created standard, this good
incentive engeneering a.k.a.
Setuptools, is doomed not to evolve anymore ?   e.g. to adapt its
standard to a common standard that is going to raise and be added in
stldib at some point ?

That's not what Distribute will aim at.  We will want to provide the
compatibility bridge for people that uses setuptools API to adopt the
upcoming standards. And if this happen in Python 2.7, that might be a
simple flag :

  use_stdlib_standard = True

IOW third party software should adapt themselves to Python and its
stdlib, through deprecations etc., not the contrary.

Regards,
Tarek

From ben+python at benfinney.id.au  Sun Nov 29 22:32:19 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 30 Nov 2009 08:32:19 +1100
Subject: [Distutils] PEP 345 - 3 new fields
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
	<87k4x9s9y1.fsf@benfinney.id.au>
	<94bdd2610911290616q6368f39fq7bbd6965b4bbfd63@mail.gmail.com>
Message-ID: <87fx7xrp98.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> On Sun, Nov 29, 2009 at 3:05 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > Stick to a simple standard of ?this file contains the changelog?,
> > which will let people simply move their existing changelog of
> > whatever format into that file. Having it in a predictable location
> > would be a big step forward, and an easy one to enforce.
>
> That doesn't differ from what we have today with long_description:

That ignores what I thought we'd just agreed in this thread: that the
changelog should be *in a separate file*. Putting it into one field
among many ignores the fact that it's often long, and growing every
release.

-- 
 \           ?It ain't so much the things we don't know that get us in |
  `\    trouble. It's the things we know that ain't so.? ?Artemus Ward |
_o__)                                     (1834?1867), U.S. journalist |
Ben Finney


From pje at telecommunity.com  Mon Nov 30 06:00:30 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 00:00:30 -0500
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <94bdd2610911291215u21724e59lddd2b9a0ac78857a@mail.gmail.co
 m>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
	<4B127852.7010908@egenix.com>
	<20091129185007.80F2C3A40A8@sparrow.telecommunity.com>
	<94bdd2610911291215u21724e59lddd2b9a0ac78857a@mail.gmail.com>
Message-ID: <20091130050036.5BE833A40A8@sparrow.telecommunity.com>

At 09:15 PM 11/29/2009 +0100, Tarek Ziad? wrote:
>2009/11/29 P.J. Eby <pje at telecommunity.com>:
>[..]
> >
> > WSGI and setuptools have been widely adopted in spite of their 
> technical and
> > ideological flaws, because they had good incentive engineering.
> >
> > Or, in other words, because practicality beats purity, every single time.
>
>Do you mean here that this independently-created standard, this good
>incentive engeneering a.k.a.
>Setuptools, is doomed not to evolve anymore ?   e.g. to adapt its
>standard to a common standard that is going to raise and be added in
>stldib at some point ?

I'm saying that ignoring backwards compatibility (as MAL and Ben have 
proposed) is bad incentive engineering.

I don't have any disagreement with what you're doing wrt PEP 386, 
btw; we already had all the technical issue discussion necessary back 
when you first wrote it.  I've just been chiming in on  a couple of 
threads where people have proposed changes that would break backward 
compatibility and present user experience.


From ben+python at benfinney.id.au  Mon Nov 30 08:11:57 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 30 Nov 2009 18:11:57 +1100
Subject: [Distutils] PEP 386 status - last round here ?
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>
	<4B0D991D.3020502@egenix.com>
	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>
	<4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
	<4B127852.7010908@egenix.com>
	<20091129185007.80F2C3A40A8@sparrow.telecommunity.com>
	<94bdd2610911291215u21724e59lddd2b9a0ac78857a@mail.gmail.com>
	<20091130050036.5BE833A40A8@sparrow.telecommunity.com>
Message-ID: <87tywcpjuq.fsf@benfinney.id.au>

"P.J. Eby" <pje at telecommunity.com> writes:

> I'm saying that ignoring backwards compatibility (as MAL and Ben have
> proposed) is bad incentive engineering.

I deny this characterisation. PEP 386, by declaring a distinction
between version string comparisons that do or do not conform to a
standard, is *necessarily* backward-incompatible: there will be version
string comparison semantics currently in use that do not meet the
standard.

So if anything, it's PEP 386 that breaks backward compatibility. That's
unavoidable, of course, and I can only trust that the proponents of PEP
386 have a means of dealing with those existing version comparisons that
won't meet the standard.

What I'm proposing is a modification to the specification; I'm not
introducing backward incompatibility, since that's inherent in the
standardisation effort.

-- 
 \        ?The flattening of underwear with pleasure is the job of the |
  `\                                  chambermaid.? ?hotel, Yugoslavia |
_o__)                                                                  |
Ben Finney


From chris at simplistix.co.uk  Mon Nov 30 10:13:31 2009
From: chris at simplistix.co.uk (Chris Withers)
Date: Mon, 30 Nov 2009 09:13:31 +0000
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>	
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
Message-ID: <4B138CBB.3000402@simplistix.co.uk>

Tarek Ziad? wrote:
> On Sat, Nov 28, 2009 at 11:33 PM, Chris Withers <chris at simplistix.co.uk> wrote:
> [..]
>> Yes to all of the above, could we also please have:
>>
>> Mailing-List-URL
> 
> +1
> 
>> Change-Log-URL
> 
> I don't see a lot of projects with a changelog file available online,

...except every single Zope-related egg ;-)

These currently get put in the long-description, often up at the top, 
usually making the actual description totally unreadable.

Having the option to put a url in a standard field would encourage 
people to just put a link to the changelog.

> Can you detail on this one ?
> 
>> Documentation-URL
> 
> How would this one be different from Home-page ?

Take my tiny errorlogger package, the homepage url is here:

http://www.simplistix.co.uk/software/python/errorhandler

Documentation is here:

http://packages.python.org/errorhandler/

These are optional fields, which I'd like to see PyPI display if they're 
present, but we need the ability to add them first! :-)

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk


From ziade.tarek at gmail.com  Mon Nov 30 10:20:56 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 10:20:56 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <87fx7xrp98.fsf@benfinney.id.au>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
	<87k4x9s9y1.fsf@benfinney.id.au>
	<94bdd2610911290616q6368f39fq7bbd6965b4bbfd63@mail.gmail.com>
	<87fx7xrp98.fsf@benfinney.id.au>
Message-ID: <94bdd2610911300120n5a51b8adifd675c4bc514ac6a@mail.gmail.com>

On Sun, Nov 29, 2009 at 10:32 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Tarek Ziad? <ziade.tarek at gmail.com> writes:
>
>> On Sun, Nov 29, 2009 at 3:05 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
>> > Stick to a simple standard of ?this file contains the changelog?,
>> > which will let people simply move their existing changelog of
>> > whatever format into that file. Having it in a predictable location
>> > would be a big step forward, and an easy one to enforce.
>>
>> That doesn't differ from what we have today with long_description:
>
> That ignores what I thought we'd just agreed in this thread: that the
> changelog should be *in a separate file*. Putting it into one field
> among many ignores the fact that it's often long, and growing every
> release.

It can be a file of course in your distribution, but at the end, it
has to be a field in the metadata,
like long_description is.

Here's an example of what could be done in your setup.py:

====

   with open('README.txt') as f:
       README = f.read()

   with open('CHANGELOG') as f:
       CHANGELOG = f.read()

   setup(name='foo', ..., long_description=README, changelog=CHANGELOG)

====

Regards
Tarek

From mal at egenix.com  Mon Nov 30 10:49:55 2009
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 30 Nov 2009 10:49:55 +0100
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <20091130050036.5BE833A40A8@sparrow.telecommunity.com>
References: <94bdd2610911240147s3ae4880bta1839957988b972b@mail.gmail.com>	<4B0D991D.3020502@egenix.com>	<94bdd2610911251333r3824d09aw5fea7b0d21705135@mail.gmail.com>	<4B0DB159.8010501@egenix.com>
	<4B0E6FC2.5020006@egenix.com>	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>	<4B1175D8.4090107@egenix.com>	<20091128211007.311B93A4119@sparrow.telecommunity.com>	<4B127852.7010908@egenix.com>	<20091129185007.80F2C3A40A8@sparrow.telecommunity.com>	<94bdd2610911291215u21724e59lddd2b9a0ac78857a@mail.gmail.com>
	<20091130050036.5BE833A40A8@sparrow.telecommunity.com>
Message-ID: <4B139543.9020606@egenix.com>

P.J. Eby wrote:
> At 09:15 PM 11/29/2009 +0100, Tarek Ziad? wrote:
>> 2009/11/29 P.J. Eby <pje at telecommunity.com>:
>> [..]
>> >
>> > WSGI and setuptools have been widely adopted in spite of their
>> technical and
>> > ideological flaws, because they had good incentive engineering.
>> >
>> > Or, in other words, because practicality beats purity, every single
>> time.
>>
>> Do you mean here that this independently-created standard, this good
>> incentive engeneering a.k.a.
>> Setuptools, is doomed not to evolve anymore ?   e.g. to adapt its
>> standard to a common standard that is going to raise and be added in
>> stldib at some point ?
> 
> I'm saying that ignoring backwards compatibility (as MAL and Ben have
> proposed) is bad incentive engineering.

Ignoring backwards compatibility is a bad thing, IMHO. Breaking
forward compatibility (sometimes) is a good thing, since it allows
evolution to find new better paths of development.

Fact is, you are just misinterpreting what I wrote. You can't expect
setuptools as-of today to work with a standard that is devised to
be implemented by future tools.

What you are referring to is breaking forward compatibility, ie.
you expect today's setuptools to work with future packages that
will be written against a future standard.

New packages relying on the the new version format will, of course,
require and use a package manager that supports the new format.

If setuptools doesn't want to support it, that's fine. I'm sure
Distribute will ... and could even support old setuptools packages
by adding a --use-old-style-setuptools-package-format option ;-) to
Distribute, so I don't see much of a problem with breaking
forward compatibility in this case.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Nov 30 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ben+python at benfinney.id.au  Mon Nov 30 13:20:27 2009
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 30 Nov 2009 23:20:27 +1100
Subject: [Distutils] PEP 345 - 3 new fields
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<4B11A54D.1020706@simplistix.co.uk>
	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
	<87k4x9s9y1.fsf@benfinney.id.au>
	<94bdd2610911290616q6368f39fq7bbd6965b4bbfd63@mail.gmail.com>
	<87fx7xrp98.fsf@benfinney.id.au>
	<94bdd2610911300120n5a51b8adifd675c4bc514ac6a@mail.gmail.com>
Message-ID: <87pr70p5kk.fsf@benfinney.id.au>

Tarek Ziad? <ziade.tarek at gmail.com> writes:

> On Sun, Nov 29, 2009 at 10:32 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > [?] I thought we'd just agreed in this thread: that the changelog
> > should be *in a separate file*. Putting it into one field among many
> > ignores the fact that it's often long, and growing every release.
>
> It can be a file of course in your distribution, but at the end, it
> has to be a field in the metadata,

I don't understand the distinction you're making here. What is the
difference between ?in your distribution? versus ?in the end??

Are you making a distinction between ?source distribution (sdist)?
versus ?installed package on the end user's computer?? Or something
else?

-- 
 \              ?Dvorak users of the world flgkd!? ?Kirsten Chevalier, |
  `\                                                rec.humor.oracle.d |
_o__)                                                                  |
Ben Finney


From eric at trueblade.com  Mon Nov 30 14:19:41 2009
From: eric at trueblade.com (Eric Smith)
Date: Mon, 30 Nov 2009 08:19:41 -0500
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <4B138CBB.3000402@simplistix.co.uk>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>		<4B11A54D.1020706@simplistix.co.uk>	<94bdd2610911281519o372599bdyc6b3201966066322@mail.gmail.com>
	<4B138CBB.3000402@simplistix.co.uk>
Message-ID: <4B13C66D.4060206@trueblade.com>

Chris Withers wrote:
> Tarek Ziad? wrote:
>> On Sat, Nov 28, 2009 at 11:33 PM, Chris Withers 
>> <chris at simplistix.co.uk> wrote:
>> [..]
>>> Yes to all of the above, could we also please have:
>>>
>>> Mailing-List-URL
>>
>> +1
>>
>>> Change-Log-URL
>>
>> I don't see a lot of projects with a changelog file available online,
> 
> ...except every single Zope-related egg ;-)
> 
> These currently get put in the long-description, often up at the top, 
> usually making the actual description totally unreadable.
> 
> Having the option to put a url in a standard field would encourage 
> people to just put a link to the changelog.

Why does it need to be in a standard field? Who's the consumer of this, 
and what are they doing with the value? Can't you just stick the URL in 
the long description?


From cool-rr at cool-rr.com  Mon Nov 30 15:09:46 2009
From: cool-rr at cool-rr.com (Ram Rachum)
Date: Mon, 30 Nov 2009 14:09:46 +0000 (UTC)
Subject: [Distutils] =?utf-8?q?Bundling_pkg=5Fresources?=
Message-ID: <loom.20091130T150801-770@post.gmane.org>

I've asked before about bundling Distribute. But now I ask, is it possible to
pull out the `pkg_resources` module from the Distribute folder and bundle only 
that with my project?

Also, is that `pkg_resources` module different than the one that comes with 
setuptools?

Thanks,
Ram.


From cournape at gmail.com  Mon Nov 30 16:09:15 2009
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 1 Dec 2009 00:09:15 +0900
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <87pr70p5kk.fsf@benfinney.id.au>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
	<87k4x9s9y1.fsf@benfinney.id.au>
	<94bdd2610911290616q6368f39fq7bbd6965b4bbfd63@mail.gmail.com>
	<87fx7xrp98.fsf@benfinney.id.au>
	<94bdd2610911300120n5a51b8adifd675c4bc514ac6a@mail.gmail.com>
	<87pr70p5kk.fsf@benfinney.id.au>
Message-ID: <5b8d13220911300709r109cf7f0hd5dbbb2ddc3339@mail.gmail.com>

On Mon, Nov 30, 2009 at 9:20 PM, Ben Finney <ben+python at benfinney.id.au> wrote:

>
> I don't understand the distinction you're making here. What is the
> difference between ?in your distribution? versus ?in the end??

It can be a file in your sdist, as long as its content is made
available to the metadata as a string, for example by generating the
metadata from the file in your setup.py

David

From ziade.tarek at gmail.com  Mon Nov 30 16:17:28 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 16:17:28 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <87pr70p5kk.fsf@benfinney.id.au>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<871vjitbyl.fsf@benfinney.id.au>
	<94bdd2610911290108x2d10025o39f71ab712fcd474@mail.gmail.com>
	<87skbxsc2o.fsf@benfinney.id.au>
	<94bdd2610911290526k609777f5j9cf63e63fd83c2c9@mail.gmail.com>
	<87k4x9s9y1.fsf@benfinney.id.au>
	<94bdd2610911290616q6368f39fq7bbd6965b4bbfd63@mail.gmail.com>
	<87fx7xrp98.fsf@benfinney.id.au>
	<94bdd2610911300120n5a51b8adifd675c4bc514ac6a@mail.gmail.com>
	<87pr70p5kk.fsf@benfinney.id.au>
Message-ID: <94bdd2610911300717l4a8b6cd8h87f9d845df6d8f9b@mail.gmail.com>

On Mon, Nov 30, 2009 at 1:20 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Tarek Ziad? <ziade.tarek at gmail.com> writes:
>
>> On Sun, Nov 29, 2009 at 10:32 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
>> > [?] I thought we'd just agreed in this thread: that the changelog
>> > should be *in a separate file*. Putting it into one field among many
>> > ignores the fact that it's often long, and growing every release.
>>
>> It can be a file of course in your distribution, but at the end, it
>> has to be a field in the metadata,
>
> I don't understand the distinction you're making here. What is the
> difference between ?in your distribution? versus ?in the end??

You have to make a distinction between the "long_description" option
in setup.py, and the "Description" metadata field, that lands in the
PKG-INFO file (==PEP 345 fields)

The PKG-INFO file, that contains all the metadata, is built by a call
to "python setup.py sdist" for example , is shipped within your
distribution, and read for instance by PyPI.

IOW, if we are going to add a distinct field for "Changelog", this
will not be a file path, but an URL *or* a raw text exactly like the
"Description" field.

Having a file for your changelog, that you are reading and using to
fill a metadata field is just a implementation detail.


Regards
Tarek

From ziade.tarek at gmail.com  Mon Nov 30 17:10:21 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 17:10:21 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <loom.20091130T150801-770@post.gmane.org>
References: <loom.20091130T150801-770@post.gmane.org>
Message-ID: <94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>

On Mon, Nov 30, 2009 at 3:09 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> I've asked before about bundling Distribute. But now I ask, is it possible to
> pull out the `pkg_resources` module from the Distribute folder and bundle only
> that with my project?

Technically, yes : it's a standalone module that depends only on the stdlib.

What would be the use case for this ?  I don't think it's the best
practice to bundle
other projects modules like that in most cases.

>
> Also, is that `pkg_resources` module different than the one that comes with
> setuptools?

It is the same API and behavior.  It has a few internals changes though to deal
with the incompatibility with the Setuptools project, but is meant to work
as a full replacement.

Regards,
Tarek

From floris.bruynooghe at gmail.com  Mon Nov 30 17:22:42 2009
From: floris.bruynooghe at gmail.com (Floris Bruynooghe)
Date: Mon, 30 Nov 2009 16:22:42 +0000
Subject: [Distutils] PEP 386 status - last round here ?
In-Reply-To: <4B139543.9020606@egenix.com>
References: <4B0DB159.8010501@egenix.com> <4B0E6FC2.5020006@egenix.com>
	<94bdd2610911260936q755c2cb4w37d4429a22a3796e@mail.gmail.com>
	<4B1175D8.4090107@egenix.com>
	<20091128211007.311B93A4119@sparrow.telecommunity.com>
	<4B127852.7010908@egenix.com>
	<20091129185007.80F2C3A40A8@sparrow.telecommunity.com>
	<94bdd2610911291215u21724e59lddd2b9a0ac78857a@mail.gmail.com>
	<20091130050036.5BE833A40A8@sparrow.telecommunity.com>
	<4B139543.9020606@egenix.com>
Message-ID: <20091130162242.GA16442@laurie.devork>

On Mon, Nov 30, 2009 at 10:49:55AM +0100, M.-A. Lemburg wrote:
> so I don't see much of a problem with breaking
> forward compatibility in this case.

Indeed, but equally so I don't see an advantage in breaking forward
compatibility in this case (i.e. underscores in PEP 386 don't allow us
to express something that we otherwise couldn't) so PJE's argument of
practicality vs purity still wins IMHO.

And besides setuptools isn't alone in not liking underscores in
versions, it's just that the others manually translate the versions
before using it so can work around it.

Regards
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

From cool-rr at cool-rr.com  Mon Nov 30 17:41:42 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 30 Nov 2009 18:41:42 +0200
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
Message-ID: <a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>

On Mon, Nov 30, 2009 at 6:10 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> On Mon, Nov 30, 2009 at 3:09 PM, Ram Rachum <cool-rr at cool-rr.com> wrote:
> > I've asked before about bundling Distribute. But now I ask, is it
> possible to
> > pull out the `pkg_resources` module from the Distribute folder and bundle
> only
> > that with my project?
>
> Technically, yes : it's a standalone module that depends only on the
> stdlib.
>

Great. So I can just yank out the `pkg_resources.py` file and it should
work?


> What would be the use case for this ?  I don't think it's the best
> practice to bundle
> other projects modules like that in most cases.
>
>
Up to now I've been requiring my users to install Distribute, but one of
them expressed reluctance to install it, partly because of the shadowing of
setuptools.

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091130/83202bf4/attachment.htm>

From ziade.tarek at gmail.com  Mon Nov 30 17:59:09 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 17:59:09 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
Message-ID: <94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>

On Mon, Nov 30, 2009 at 5:41 PM, cool-RR <cool-rr at cool-rr.com> wrote:
[..]
>> What would be the use case for this ? ?I don't think it's the best
>> practice to bundle
>> other projects modules like that in most cases.
>>
>
> Up to now I've been requiring my users to install Distribute, but one of
> them expressed reluctance to install it, partly because of the shadowing of
> setuptools.

The only extra features Distribute provides are:

- python 3 support
- the upload_doc command

The rest is the same APIs, minus some bug fixes we did, and the code that
prevents concurrency problems with an installed setuptools.

So if you just depend on pkg_resources, it means that your program can
also work with Setuptools.

So I would suggest to tell this user to install the tool he wants, and
just warn him that
he might bump into bugs in Setuptools that were fixed in Distribute.

IOW, choosing between Distribute or Setuptools should be up to your users,
*unless* you use python 3 support in your setup.py, that is only
present in Distribute.

And as far as I can see, Distribute as a drop-in replacement works
well, besides some
glitches we had to fix in zc.buildout and virtualenv because those
were "married" to setuptools.

Semi-related: If I can help in convincing this user, let me know

Regards
Tarek

From cool-rr at cool-rr.com  Mon Nov 30 18:16:12 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 30 Nov 2009 19:16:12 +0200
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
Message-ID: <a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>

You didn't answer my question about whether I can just yank out
`pkg_resources.py` and use it.


>> What would be the use case for this ?  I don't think it's the best
> >> practice to bundle
> >> other projects modules like that in most cases.
> >>
> >
> > Up to now I've been requiring my users to install Distribute, but one of
> > them expressed reluctance to install it, partly because of the shadowing
> of
> > setuptools.
>
> The only extra features Distribute provides are:
>
> - python 3 support
> - the upload_doc command
>
> The rest is the same APIs, minus some bug fixes we did, and the code that
> prevents concurrency problems with an installed setuptools.
>
> So if you just depend on pkg_resources, it means that your program can
> also work with Setuptools.
>
> So I would suggest to tell this user to install the tool he wants, and
> just warn him that
> he might bump into bugs in Setuptools that were fixed in Distribute.
>

Well, that sort of sucks. And this is my motivation for bundling the
`pkg_resources` from Distribute. The last thing I want is having my software
fail for my users because of setuptools while I have Distribute installed
locally and can't see the bug on my computer.


> IOW, choosing between Distribute or Setuptools should be up to your users,
> *unless* you use python 3 support in your setup.py, that is only
> present in Distribute.
>
> And as far as I can see, Distribute as a drop-in replacement works
> well, besides some
> glitches we had to fix in zc.buildout and virtualenv because those
> were "married" to setuptools.
>
> Semi-related: If I can help in convincing this user, let me know
>
>
I tried a bit, but I don't want to bother him more than I currently am.

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091130/654ad6e7/attachment.htm>

From pje at telecommunity.com  Mon Nov 30 18:57:01 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 12:57:01 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <loom.20091130T150801-770@post.gmane.org>
References: <loom.20091130T150801-770@post.gmane.org>
Message-ID: <20091130175708.5870B3A408B@sparrow.telecommunity.com>

At 02:09 PM 11/30/2009 +0000, Ram Rachum wrote:
>I've asked before about bundling Distribute. But now I ask, is it possible to
>pull out the `pkg_resources` module from the Distribute folder and 
>bundle only
>that with my project?

If your project is a standalone application with a completely 
isolated sys.path, then yes, you can bundle it.  If your project is a 
library or shares sys.path directories with other libraries or 
applications, then no, *do not bundle it*.

(This is true even for the original pkg_resources, but it's doubly 
true for Distribute's pkg_resources, as installing it may break a 
setuptools-based installation's ability to upgrade setuptools.)


From cool-rr at cool-rr.com  Mon Nov 30 19:00:25 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 30 Nov 2009 20:00:25 +0200
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <20091130175708.5870B3A408B@sparrow.telecommunity.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<20091130175708.5870B3A408B@sparrow.telecommunity.com>
Message-ID: <a422b72d0911301000s388616e6g6e512e648c3842f6@mail.gmail.com>

On Mon, Nov 30, 2009 at 7:57 PM, P.J. Eby <pje at telecommunity.com> wrote:

> At 02:09 PM 11/30/2009 +0000, Ram Rachum wrote:
>
>> I've asked before about bundling Distribute. But now I ask, is it possible
>> to
>> pull out the `pkg_resources` module from the Distribute folder and bundle
>> only
>> that with my project?
>>
>
> If your project is a standalone application with a completely isolated
> sys.path, then yes, you can bundle it.  If your project is a library or
> shares sys.path directories with other libraries or applications, then no,
> *do not bundle it*.
>
> (This is true even for the original pkg_resources, but it's doubly true for
> Distribute's pkg_resources, as installing it may break a setuptools-based
> installation's ability to upgrade setuptools.)
>

What I was thinking is to put it in its own package, and then import it like
`from my_package import pkg_resources`.

Would that still be problematic?

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091130/bd1cf20c/attachment-0001.htm>

From pje at telecommunity.com  Mon Nov 30 19:06:01 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 13:06:01 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.co
 m>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
Message-ID: <20091130180608.4F0183A408B@sparrow.telecommunity.com>

At 07:16 PM 11/30/2009 +0200, cool-RR wrote:
>Well, that sort of sucks. And this is my motivation for bundling the 
>`pkg_resources` from Distribute. The last thing I want is having my 
>software fail for my users because of setuptools while I have 
>Distribute installed locally and can't see the bug on my computer.

That's *really* unlikely.  Setuptools' runtime functionality (i.e., 
pkg_resources) has an extremely low bug count.  There have actually 
been more new pkg_resources bugs in Distribute's version of it (due 
to their changes) than there are outstanding reported bugs in the 
original pkg_resources.


From cool-rr at cool-rr.com  Mon Nov 30 19:10:29 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 30 Nov 2009 20:10:29 +0200
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <20091130180608.4F0183A408B@sparrow.telecommunity.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
Message-ID: <a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>

2009/11/30 P.J. Eby <pje at telecommunity.com>

> At 07:16 PM 11/30/2009 +0200, cool-RR wrote:
>
>> Well, that sort of sucks. And this is my motivation for bundling the
>> `pkg_resources` from Distribute. The last thing I want is having my software
>> fail for my users because of setuptools while I have Distribute installed
>> locally and can't see the bug on my computer.
>>
>
> That's *really* unlikely.  Setuptools' runtime functionality (i.e.,
> pkg_resources) has an extremely low bug count.  There have actually been
> more new pkg_resources bugs in Distribute's version of it (due to their
> changes) than there are outstanding reported bugs in the original
> pkg_resources.
>

Now I'm confused. If that's true, what reason is there to use Distribute's
`pkg_resources` at all?

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091130/24a3b518/attachment.htm>

From pje at telecommunity.com  Mon Nov 30 19:13:00 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 13:13:00 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911301000s388616e6g6e512e648c3842f6@mail.gmail.co
 m>
References: <loom.20091130T150801-770@post.gmane.org>
	<20091130175708.5870B3A408B@sparrow.telecommunity.com>
	<a422b72d0911301000s388616e6g6e512e648c3842f6@mail.gmail.com>
Message-ID: <20091130181307.8CC4A3A408B@sparrow.telecommunity.com>

At 08:00 PM 11/30/2009 +0200, cool-RR wrote:
>On Mon, Nov 30, 2009 at 7:57 PM, P.J. Eby 
><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
>At 02:09 PM 11/30/2009 +0000, Ram Rachum wrote:
>I've asked before about bundling Distribute. But now I ask, is it possible to
>pull out the `pkg_resources` module from the Distribute folder and bundle only
>that with my project?
>
>
>If your project is a standalone application with a completely 
>isolated sys.path, then yes, you can bundle it.  If your project is 
>a library or shares sys.path directories with other libraries or 
>applications, then no, *do not bundle it*.
>
>(This is true even for the original pkg_resources, but it's doubly 
>true for Distribute's pkg_resources, as installing it may break a 
>setuptools-based installation's ability to upgrade setuptools.)
>
>
>What I was thinking is to put it in its own package, and then import 
>it like `from my_package import pkg_resources`.
>
>Would that still be problematic?

Not if your modules are the only ones using any pkg_resources APIs 
within a given program.  But consider what happens if both your 
package and another package are calling 'require()' -- just to give a 
really simple example.  You will have two working_set objects that 
don't agree with each other, or with sys.path.  (Similar issues may 
also apply for other pkg_resources data structures and APIs.)

Bundling pkg_resources is really not a good idea for anything but a 
100% standalone application or similarly controlled environment, 
where you know that nothing else will be doing that kind of 
thing.  (Bundling it alongside your setup.py to use during 
installation -- but not actually installing it -- would also be 
relatively safe.)

If all you're worried about is that a user's version of setuptools 
might have some sort of bug in pkg_resources, you needn't be.  It's 
the single most stable part of the setuptools code base, both in the 
senses of "infrequently changing" and "not a lot of bugs".  Even if 
someone's using a version of pkg_resources that's say, 2 or 3 years 
old, you're not likely to see any bugs more severe than a spurious 
warning message that something might be being imported from more than 
one place.


From ziade.tarek at gmail.com  Mon Nov 30 20:10:34 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 20:10:34 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <20091130180608.4F0183A408B@sparrow.telecommunity.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
Message-ID: <94bdd2610911301110l71875557k1961ca69d51cf92d@mail.gmail.com>

2009/11/30 P.J. Eby <pje at telecommunity.com>:
> At 07:16 PM 11/30/2009 +0200, cool-RR wrote:
>>
>> Well, that sort of sucks. And this is my motivation for bundling the
>> `pkg_resources` from Distribute. The last thing I want is having my software
>> fail for my users because of setuptools while I have Distribute installed
>> locally and can't see the bug on my computer.
>
> That's *really* unlikely. ?Setuptools' runtime functionality (i.e.,
> pkg_resources) has an extremely low bug count. ?There have actually been
> more new pkg_resources bugs in Distribute's version of it (due to their
> changes) than there are outstanding reported bugs in the original
> pkg_resources.

As I said earlier, we've had our share of bugs because we needed to
make Distribute
work in some particular environments, but that was bootstraping issues
we've fixed. And
if we have more we will fix them and release another version of Distribute.

Those bugs have existed because you didn't open setuptools to more
maintainers, and
we had to find some strategy to bypass you. They are working good now,
and now we are just fixing
original bugs.

Some of those bugs was also due to the fact that you introduced

So I am afraid your statement is incorrect.

We have fixed for instance a bug in pkg_resources that was making it
fail in GAE. And there are more bugs that will
be fixed before you release the 12th pre-release of your 0.6 version.

From ianb at colorstudy.com  Mon Nov 30 20:17:18 2009
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 30 Nov 2009 13:17:18 -0600
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
Message-ID: <b654cd2e0911301117n7d94da6m26bbfc544c40749d@mail.gmail.com>

On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> As suggested in Catalog-SIG by some people, I would like to propose
> the addition of three more fields for 1.2:
>
> "Repository-URL"
>    A string containing the URL for the package's repository.
>   Example:
>        http://svn.python.org/projects/python/trunk/
>

I like this in concept, but it's a bit vague as just a URL.  What kind of
repository is it?  Sniffing is possible, but doesn't work that well.  For
pip I require something like hg+http://... (with exceptions for schemes that
are self-describing, like svn: and git:).

It's also unclear whether it should point to the trunk, a branch, or a tag
(related to the release).  As a URL that's only even possible for svn, as
git/hg/bzr all point to the repository and URLs can't point to revisions,
tags, etc.  (pip has some extensions to the URL format to accommodate these,
but they aren't formally defined at this time).

-- 
Ian Bicking  |  http://blog.ianbicking.org  |
http://topplabs.org/civichacker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091130/2b45a38d/attachment.htm>

From regebro at gmail.com  Mon Nov 30 20:22:35 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 30 Nov 2009 20:22:35 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
Message-ID: <319e029f0911301122n7620b0a7k295fe6fd33cfee37@mail.gmail.com>

On Mon, Nov 30, 2009 at 18:16, cool-RR <cool-rr at cool-rr.com> wrote:
> You didn't answer my question about whether I can just yank out
> `pkg_resources.py` and use it.

Yes, but then it will again shadow the pkg_resources.py from
setuptools/Distribute, so why your customer would be more happy with
that I don't know.

> Well, that sort of sucks. And this is my motivation for bundling the
> `pkg_resources` from Distribute. The last thing I want is having my software
> fail for my users because of setuptools while I have Distribute installed
> locally and can't see the bug on my computer.

You can have two pythons installed, one with setuptools and one with Distribute.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From pje at telecommunity.com  Mon Nov 30 20:25:16 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 14:25:16 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <94bdd2610911301110l71875557k1961ca69d51cf92d@mail.gmail.co
 m>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<94bdd2610911301110l71875557k1961ca69d51cf92d@mail.gmail.com>
Message-ID: <20091130192523.A68213A408B@sparrow.telecommunity.com>

At 08:10 PM 11/30/2009 +0100, Tarek Ziad? wrote:
>2009/11/30 P.J. Eby <pje at telecommunity.com>:
> > At 07:16 PM 11/30/2009 +0200, cool-RR wrote:
> >>
> >> Well, that sort of sucks. And this is my motivation for bundling the
> >> `pkg_resources` from Distribute. The last thing I want is having 
> my software
> >> fail for my users because of setuptools while I have Distribute installed
> >> locally and can't see the bug on my computer.
> >
> > That's *really* unlikely.  Setuptools' runtime functionality (i.e.,
> > pkg_resources) has an extremely low bug count.  There have actually been
> > more new pkg_resources bugs in Distribute's version of it (due to their
> > changes) than there are outstanding reported bugs in the original
> > pkg_resources.
>
>As I said earlier, we've had our share of bugs because we needed to
>make Distribute
>work in some particular environments, but that was bootstraping issues
>we've fixed. And
>if we have more we will fix them and release another version of Distribute.

I wasn't criticizing Distribute - I was using Distribute to show just 
*how low* pkg_resources' bug count is.

(You know, this is now the third time in the last few days where 
you've interpreted my *positive* comments about your work (e.g. PEP 
386) to someone else here as being some sort of criticism or argument 
with you.)


From pje at telecommunity.com  Mon Nov 30 20:28:46 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 14:28:46 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.co
 m>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
Message-ID: <20091130192851.9D8163A408B@sparrow.telecommunity.com>

At 08:10 PM 11/30/2009 +0200, cool-RR wrote:
>Now I'm confused. If that's true, what reason is there to use 
>Distribute's `pkg_resources` at all?

Unless there's some bug they've fixed in it that nobody has reported 
here or on the setuptools bug tracker, there isn't any.

(And of course, if such a bug exists, I would certainly like to know about it.)


From regebro at gmail.com  Mon Nov 30 20:30:20 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 30 Nov 2009 20:30:20 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
Message-ID: <319e029f0911301130p7df2bfeev520bae64bc457e3@mail.gmail.com>

On Mon, Nov 30, 2009 at 19:10, cool-RR <cool-rr at cool-rr.com> wrote:
> Now I'm confused. If that's true, what reason is there to use Distribute's
> `pkg_resources` at all?

Same reason as for using distribute in the first place: You use it if
you encounter bugs in setuptools (or use Python 3). Otherwise there's
no benefit. Distribute doesn't have additional features.

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

From regebro at gmail.com  Mon Nov 30 20:30:49 2009
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 30 Nov 2009 20:30:49 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <319e029f0911301130p7df2bfeev520bae64bc457e3@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
	<319e029f0911301130p7df2bfeev520bae64bc457e3@mail.gmail.com>
Message-ID: <319e029f0911301130j30a21bffq7b6bbbd4831512b7@mail.gmail.com>

On Mon, Nov 30, 2009 at 20:30, Lennart Regebro <regebro at gmail.com> wrote:
> Distribute doesn't have additional features.
(Well, of any great significance anyway).

>
> --
> Lennart Regebro: Python, Zope, Plone, Grok
> http://regebro.wordpress.com/
> +33 661 58 14 64
>

From pje at telecommunity.com  Mon Nov 30 20:31:10 2009
From: pje at telecommunity.com (P.J. Eby)
Date: Mon, 30 Nov 2009 14:31:10 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911301059w5e094e93o9116fc85b3f33216@mail.gmail.co
 m>
References: <loom.20091130T150801-770@post.gmane.org>
	<20091130175708.5870B3A408B@sparrow.telecommunity.com>
	<a422b72d0911301000s388616e6g6e512e648c3842f6@mail.gmail.com>
	<20091130181307.8CC4A3A408B@sparrow.telecommunity.com>
	<a422b72d0911301059w5e094e93o9116fc85b3f33216@mail.gmail.com>
Message-ID: <20091130193116.0F7F13A408B@sparrow.telecommunity.com>

At 08:59 PM 11/30/2009 +0200, cool-RR wrote:
>Okay. But I don't use `require()`, the only thing I need from 
>`pkg_resources` is the ability to extract resources from folders. So 
>will there be any problem if I bundle it for that?

If that's all you're using, probably not.  In any case, if that's the 
only functionality you're using, it's doubtful that there's really 
any difference to that functionality in Distribute, especially in the 
case of packages that aren't installed in zipfile format.


From ziade.tarek at gmail.com  Mon Nov 30 20:32:07 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 20:32:07 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <20091130192523.A68213A408B@sparrow.telecommunity.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<94bdd2610911301110l71875557k1961ca69d51cf92d@mail.gmail.com>
	<20091130192523.A68213A408B@sparrow.telecommunity.com>
Message-ID: <94bdd2610911301132j55e77adbvccb42e21b4a6308c@mail.gmail.com>

On Mon, Nov 30, 2009 at 8:25 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 08:10 PM 11/30/2009 +0100, Tarek Ziad? wrote:
>>
>> 2009/11/30 P.J. Eby <pje at telecommunity.com>:
>> > At 07:16 PM 11/30/2009 +0200, cool-RR wrote:
>> >>
>> >> Well, that sort of sucks. And this is my motivation for bundling the
>> >> `pkg_resources` from Distribute. The last thing I want is having my
>> >> software
>> >> fail for my users because of setuptools while I have Distribute
>> >> installed
>> >> locally and can't see the bug on my computer.
>> >
>> > That's *really* unlikely. ?Setuptools' runtime functionality (i.e.,
>> > pkg_resources) has an extremely low bug count. ?There have actually been
>> > more new pkg_resources bugs in Distribute's version of it (due to their
>> > changes) than there are outstanding reported bugs in the original
>> > pkg_resources.
>>
>> As I said earlier, we've had our share of bugs because we needed to
>> make Distribute
>> work in some particular environments, but that was bootstraping issues
>> we've fixed. And
>> if we have more we will fix them and release another version of
>> Distribute.
>
> I wasn't criticizing Distribute - I was using Distribute to show just *how
> low* pkg_resources' bug count is.
>
> (You know, this is now the third time in the last few days where you've
> interpreted my *positive* comments about your work (e.g. PEP 386) to someone
> else here as being some sort of criticism or argument with you.)

That was not a positive or negative comment about Distribute, but an
incomplete statement.

So I've corrected this statement.

I agree that pkg_resources doesn't have a lot of bug. But I am also
saying that Distribute's versions has less bugs.

From ziade.tarek at gmail.com  Mon Nov 30 20:39:32 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 20:39:32 +0100
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
Message-ID: <94bdd2610911301139g5e85bbbfy462073a209f362a7@mail.gmail.com>

On Mon, Nov 30, 2009 at 7:10 PM, cool-RR <cool-rr at cool-rr.com> wrote:
>
>
> 2009/11/30 P.J. Eby <pje at telecommunity.com>
>>
>> At 07:16 PM 11/30/2009 +0200, cool-RR wrote:
>>>
>>> Well, that sort of sucks. And this is my motivation for bundling the
>>> `pkg_resources` from Distribute. The last thing I want is having my software
>>> fail for my users because of setuptools while I have Distribute installed
>>> locally and can't see the bug on my computer.
>>
>> That's *really* unlikely. ?Setuptools' runtime functionality (i.e.,
>> pkg_resources) has an extremely low bug count. ?There have actually been
>> more new pkg_resources bugs in Distribute's version of it (due to their
>> changes) than there are outstanding reported bugs in the original
>> pkg_resources.
>
> Now I'm confused. If that's true, what reason is there to use Distribute's
> `pkg_resources` at all?

See my previous message. Right now the biggest difference in this
module is the fact that Distribute's
one works for GAE.

The other big difference is that if you happen to have another problem
with it in the future, you can count on an actively
maintained, community-driven project. (IOW not locked by a single
maintainer that has been AWOL for most of
the time in the past 2 years)

From cool-rr at cool-rr.com  Mon Nov 30 19:59:43 2009
From: cool-rr at cool-rr.com (cool-RR)
Date: Mon, 30 Nov 2009 20:59:43 +0200
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <20091130181307.8CC4A3A408B@sparrow.telecommunity.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<20091130175708.5870B3A408B@sparrow.telecommunity.com>
	<a422b72d0911301000s388616e6g6e512e648c3842f6@mail.gmail.com>
	<20091130181307.8CC4A3A408B@sparrow.telecommunity.com>
Message-ID: <a422b72d0911301059w5e094e93o9116fc85b3f33216@mail.gmail.com>

> What I was thinking is to put it in its own package, and then import it
>> like `from my_package import pkg_resources`.
>>
>> Would that still be problematic?
>>
>
> Not if your modules are the only ones using any pkg_resources APIs within a
> given program.  But consider what happens if both your package and another
> package are calling 'require()' -- just to give a really simple example.
>  You will have two working_set objects that don't agree with each other, or
> with sys.path.  (Similar issues may also apply for other pkg_resources data
> structures and APIs.)
>
> Bundling pkg_resources is really not a good idea for anything but a 100%
> standalone application or similarly controlled environment, where you know
> that nothing else will be doing that kind of thing.  (Bundling it alongside
> your setup.py to use during installation -- but not actually installing it
> -- would also be relatively safe.)
>
> If all you're worried about is that a user's version of setuptools might
> have some sort of bug in pkg_resources, you needn't be.  It's the single
> most stable part of the setuptools code base, both in the senses of
> "infrequently changing" and "not a lot of bugs".  Even if someone's using a
> version of pkg_resources that's say, 2 or 3 years old, you're not likely to
> see any bugs more severe than a spurious warning message that something
> might be being imported from more than one place.
>

Okay. But I don't use `require()`, the only thing I need from
`pkg_resources` is the ability to extract resources from folders. So will
there be any problem if I bundle it for that?

Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091130/6705794c/attachment.htm>

From restless.being at gmail.com  Mon Nov 23 12:02:47 2009
From: restless.being at gmail.com (Restless Being)
Date: Mon, 23 Nov 2009 12:02:47 +0100
Subject: [Distutils] Buildout and PIL problem
Message-ID: <d5fde36f0911230302s1a7f0c85y2511d12116c845c2@mail.gmail.com>

Hi!

  I have a virtualenv created with --no-site-packages option and I'm going
to install there Django and some required libs using buildout. The
problematic library is PIL. Here is my buildout.cfg

[buildout]
parts = PIL django

[django]
recipe = djangorecipe
version = 1.1
project = project
eggs =
psycopg2
PIL
south
django-reversion
django-cms
wsgi = true
settings = production

[PIL]
recipe = zc.recipe.egg:custom
egg = PIL==1.1.6
find-links = http://dist.repoze.org

and it is something generally recommended by other users. See:
http://www.stereoplex.com/two-voices/a-django-development-environment-with-zc-buildout

I also tried some other versions of [PIL] like:
[PIL]
recipe = zc.recipe.egg:custom
egg = PIL==1.1.6
find-links = http://dist.repoze.org/PIL-1.1.6.tar.gz
index = http://dist.repoze.org/index


The first problem is that the above recipe installs PIL 1.1.7 instead of
1.1.6. Here is install log:

python bin/buildout -c devel.cfg
Updating django.
Getting distribution for 'PIL'.
WARNING: '' not a valid package name; please use only.-separated package
names in setup.py
In file included from /usr/include/python2.6/Python.h:8,
from libImaging/ImPlatform.h:10,
from libImaging/Imaging.h:14,
from Tk/tkImaging.c:53:
/usr/include/python2.6/pyconfig.h:1028:1: warning: "_POSIX_C_SOURCE"
redefined
In file included from /usr/include/stdio.h:28,
from /usr/include/tcl.h:141,
from /usr/include/tk.h:21,
from Tk/tkImaging.c:51:
/usr/include/features.h:210:1: warning: this is the location of the previous
definition
--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version 1.1.7
platform linux2 2.6.4 (r264:75706, Oct 27 2009, 06:16:59)
[GCC 4.4.1]
--------------------------------------------------------------------
--- TKINTER support available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
--- FREETYPE2 support available
--- LITTLECMS support available
--------------------------------------------------------------------
To check the build, run the selftest.py script.
zip_safe flag not set; analyzing archive contents...
Image: module references __file__
Got PIL 1.1.7.

Second problem is that the above PIL installation doesn't work with Django:

bin/django validate
Error: One or more models did not validate:
picture.picture: "image": To use ImageFields, you need to install the Python
Imaging Library. Get it at http://www.pythonware.com/products/pil/ .


At the same time, everything is ok when I manually install PIL using: pip
install http://effbot.org/downloads/Imaging-1.1.6.tar.gz

Any hints?
Jakub
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20091123/a547dfca/attachment.htm>

From ssteinerx at gmail.com  Mon Nov 30 23:08:56 2009
From: ssteinerx at gmail.com (ssteinerX@gmail.com)
Date: Mon, 30 Nov 2009 17:08:56 -0500
Subject: [Distutils] Bundling pkg_resources
In-Reply-To: <94bdd2610911301139g5e85bbbfy462073a209f362a7@mail.gmail.com>
References: <loom.20091130T150801-770@post.gmane.org>
	<94bdd2610911300810he464899h1bb07cedff6ef2c3@mail.gmail.com>
	<a422b72d0911300841g6af83325mb13d3c7a9df0f0a3@mail.gmail.com>
	<94bdd2610911300859n6b572b10jc16292a44465cdf1@mail.gmail.com>
	<a422b72d0911300916o133a14b3pb0e97a7438acbd9d@mail.gmail.com>
	<20091130180608.4F0183A408B@sparrow.telecommunity.com>
	<a422b72d0911301010k5a02e70ai851e48b354b100ac@mail.gmail.com>
	<94bdd2610911301139g5e85bbbfy462073a209f362a7@mail.gmail.com>
Message-ID: <501846A1-B278-4FC8-BFF6-2B1E3670239F@gmail.com>


On Nov 30, 2009, at 2:39 PM, Tarek Ziad? wrote:

> maintainer that has been AWOL for most of the time in the past 2 years)

AWOL is an acronym for "absent without leave" or "absent without official leave".

Yes, it sucked that setuptools got abandoned but nobody had to give him leave.

S


From ziade.tarek at gmail.com  Mon Nov 30 23:30:19 2009
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Mon, 30 Nov 2009 23:30:19 +0100
Subject: [Distutils] PEP 345 - 3 new fields
In-Reply-To: <b654cd2e0911301117n7d94da6m26bbfc544c40749d@mail.gmail.com>
References: <94bdd2610911221452k282d59f6ud465dd4db3b8ba08@mail.gmail.com>
	<b654cd2e0911301117n7d94da6m26bbfc544c40749d@mail.gmail.com>
Message-ID: <94bdd2610911301430x613869c9j8909e7f8ecc07de@mail.gmail.com>

On Mon, Nov 30, 2009 at 8:17 PM, Ian Bicking <ianb at colorstudy.com> wrote:
> On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>>
>> As suggested in Catalog-SIG by some people, I would like to propose
>> the addition of three more fields for 1.2:
>>
>> "Repository-URL"
>> ? ?A string containing the URL for the package's repository.
>> ? Example:
>> ? ? ? ?http://svn.python.org/projects/python/trunk/
>
> I like this in concept, but it's a bit vague as just a URL. ?What kind of
> repository is it? ?Sniffing is possible, but doesn't work that well. ?For
> pip I require something like hg+http://... (with exceptions for schemes that
> are self-describing, like svn: and git:).

I didn't think that Pip would use this field to do some automatic process,
I think the use case was to simply display that url at PyPI.

OTHO, Repository-Browse-URL already plays that role, so it would be
interesting to
provide in "Repository-URL", urls that could be processed by installers.

Do you have a particular use case in mind ?

> It's also unclear whether it should point to the trunk, a branch, or a tag
> (related to the release). ?As a URL that's only even possible for svn, as
> git/hg/bzr all point to the repository and URLs can't point to revisions,
> tags, etc. ?(pip has some extensions to the URL format to?accommodate?these,
> but they aren't formally defined at this time).

Good point. I think this would end up being the trunk/default path *or* the root
of their repository, because I doubt  developers will update this
field for every release they
have. And, as you said, in mercurial for instance, we would need to
"hg up some_tag"  to have
a similar feature than svn branches.

So at the end, maybe we could just keep "Repository-Browse-URL" and
drop the other one unless
we can come up with some use cases for automatic processing of the url.


Tarek.